# Autonomous Driving Control Using the DDPG and RDPG Algorithms
## 3 Our DRL Control Strategies
### Designs for the Reward Mechanism  
The reward mechanism is composed of several main components, i.e., \(R_{area}\), \(R_{sensor}\), \(R_{velocity}\), \(R_{direction}\), and \(R_{punishment}\). Consider the following scenario: if a car is following the road accurately, the image captured by the front camera mounted on the vehicle will include a higher percentage of the road area (Figure 5a). On the contrary, if the image includes a lower percentage of the road area, this means that the car cannot properly follow the road and will become out of control in a short time (Figure 5b). According to this idea, the first component of the reward mechanism was set to:  
\[R_{area}=(A_{road}-T_{1})\times F_{1}, \tag{1}\]  
where \(A_{road}\) is the measure of the road area, \(T_{1}\) the threshold, and \(F_{1}\) the weight. Both \(T_{1}\) and \(F_{1}\) are responsible for confining \(R_{area}\) within a certain range.  
Figure 4: The architecture of our work.  
Here, we want to detail how to use computer vision to obtain the measure of the road area. In accordance with the discussion presented in Section 2.6, the HSV color model is better for the color gradations found in nature. Hence, using Figure 6a as an example, the first step is to convert the color space (Figure 6b). Then, binarization [19; 20] is applied to replace each pixel in the image with a black/white pixel (Figure 6c); this helps us obtain the measure of the road area. The corresponding result of the example in Figure 6a is illustrated in Figure 6d.  
Figure 5: (**a**) An example of an image including a higher percentage of the road area; (**b**) an example of an image including a lower percentage of the road area.  
Figure 6: An example of an image to obtain the measure of the road area: (**a**) original image; (**b**) converted to the HSV color space; (**c**) binarization; (**d**) result with the contour line.  
The second component is \(R_{sensor}\), which is motivated by the difference between the distance from the ego-vehicle to the right and left road edges. Since the environment is right-hand traffic (RHT), we will obtain a higher reward while the vehicle keeps to the right side. Note that the distance from the ego-vehicle to the right and left road edges can be obtained by attaching the distance sensors to the vehicle.  
\[R_{sensor}=(D_{left}-D_{right})\times F_{2}, \tag{2}\]  
where \(D_{left}\) and \(D_{right}\) are obtained by the distance sensors and \(F_{2}\) is responsible for confining \(R_{sensor}\) within a certain range.  
The next two components regard the stability of the ego-vehicle while driving, \(R_{velocity}\) and \(R_{direction}\). More specifically, the vehicle speed is similar to that at the latest point in time; it will be given a larger award. Furthermore, the concept of vehicle direction is similar to that of vehicle speed. Hence, we can acquire two designs for the reward components:  
\[R_{velocity} =1/((|V_{t}-V_{t-1}|-T_{3})\times F_{3}), \tag{3}\] \[R_{direction} =1/((|D_{t}-D_{t-1}|-T_{4})\times F_{4}), \tag{4}\]  
where \(V_{t}\) is the vehicle speed at time \(t\) and \(D_{t}\) the vehicle direction at time \(t\). Moreover, \(T_{3}\), \(T_{4}\), \(F_{3}\), and \(F_{4}\) are responsible for confining \(R_{velocity}\) and \(R_{direction}\) within certain limits.  
The last one is the punishment part, \(R_{punishment}\). Here, violating a traffic regulation will cause a negative reward and then reset the environment to the initial state. The design is shown as follows:  
\[R_{punishment}=\left\{\begin{array}{l}Reset=C(constant)\\ Other=0\end{array}\right., \tag{5}\]  
Finally, the complete reward mechanism is composed of all the above components:  
\[R_{total}=R_{area}+R_{sensor}+R_{velocity}+R_{direction}+R_{punishment}. \tag{6}\]