# A Deep Reinforcement Learning Approach for Efficient, Safe and Conformtable Driving +
## 3 Design and Implementation of the DRL Framework
### The DRL Model
#### 3.1.2 DRL-Based Acc Application
This causes undesirable effects in DRL states, rewards, and the agent's learning progress. To overcome the issue, we have saturated the ego vehicle velocity to 2.16 m/s with the secure stand-still distance between two successive vehicles as 2.81 m [32], and the optimal headway of 1.3 s, as reported in Equation (5). \[\mathcal{V}_{ego}= \begin{cases}\mathcal{V}_{ego},\mathcal{V}_{ego}>2.16\\ \mathcal{V}_{ego}^{sqr},\mathcal{V}_{ego}\leq 2.16\\ \mathcal{V}_{ego}^{sat}=\frac{2.81\,m}{1.3\,s}\Rightarrow 2.16\,m/s\end{cases}  \tag{5}\]  
A Log-Normal distribution function is used to model the headway reward component (\(r_{hw}(s(t),a(t))\)), which provides a maximum reward of +1 for a headway of 1.3 s and a minimum of \(-1\) for 0.5 s. It is formulated as  
\[r_{hw}(s(t),a(t))=2\cdot(F_{hw}-0.5),\quad\text{with} \tag{6}\] \[F_{hw}=0.4944\cdot f_{lognorm}(\vartheta(t)|\epsilon=0.285, \sigma=0.15),\] (7) \[f_{lognorm}(x|\epsilon,\sigma)=\frac{1}{\sigma\sqrt{2\pi}}\exp \frac{(-(lnx-\epsilon)^{2})}{2\sigma^{2}} \tag{8}\]  
_Comfort reward component:_ Jerk \(j(t)\), i.e., the time derivative of the vehicle's acceleration, measures the comfort of the driver and passengers. In this work, the comfort reward component is designed using the absolute values of the jerk with values less than 0.9 m/s\({}^{3}\) corresponding to the best comfort, while values above 1.3 m/s\({}^{3}\) fall under aggressive driving behavior [29]. When \(j(t)\) increases from 0.6 m/s\({}^{3}\) to 2 m/s\({}^{3}\), the comfort reward value gradually drops until it reaches -1, the lower reward limit. The desired reward trend for jerk behavior has been formulated through the polynomial curve fitting method. However, in critical situations, the safety of passengers is of more importance than comfort. Thus, we consider the Time-to-Collision (TTC), which indicates the time until the potential collision occurs, as a safety indicator and discount comfort, instead giving priority to safety in dangerous situations. According to European regulations, \(\text{TTC}\leq 4\,\text{s}\) indicates a critical situation. Thus, we write:  
\[r_{c}(s(t),a(t))=w_{cmf}\cdot f(jerk),\quad\text{with} \tag{9}\]  
\[f(jerk)=polytfit(j(t),9),\qquad w_{cmf}= \begin{cases}1,TTC(t)>4\\ 0,TTC(t)\leq 4\,.\end{cases}  \tag{10}\]  
where \(w_{cmf}\) is the weight indicator to disregard comfort in critical situations. _Stability reward component:_ The longitudinal stability is valued in terms of the maximum tractive effort of a tire on a road contact area. According to the experimental data, a longitudinal slip lower than 0.2 is regarded as a stable condition [33]. In this case, a tanh function is used to represent the stability reward component. We write:  
\[r_{s}(s(t),a(t))=2.0099\cdot(F_{stb}+1)\quad\text{with}\quad F_{stb}=\tanh(-3 \cdot\xi(t))\,. \tag{11}\]  
_Dynamic weight coefficients:_ The dynamic coefficients \(x_{hw}\), \(x_{stb}\), and \(x_{cmf}\) play an important role in weighing each reward component; their values depend on the operating region, with the ideal regions being defined as \(1.25\leq\vartheta(t)\leq 1.35\) for headway, \(0\leq|\xi(t)|\leq 0.2\) for stability, and \(0\leq|j(t)|\leq 0.9\) for comfort. To amplify the negative rewards, those reward components taking values outside the ideal region are given more importance than others. However, since the coefficients must always sum to 1, the maximum (minimum) total reward a DRL agent can receive in time step \(t\) is still \(+1\) (\(-1\)). In particular, we set the dynamic coefficients in such a way that, if all reward components are outside or inside the ideal regions, then each of them will obtain \(1/3\) as their coefficient. In other cases, with one of the reward components (e.g., comfort) being outside the ideal region, the dynamic coefficient values will be \(2/3\) for the comfort reward component and \(1/6\) for each of the other components. Therefore, the dynamic coefficient values will sum to \(+1\) for all combinations, and the resulting reward will be between \([-1,\,+1]\).