# A Deep Reinforcement Learning Approach for Efficient, Safe and Conformtable Driving +
## 3 Design and Implementation of the DRL Framework
### The DRL Model
#### 3.1.2 DRL-Based Acc Application  
Figure 1 presents the high-level view of the framework we propose. The framework is considered to be deployed in the ego vehicle, where data about the system state are collected from local sensors as well as from neighboring vehicles, and a decision on the desired acceleration is made. _State Space:_ The state space at any given time slot \(t\) comprises: (i) the lead vehicle acceleration \(\alpha(t)\), (ii) the headway \(\vartheta(t)\), (iii) the headway derivative \(\Delta\vartheta(t)\), (iv) the longitudinal slip \(\xi(t)\), and (v) the friction coefficient \(\mu(t)\). In the state space, V2X communication is used to acquire the lead vehicle acceleration (\(\alpha(t)\)), and it is assumed that the ego vehicle is equipped with an estimation mechanism to ascertain the tire-road friction coefficient (\(\mu(t)\)). The rest of the state space parameters are expressed in the following equations:  
\[\vartheta(t)=\frac{\Delta P_{lead}(t)}{\mathcal{V}_{ego}(t)} \tag{1}\] \[\Delta\vartheta(t)=\vartheta(t)-\vartheta(t-1)\] (2) \[\xi(t)= \begin{cases}\frac{\chi^{\Psi_{eq}}_{eq}(t)-\mathcal{V}^{ \Psi}_{ego}(t)}{\mathcal{V}^{\Psi}_{ego}(t)},\ \dot{x}(t)>=0\\ \frac{\chi^{\Psi}_{ego}(t)-\mathcal{V}^{\Psi}_{ego}(t)}{\mathcal{V}^{\Psi}_{ego }(t)},\ \text{otherwise}\end{cases}  \tag{3}\]where \(\Delta P_{lead}(t)\) represents the inter-vehicular distance between two car following vehicles, \(\mathcal{V}_{ego}(t)\) indicates the ego vehicle's velocity, \(\vartheta(t-1)\) and \(\vartheta(t)\) are the headway values at time \(t-1\) and \(t\) (respectively), \(\mathcal{V}_{ego}^{R}(t)\) and \(\mathcal{V}_{ego}^{W}(t)\) are the ego vehicle's equivalent rotational velocity and longitudinal axle velocity (respectively) [28], and \(\bar{x}(t)\) is the ego vehicle's acceleration at time \(t\). Compared to [6; 7], our model takes into account not just the wheel longitudinal slip ratio but also the tire-road friction coefficient accounting for the vehicle stability. _Action Space:_ The action space \(a(t)\in\mathcal{A}\), which denotes the acceleration to be adopted by the ego vehicle, encompasses values within the range [\(-2\), 1.47] to ensure a comfortable ride for the driver and passengers [29]. The sampling period (\(\tau\)) of our framework is set at 100 ms, with state observation and action decisions taken at each such interval. _Reward Components:_ We express the reward, i.e., the numerical value received by the DRL Agent from the environment as a direct response to the DRL Agent's action, as a multi-objective function. Specifically, it includes three components: Headway (signifying traffic flow efficiency), Comfort (for ride quality), and Stability (denoting safety), each component taking values in the [\(-1\),\(1\)] range. More formally, given state \(s(t)\) and action \(a(t)\), the reward is given by  
\[r(s(t),a(t))=x_{hw}\cdot r_{hw}(s(t),a(t))+x_{cmf}\cdot r_{c}(s(t),a(t))+x_{ stb}\cdot r_{s}(s(t),a(t)) \tag{4}\]  
where \(x_{hw}\), \(x_{cmf}\), and \(x_{stb}\) are weighting coefficients that, as detailed later, are set dynamically over time, and \(r_{hw}(s(t),a(t))\), \(r_{c}(s(t),a(t))\), and \(r_{s}(s(t),a(t))\) are, respectively, the headway, comfort, and vehicle stability reward component at time step \(t\). The reward components are described in detail below. _Headway reward component:_ Time headway can be used as an alternative way to calculate the distance between the lead and ego vehicle, and it is expressed as in Equation (1). Although the term gap is widely used to define this time interval between consecutive vehicles, in this study, we will use the term \"headway\", according to the recent related papers on ACC [30; 31]. The desired time headway between two vehicles is set at 1.3 s, and a headway value under 0.5 s indicates a high risk of collision between the two vehicles [7]. However, the headway value becomes \(\infty\) when the ego vehicle approaches a stand-still situation, i.e., zero velocity.