# A Deep Reinforcement Learning Approach for Efficient, Safe and Conformtable Driving +
## 3 Design and Implementation of the DRL Framework
### The DRL Model
#### 3.1.1 Preliminaries  
In general, Reinforcement Learning (RL) is a sequential decision-making problem in which an agent observes certain states (\(s(t)\)) as a representation of the environment and chooses an action (\(a(t)\)) based on the policy (\(\pi\)) at a given time step \(t\). Given the action, the environment will transition to a new state \((s^{\prime})\) and receive a numerical value (\(r(s(t),a(t))\)) as a reward. In RL, the interaction between the agent and the environment is typically formulated as a discounted Markov Decision Process (MDP). The discounted MDP can be formally defined as a tuple \((\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma)\) where \(\mathcal{S}\) is the state space, \(\mathcal{A}\) is the action space, \(\mathcal{P}\) is the state transition probability matrix (\(\mathcal{P}^{a}_{ss^{\prime}}=\mathbb{P}[s(t+1)=s^{\prime}|s(t)=s,a(t)=a]\)) conditioned to the action taken by policy \(\pi\), \(\mathcal{R}\) is the reward function (\(r(s(t),a(t))=\mathbb{E}[r(s(t),a(t))|s(t)=s,a(t)=a]\)) and \(\gamma\in[0,1]\) is a discount factor that defines the importance of future rewards relative to the instant rewards. Policy \(\pi\) defines the decision-making strategy through which an RL agent chooses an action based on the previous set of observations. The RL agent aims to choose actions that maximize the expected discounted reward \(\mathbb{E}[\sum_{k=0}^{T}\gamma^{k}r(s(t+k),a(t+k))]\) where \(\gamma\) is the discount rate and \(r(s(t+k),a(t+k))\) is the immediate reward received at time step \(t+k\).  
An RL agent can utilize either model-based or model-free techniques to find the optimal solution to a given problem. Model-based approaches properly model the environment's transitions, where Dynamic Programming (DP)-based algorithms can identify the optimal solution. We resort to model-free solutions because, in most situations, the state transitions of the environment are challenging to model due to the intrinsic nature of the transitions. Such methods do not require a comprehensive understanding of transitions; instead, they rely on experience samples from the environment where Monte-Carlo Temporal-Difference (TD) Learning-based methods are applied. Q-learning [26] and SARSA [27] are the two most popular TD model-free algorithms used to solve RL problems. However, tabular approaches such as Q-learning become increasingly difficult to implement when there are a vast number of states and actions, as the \(Q(s,a)\) values must be stored for all state-action pairs. Therefore, with deep neural networks as function approximators, \(Q(s,a|\theta)\) can be used to represent the state and action value pairs, with \(\theta\) signifying the neural network's weights. The integration of deep neural networks and reinforcement learning techniques in Deep Reinforcement Learning facilitates the training of a DRL agent to reach the ideal solution for complex real-world problems. In this study, the Deep Deterministic Policy Gradient (DDPG) [11], a model-free Deep Reinforcement Learning algorithm, is used to learn the optimal policy for the DRL-based ACC application. It is worth noting that the DDPG algorithm is equipped to handle control action space in a continuous domain and uses an actor-critic framework to learn a deterministic policy.