# Vehicle-Following Control Based on Deep Reinforcement Learning
## 4 Methodology
### Proximal Policy Optimization Algorithm  
The proximal policy optimization algorithm is a reinforcement learning algorithm based on policy gradient, which is evolved from the trust region policy optimization (TRPO) [28]. If the agent's reward in the environment is higher, it means that they have a stronger ability to complete tasks, and the ultimate goal of all policy gradient methods is to maximize the cumulative reward, that is, to maximize \(\eta(\pi)=\mathbb{E}_{\pi_{0},d_{0},s_{1},\mu_{1},...}\left[\sum_{t=0}^{ \infty}\gamma^{t}(s_{t})\right],\) among them, \(\gamma\) is the discount factor, indicating that the farther away from the current state, the smaller the impact on the current state, \(\eta(\pi)\) refers to the cumulative reward obtained \begin{table}
\begin{tabular}{c c c} \hline \hline
**Action** & **Range** & **Description** \\ \hline _acceleration_ & \([0,1]\) & \
\begin{tabular}{c} Throttle force 0 means not to step on the throttle, 1 means fully depressed. \\ \end{tabular} \\ \hline _brake_ & \([0,1]\) &
\begin{tabular}{c} Braking force, 0 means no braking, 1 means fully depressed. \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Action Space. when performing actions according to the policy \(\pi\). \(S_{0},S_{1},S_{2}\ldots\) represents the state transition of the agent in the environment, treat state transitions as a given distribution \(s\sim\rho(s)\). The TRPO algorithm proposes to use the advantage function \(A_{\pi}(s,a)=Q_{\pi}(s,a)-V_{\pi}(s)\) to evaluate the quality of executing an action, where \(Q_{\pi}(s,a)\) is the value-action pair, and \(V_{\pi}(s)\) is the state value. It has been proved that the cumulative reward of a new policy \(\pi_{\text{new}}\) can be expressed as:  
\[\eta(\pi_{\text{new}})=\eta(\pi)+\mathbb{E}_{s_{0},a_{0},\cdots \cdots\pi_{\text{new}}}\left[\sum_{t=0}^{\infty}\gamma^{t}A_{\pi_{\text{old}}} \left(s_{t},a_{t}\right)\right] \tag{2}\]  
That is, the cumulative reward of the old policy plus the cumulative advantage function of the new policy. Therefore, if \(\mathbb{E}_{s_{0},a_{0},\cdots\sim\pi_{\text{new}}}\left[\sum_{t=0}^{\infty} \gamma^{t}A_{\pi_{\text{old}}}\left(s_{t},a_{t}\right)\right]\) can be guaranteed to be greater than or equal to 0, the monotonic increase of the cumulative reward can be guaranteed, that is, the optimization of the strategy. Since the cumulative advantage function cannot be calculated directly, TRPO uses the importance sampling method to estimate the advantage function and uses the KL divergence to limit the update range of the policy to ensure the monotony of the cumulative reward, that is, to ensure the continuous optimization of policy. The core problem of TRPO algorithm is defined as:  
\[ \begin{split}\text{maximize}_{\theta}\,\mathbb{E}_{s\sim\pi_{ \theta_{old}},a\sim\pi_{\theta_{old}}}\left[\frac{\pi_{\theta}(a|s)}{\pi_{ \theta_{old}}(a|s)}A_{\theta_{\text{old}}}\left(s,a\right)\right]\\ \text{subject to }\mathbb{E}_{s\sim\pi_{\theta_{old}}}\left[D_{KL} \big{(}\pi_{\theta_{old}}(\cdot\mid s)\|\pi_{\theta}(\cdot\mid s)\big{)} \right]\leq\delta\end{split}  \tag{3}\]  
The TRPO algorithm uses the KL divergence method to calculate the confidence region, and the update range of the control policy network is within a certain range to ensure that the value of the cumulative advantage function is greater than or equal to 0. However, the method of updating the policy in the trust region is complex and inefficient, so the PPO algorithm proposes to use the method of clipping the importance weight to limit the updating range of the policy.