# Autonomous Driving Decision Control Based on Improved Proximal Policy Optimization Algorithm
## 2 Reinforcement Learning  
The nature of reinforcement learning is interactive learning, i.e., it allows an agent to continuously interact with the environment to obtain the maximum cumulative reward value and thus learn the optimal strategy. The description of reinforcement learning is usually based on the well-known Markov decision process, as shown in Equation (1).  
\[E=<S,A,P_{sa}^{s^{\prime}},P_{0},R,\gamma>. \tag{1}\]  
Six tuples are defined, where S and A are the set of states and the set of actions, \(P_{sa^{\prime}}^{s^{\prime}}\) denotes the probability of transferring from state \(s\in S\) to state \(s^{{}^{\prime}}\in S\) through a given action \(a\in A\), \(P_{0}\) denotes the initial state distribution, and \(R\) is the reward function, which usually uses the cumulative discounted reward to define the state reward at moment \(t\), \(\gamma\) is the discount factor, as shown in Equation (2).  
\[R=\sum_{i=t}^{T}\gamma^{(i-t)}r(s_{i},a_{i}), \tag{2}\]  
where the discount factor \(\gamma\in[0,1]\), whose smaller value indicates a greater focus on the current reward, and \(r(s_{i},a_{i})\) indicates the value of the reward obtained by choosing action \(a\in A\) in state \(s\in S\). A policy \(\pi\) is a mapping from states to action probability distributions: \(\pi:S\to p(A=a|S)\). The goal of reinforcement learning is to learn an optimal policy \(\pi^{*}\) that maximizes the expected cumulative reward for all states, i.e., the goal of reinforcement learning is as shown in Equation (3).  
\[J=E_{s-p,a-\pi}[R_{0}]. \tag{3}\]  
Under policy \(\pi\), the value of state \(s\) is denoted as \(V^{\pi}(s)\), which represents the cumulative discounted reward brought by executing policy \(\pi\) from state \(s\). It is usually used as an actor in the actor-critic algorithm and is defined as shown in Equation (4).  
\[V^{\pi}(s_{t})=E_{s-p,a-\pi}[R_{t}|s_{t}]. \tag{4}\]  
Similarly, under policy \(\pi\), the value of action a taken for state \(s\) is \(Q^{\pi}(s,a)\), which represents the cumulative discounted reward brought by following policy \(\pi\) after performing action a from state \(s\). It is usually used as a critic in the actor-critic algorithm and is defined as shown in Equation (5).  
\[Q^{\pi}(s_{t},a_{t})=V^{\pi}(s_{t})E_{s-p,a-\pi}[R_{t}|s_{t},a_{t}]. \tag{5}\]  
In the policy gradient algorithm, a good policy is obtained by computing a policy gradient estimate and then using a stochastic gradient descent algorithm. The objective function of the network parameter \(\theta\) update is as shown in Equation (6).  
\[L(\theta)=E[\log\pi(a_{t}|s_{t};\theta)A_{t}(s_{t},a_{t})]. \tag{6}\]  
The update method for network parameters \(\theta\) can be expressed as shown in Equation (7).  
\[\theta_{t+1}=\theta_{t}+\alpha\Delta_{\theta}L(\theta_{t}), \tag{7}\]where \(A_{t}(s_{t},a_{t})\) is the estimate of the advantage function at moment t to measure the gap between the score obtained by the current policy interacting with the environment and the benchmark. The function is defined as shown in Equation (8).  
\[A_{t}(s_{t},a_{t})=Q_{t}(s_{t},a_{t})-V(s_{t}). \tag{8}\]  
In the PPO algorithm, a truncation method is usually used to limit the update of new policies with a loss function defined as shown in Equation (9).  
\[L(\theta)=E[min(r_{r}(\theta)A_{t},clip(r_{r}(\theta)),1-e,1+e)A_{t}], \tag{9}\]  
where the ratio of the old to the new policy is denoted as shown in Equation (10).  
\[r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta old}(a_{t}|s_{t})}. \tag{10}\]  
\(\epsilon\) is the truncation constant, generally, with a value of 0.2; clip is the truncation function; and the value of \(r_{t}(\theta)\) is limited to the range \(1-e\) and \(1+e\). In the process of parameter updating, the PPO algorithm uses truncation to limit the update of new policies, to avoid the problem of too-large differences in policies and improve the generalization of the algorithm.