## 4 Methodology
### Proximal Policy Optimization  
PPO is an on-policy reinforcement learning algorithm, which can be used for problems with continuous action spaces [45]. Generally, the on-policy interaction takes the return in each step to estimate the long-term return \(\sum_{t=0}^{K}\gamma^{\tau}r_{t+\tau+1}\), and this will lead to large variance. Accordingly, some studies use an actor-critic framework with an advantage function to estimate the return, but it will increase the bias. Therefore, the paper applies generalized advantage estimation (GAE) to optimize the advantage function to get a trade-off between variance and bias. The movement of the ego vehicle can be optimized when the parametric policy \(\pi(\theta^{p})\) approaches the optimal policy. In this case, the main goal is to find the optimal parameter set \(\theta^{p}_{*}\) of the neural network. That means the objective function should be maximized:  
\[R(\theta^{p})=\mathbb{E}_{\eta\sim\mathcal{D}_{k}}\Big{[}\sum_{\tau=0}^{ \infty}\gamma^{\tau}r_{t+\tau}(s_{t+\tau},a_{t+\tau})\Big{]}, \tag{19}\]  
in which the \(\mathcal{D}_{k}=\{\eta_{i}\}\) is a set of transitions collected from the interaction between the agent and the environment. The transition sequence \(\eta_{i}\) herein refers to a set consisting of \(\{s_{1},a_{1},r_{2},s_{2},a_{2},r_{3},\ldots\}\), and a collection of transitions can be used in training. The training process is shown in Algorithm 1. PPO makes use of a value function approximation (VFA) \(V_{\theta^{p}}\), which also takes an artificial neural network (ANN) with a parameter set to \(\theta^{v}\) to estimate the expected total reward in the future. Over each epoch \(k=1,2,3,\ldots K\), the parameters of the neural networks can be updated. The parameter \(\theta\) is learned in the following way:  
\[\theta_{k+1}=argmax_{\theta}\underset{s,a,\sim\pi_{\theta_{k}}}{E}[L(s,a, \theta_{i},\theta)], \tag{20}\]  
where \(L(s,a,\theta_{k},\theta)\) is given by:  
\[L(s,a,\theta_{k},\theta)=min\big{(}\beta_{k}(\theta)A^{\pi_{\theta_{k}}}(s,a),g\big{(}\varepsilon,A^{\pi_{\theta_{k}}}(s_{t},a_{t})\big{)}\big{)}, \tag{21}\]  
where the definition of \(\beta_{k}(\theta)\) and \(g\big{(}\varepsilon,A^{\pi_{\theta_{k}}}(s_{t},a_{t})\big{)}\) is given by Equations (22) and (23):  
\[\beta_{k}(\theta)=\frac{\pi_{\theta}(a|s)}{\pi_{\theta_{k}}(a|s)}\;, \tag{22}\]  
\[g=\text{clip}\bigg{(}\frac{\pi_{\theta}(a|s)}{\pi_{\theta_{k}}(a|s)},\;1- \varepsilon,1+\varepsilon\bigg{)}A^{\pi_{\theta_{k}}}(s,a)\;. \tag{23}\]  
The parameter \(\varepsilon\) is preset to stipulate an interval \([1-\varepsilon,1+\varepsilon]\) that defines the upper and lower bound of \(\beta_{k}(\theta)\). The purpose of (23) is to restrict the search of \(\theta_{k+1}\) within a predefined range that is close to \(\theta_{k}\). The notation \(A^{\pi_{\theta_{k}}}\in\mathbb{R}\) is the value produced by an advantage function reflecting the value of a selected action in accordance with current policy \(\pi_{\theta k}\) concerning other feasible decisions [46]. Moreover, the clip operation output \((1+\varepsilon)A^{\pi_{\theta_{k}}}\) if \(A^{\pi_{\theta_{k}}}\geq 0\), otherwise \(\text{e }(1-\varepsilon)A^{\pi_{\theta_{k}}}\) will be exported. In this paper, GAE is taken to calculate \(A^{\pi_{\theta_{k}}}\):  
\[A^{\pi_{\theta_{k}}}=\lambda_{k}+(\gamma\chi)\lambda_{k+1}+\cdots+(\gamma\chi )\lambda_{K-1}, \tag{24}\]  
where the discount factor \(\gamma\in[0,\;1]\) and the GAE parameter \(\chi\in[0,\;1]\) are used to balance the estimated biases and variances. The notation \(\lambda_{k}\) is the one-step temporal difference (TD), which is defined as:  
\[\lambda_{k}=\left\{ \begin{array}{cc}\tau_{k+1}-V_{\theta^{p}}(s_{t}),&if\;k =K-1\\ r_{k+1}+\gamma V_{\theta^{p}}(s_{t+1})-V_{\theta^{p}}(s_{t}),&\text{otherwise}\end{array}  \right.\;. \tag{25}\]  
During the training process, both \(\theta^{p}\) and \(\theta^{v}\) can be updated iteratively through the Adam optimization algorithm [47].