## 2 Methods
### Policy Optimization  
Policy gradient methods attempt to compute an estimator of a parameterized policy function using a gradient descent algorithm rather than an action-value or a state-value function. Thus, they avoid the convergence problems that occur with estimation functions due to non-linear approximation and partial observation. We applied the MLP policy to optimize the control policy directly in the simulation \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameters** & **Value** \\ \hline Desired speed (m/s) & 15 \\ Time gap (s) & 1.0 \\ Minimum gap (m) & 2.0 \\ Acceleration exponent & 4.0 \\ Acceleration (m/s\({}^{2}\)) & 1.0 \\ Comfortable acceleration (m/s\({}^{2}\)) & 1.5 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Typical parameters of an intelligent driver model (IDM) controller for city traffic. of the non-signalized intersection. The policy gradient laws, which are based on the expectation over the probabilities of the policy actions (\(\log\pi_{\theta}\)) and an estimate of the advantage function at time step t (\(\hat{A_{t}}\)), are expressed as follows. \[\hat{g}=\hat{E}_{t}\Big{[}\
abla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})\hat{A}_{ t}\Big{]} \tag{6}\]  
where \(\hat{E}_{t}[. ]\) is the expectation operator over a finite batch of samples, \(\pi_{\theta}\) indicates a stochastic policy, \(\hat{A}_{t}\) is defined by the discounted sum of rewards and a baseline estimate, and \(a_{t}\) and \(s_{t}\) express the action and state at time step \(t\), respectively. PPO, which was proposed by [PERSON] et al. [PERSON] et al. (2016), is a simple TRPO that is provided by the RLlib library. In other words, PPO's objective is the same as that of TRPO, which uses a trust region constraint to force the policy update to ensure that the new policy is not too far away from the old policy. There are two types of PPO: adaptive Kullback-Leibler (KL) penalty and clipped objective. The PPO generates policy updates by adopting a surrogate loss function. This process avoids a reduction in performance during the training process. The surrogate object (\(J^{CPI}\)) is described as follows. \[J^{CPI}(\theta)=\hat{E}_{t}\bigg{[}\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{ \theta_{old}}(a_{t}|s_{t})}\hat{A}_{t}\bigg{]}=\hat{E}_{t}\Big{[}r_{t}(\theta )\hat{A}_{t}\Big{]} \tag{7}\]  
where \(\pi_{\theta_{old}}\) indicates a policy parameter before update, \(\pi_{\theta}\) indicates a policy parameter after update, and \(r_{t}(\theta)\) indicates the probability ratio. For continuous actions, the PPO's policy output is a parameter of the Gaussian distribution for each action. The policy then generates a continuous output based on these distributions. In this work, PPO with an adaptive KL penalty is used to optimize the KL-penalized objective by using minibatch stochastic gradient descent (SGD) as follows. \[\underset{\theta}{\text{maximize}}\ \hat{E}_{t}\Big{[}r_{t}(\theta)\hat{A}_{t} \Big{]}-\beta\hat{E}_{t}\Big{[}KL\Big{[}\pi_{\theta_{old}}(a_{t}|s_{t}),\pi_{ \theta}(a_{t}|s_{t})\Big{]}\Big{]} \tag{8}\]  
\[\text{Subject to}\ \hat{E}_{t}\Big{[}KL\Big{[}\pi_{\theta_{old}}(a_{t}|s_{t}),\pi_{ \theta}(a_{t}|s_{t})\Big{]}\Big{]}\leq\delta \tag{9}\]  
where \(\beta\) is the weight control coefficient that is updated after every policy update. If the current KL divergence is greater than the target KL divergence, we increase \(\beta\). Similarly, if the current KL divergence is less than the target KL divergence, we decrease \(\beta\). In the PPO algorithm, first, the current policy interacts with the environment to generate the episode sequences. Next, the advantage function is estimated using the baseline estimate for the state value. Finally, we collect all experiences and execute the gradient descent algorithm over the policy network. The complete PPO with an adaptive KL penalty algorithm is presented in pseudocode in Algorithm 1, shown below [PERSON] et al. (2016):  
Importantly, the PPO hyperparameters provide a robust approach to enhancing the effectiveness of RL at various tasks.