## 3 Method
### PPO and SAC Algorithms  
The PPO algorithm is one of the Policy Gradient (PG) techniques introduced by the OpenAI team in 2017 [26]. The agent alternately performs sampling data through interaction with the environment and optimizing the Surrogate Objective Function using Stochastic Gradient Ascent [27]. Unlike the conventional method of updating data every time they are sampled, the PPO algorithm uses a new objective function that enables multiple epochs of mini-batch updates. In particular, in autonomous driving, the PPO algorithm shows good performance compared to other AI reinforcement-learning algorithms [28]. The PPO-Clip pseudocode of the PPO algorithm is shown in Algorithm 1. The algorithm declares the initial parameters \(\theta\_0\) and initial value function parameters [29]. The policy of the pseudocode collects trajectories with \(\pi\_x=\pi(\theta\_x)\), calculates the advantage with the rewards-to-go and value function, and updates the policy by maximizing the PPO-Clip. The PPO algorithm learns a value function using stochastic gradient ascent such as Adam and regression through mean-squared error. It also computes an expression, usually using the gradient descent algorithm and ending with the last iteration [30]. ```
1:Input: initial policy parameters \(\theta_{0_{i}}\) initial value function parameters \(\phi_{0}\)
2:For\(k=0,1,2,\dots\)do
3:Collect set of trajectories \(D_{\xi}=\{\tau_{i}\}\) by running policy \(\pi_{\kappa}=\pi(\theta_{\kappa})\) in the environment
4:Computer rewards-to-go \(\widehat{R_{t}}\)
5:Compute advantage estimates, \(\widehat{A_{t}}\) (using any method of advantage estimation) based on the current value function \(V_{\phi_{\kappa}}\)
6:Update the policy by maximizing the PPO-Clip objective: d/d. \[\theta_{\kappa+1}=arg\max_{\theta}\frac{1}{|D_{\kappa}|\T}\sum_{t\in D_{\xi} }\sum_{t=0}^{T}min\left(\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{\kappa }}(a_{t}|s_{t})}\right)A^{\pi_{0_{i}}}(s_{t},a_{t}),g\big{(}\epsilon,A^{\pi_{0 _{i}}}(s_{t},a_{t})\big{)},\]

Typically via stochastic gradient ascent with Adam. 7:Fit value function by regression on mean-squared error:

\[\Phi_{\kappa+1}=arg\max_{\theta}\frac{1}{|D_{\kappa}|\T}\sum_{t\in D_{\xi}} \sum_{t=0}^{T}\left(V_{\phi}(s_{t})-\widehat{R_{t}}\right)^{2},\]

Typically via some gradient descent algorithm. 8:End for
```  
**Algorithm 1** PPO-Clip  
Similar to the PPO algorithm, the Soft Actor\(-\)Critic (SAC) algorithm is a policy-based algorithm of policy optimization that solves the exploration problem of reinforcement learning to some extent, inherits the actor\(-\)critic structure, and establishes the actor's skill improvement using KL-Divergence instead of Policy Gradient [31]. The characteristics of the SAC algorithm are: stable learning; easy-to-obtain efficient samples, suitable for continuous action space and high-dimensional space problems; and use as a baseline for model pre-off policy [32; 33]. Algorithm 2 shows the reset of the parameters at the start of the SAC algorithm, resetting the target network weight of the initialized parameters as shown in the following equation, and also resetting the pooling to be repeated. In addition, it expresses the policy for the default behavior as a whole, performs the default conversion in the environment, and saves the conversion of the iteration pooling. The following iteration updates the Q function's parameters and policy weights, adjusts each attribute value, updates the target network weight, and finishes the iteration. Finally, the resulting value becomes the parameter.