# Reinforcement Learning Decision-Making for Autonomous Vehicles Based on Semantic Segmentation
## 2 Materials and Methods
### Reinforcement Learning (RL)
#### 2.2.2 Proximal Policy Optimization Algorithm (PPO)  
In autonomous driving, PPO handled continuous control actions better than DDQN. PPO supported autonomous driving systems in learning decisions more effectively through a stable policy optimization method, thereby improving the safety and robustness of the systems. PPO was a policy gradient-based algorithm that directly optimized the policy function. The policy function \(\pi(a|s)\) represented the probability of taking action \(a\) in state \(s\). In autonomous driving scenarios, PPO learned the optimal probability distribution of selecting different driving actions given the vehicle's state. The PPO algorithm used policy gradient-based optimization to maximize the expected cumulative return of the policy function. Its objective function based on the policy gradient method is:  
\[L^{P\text{G}}(\theta)=\hat{E}_{t}\big{[}\log\pi_{\theta}(a_{t}|s_{t})\hat{A}_ {t}\big{]} \tag{3}\]  
In the formula, \(\theta\) represents the parameters of the current policy, \(\hat{E}_{t}\) denotes the expectation operation at time step \(t\), and \(\log\pi_{\theta}(a_{t}|s_{t})\) is the logarithm of the policy output, which is the probability of an action based on the observed state as input. \(a_{t}\) and \(s_{t}\) represent the action and state, respectively, and \(\hat{A}_{t}\) denotes the advantage function at time step \(t\), which measures the advantage of selecting action \(a_{t}\) in state \(s_{t}\) compared to other actions.  
To prevent the policy updates from being overly large and causing significant deviations in vehicle actions or trajectories, the environment for simulating the PPO algorithm was introduced into the simulation platform. The PPO algorithm incorporated a clipped objective function \(L^{CPI}(\theta)\) to limit the extent of policy updates. The formula is:  
\[L^{CLIP}(\theta)=E_{t}\big{[}\min\big{(}r_{t}(\theta)\hat{A}_{t},clip(r_{t}( \theta),1-\epsilon,1+\epsilon)\hat{A}_{t}\big{)}\big{]} \tag{4}\]  
In the formula, \(r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta\text{old}}(a_{t}|s _{t})}\) is the probability ratio of the new policy to the old policy, which measures the extent of policy changes. \(\epsilon\) is a hyperparameter that represents the maximum extent to which the policy ratio is allowed to deviate from 1. \(clip(r_{t}(\theta),1-\epsilon,1+\epsilon)\) is the clipping function that restricts \(r_{t}(\theta)\) to the interval \([1-\epsilon,1+\epsilon]\).  
PPO combines policy optimization and value function optimization, and the final objective function is:  
\[L^{PPO}(\theta)=E_{t}\bigg{[}L^{CLIP}(\theta)-c_{1}\cdot E_{t}\Big{(}V_{ \theta}(s_{t})-V_{t}^{target}\Big{)}^{2}+c_{2}\cdot H(\pi_{\theta})\bigg{]} \tag{5}\]  
Figure 2: Double Deep Q-Network algorithm block diagram.  
In the formula, \(L^{CLIP}(\theta)\) is the clipped objective function, which optimizes the cumulative reward of the policy. \(\left(V_{\theta}(s_{t})-V_{t}^{target}\right)^{2}\) is the loss of the value function, minimizing the prediction error of the state value. \(H(\pi_{\theta})\) is the entropy regularization term of the policy, which increases the exploration capability of the policy. \(c_{1},c_{2}\) are the weight coefficients, controlling the influence of each part.  
The algorithm framework of PPO is shown in Figure 3. The Actor used the old policy and the new policy to obtain the probability ratio of a specific action for each policy, and the ratio was used in the clipped objective function. The Critic predicted the state value \(V(s_{t})\) for a given state \(s_{t}\), which was used to evaluate the agent's expected long-term returns in that state and calculate the advantage function \(\hat{A}_{t}\) for the clipped objective function. Finally, the clipped objective function provided new policy updates.