## 2 PPO Algorithm Model  
### PPO Optimization Algorithm  
Deep reinforcement learning (DRL), an artificial intelligence method that combines deep learning and reinforcement learning, exhibits a cognitive resemblance to human decision-making processes. Deep learning, as a multi-layer neural network, possesses exceptional perceptual capabilities in machine learning. Reinforcement learning involves the interaction between an agent and its environment, where the agent learns from feedback information in the form of reward values to optimize its strategy for maximizing rewards. By integrating deep learning's ability for end-to-end nonlinear fitting with reinforcement learning's problem-solving capacity, deep reinforcement learning algorithms are formed. The actor-critic algorithm framework is widely employed in these algorithms due to its combination of value function estimation and strategy search methods, effectively addressing both high bias and high variance issues associated with traditional approaches. This framework enables the design of agents capable of generating strategies while simultaneously evaluating their effectiveness through value functions. The utilization of deep networks within deep reinforcement learning facilitates the accurate fitting of output value functions and strategies over iterative updates towards optimal performance (Figure 1). PPO is a strategy optimization algorithm for reinforcement learning, which was proposed and published by OpenAI researchers and is widely used in various reinforcement learning tasks. The PPO algorithm utilizes strategy gradient-based optimization to maximize the expected cumulative return of the strategy function. In comparison with other strategy optimization algorithms, PPO ensures improved stability and convergence speed through constraint optimization, thereby controlling the update amplitude within each iteration. The PPO algorithm is enhanced by incorporating the CoordConv network and a dynamic model multi-objective reward mechanism. The CoordConv network establishes an aerial perspective of the vehicle to acquire spatial perception information, capturing low-dimensional details such as road conditions and surrounding vehicles. In autonomous driving simulations, data processing plays a crucial role. By utilizing an aerial view, the data processing procedure can be simplified, facilitating easier analysis and enhancing simulation efficiency and accuracy. The global overview provided by the aerial view enables better comprehension of the surrounding environment and obstacles for self-driving systems, extracting features from lower-dimensional data. These feature data are stored, with the CoordConv network employed to extract relevant information from the aerial view's features such as vehicle speed, distance to preceding vehicles, and their speeds--all serving as inputs for decision-making in the algorithm [22; 23]. This approach effectively reduces dimensionality in state feature representation within the algorithm. The optimal reward function model is established based on a multi-objective reward mechanism to optimize the decision-making strategy of autonomous vehicles in unsignalized roundabout scenarios, which differ from normal roads. The multi-objective reward  
Figure 1: Actor–critic algorithm framework. mechanism encompasses lane offset, lateral speed, longitudinal speed, vehicle collision, brake and throttle rewards, workshop distance reward, and critical clearance reward to enhance the driving efficiency, safety, and stability of autonomous vehicles. Figure 2 illustrates the structural diagram of PPO algorithm. The gap acceptance theory dictates that a circular vehicle can enter the roundabout when its headway exceeds a critical gap, while it must wait if the headway is less than this threshold. By calculating the maximum number of vehicles that can enter per unit of time, capacity calculation models based on gap acceptance theory accurately reflect the traffic flow characteristics of roundabouts.