## 2 PPO Algorithm Model
### PPO Optimization Algorithm
The critical gap is a crucial research topic in autonomous vehicle navigation at roundabouts, as it involves determining the appropriate gap for the vehicle to enter. This refers to the minimum distance required for a branching vehicle to safely merge into the circular traffic flow. The critical gap serves as a decision criterion for vehicles approaching the roundabout entrance. Once an acceptable gap is identified, the vehicle accelerates to its desired speed and seamlessly merges with existing traffic while maintaining a safe following distance. The time required for vehicles to enter the intersection is mainly divided into three sections: the first section is the driver's reaction time \(t_{1}\); the second section is the time \(t_{2}\) from moving the foot to pressing the accelerator pedal to the start of the vehicle movement; and the third section is the time \(t_{3}\) required for the vehicle to enter the circular lane after the vehicle starts to move. When the vehicle enters the intersection, it must maintain a certain distance from the vehicle in front, so it is necessary to calculate the on-board clearance  
Figure 2: Optimization algorithm structure diagram. with the vehicle in front, and \(t_{c}\) is the critical clearance. Therefore, the calculation of critical gap is shown in Equation (1):  
\[t_{c}=t_{1}+t_{2}+t_{3}+t_{4} \tag{1}\]  
The entry function is a function established according to the critical reception theory, which represents the number of vehicles allowed to pass by the time headway. The entry function is used to reflect the overall traffic efficiency of the unsignalized roundabout. The formula _g(t)_ of the entry function is:  
\[\text{g(t)}=\left\{ \begin{array}{l}\frac{\text{t}-\text{t}}{t^{\prime}},\ t>t0\\ 0,t\leq t0\end{array} \right\} \tag{2}\]  
\[\text{t}_{0}=\text{t}_{c}+\frac{t_{f}}{2} \tag{3}\]  
Among them, \(\text{t}_{0}\) is the minimum acceptable gap; \(t_{f}\) is the vehicle spacing; \(t_{c}\) is the critical gap. In the process of building and training deep reinforcement learning models, there is a continuous optimization of the decision models. However, during the initial stage of training, the reward value remains close to zero, making it ineffective to repeatedly employ poor historical strategies that hinder training efficiency improvement. Therefore, optimizing the experience pool of the PPO algorithm becomes necessary. The PPO algorithm discards previously collected experiences after each policy update as they become irrelevant to the new policy. One advantage is that this ensures consistency between the policy and value function, enabling the accurate estimation of current policy performance. To store and reuse historical experiences requiring correction, PPO algorithms can utilize experience playback buffers. In the research experiments conducted here, prioritized experience replay was employed which assigns priority based on data value or strength; specifically favoring completed experiment data and high-reward values. This approach allows for more frequent sampling of valuable or advantageous data, thereby enhancing learning efficiency while reducing collisions with autonomous vehicles.