## 2 Methods
### Proposed Method Architecture  
More recently, the DNN algorithm enables automatic feature extraction through multiple layers. This means that different hidden layers can show different transformations according to their input. For instance, the artificial neural network (ANN) was designed to obtain a reliable performance from more data. The ANN paradigm tries to achieve the desired output through the input data. In this study, the MLP was designed to generate a \begin{table}
\begin{tabular}{l c} \hline \hline
**Parameters** & **Value** \\ \hline Desired velocity & 15 m/s \\ Time headway & 1.0 s \\ Minimum headway & 2.0 m \\ Acceleration exponent & 4.0 \\ Acceleration & 1.0 m/s\({}^{2}\) \\ Comfortable acceleration & 1.5 m/s\({}^{2}\) \\ Desired velocity & 15 m/s \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The typical parameters for the IDM algorithm at a non-signalized intersection. set of acceleration policies related to updated states converted from the SUMO simulator over time. The MLP method was suggested by [PERSON] et al. [28]. Additionally, the backpropagation process was also used to optimize the policy through the PPO algorithm. The PPO algorithm relies upon policy gradient methods to estimate the parameterized policy function to improve the convergence under nonlinear functions. The policy gradient is expressed as follows:  
\[g=E[\
abla_{\theta}log\pi_{\theta}(a_{t}|s_{t})A^{\pi_{t},\gamma}(a_{t}|s_{t})], \tag{6}\]  
where \(E[. ]\) refers to the expectation operator, \(\pi_{\theta}\) refers to the stochastic policy, \(log\pi_{\theta}\) refers to the probabilities of the stochastic policy, and \(A^{\pi_{t},\gamma}\) refers to the advantage function. The PPO algorithm, which was supported by the Flow tool through the RLlib library, ensures that the new policy is not too dissimilar from the old policy [29]. The surrogate loss function is applied to update the policy using the Gaussian distribution. The PPO paradigm consists of clipped PPO and the adaptive KL penalty. In this study, the clipped PPO, which performs well under complicated conditions, was implemented to obtain a new objective function through stochastic gradient descent (SGD). \[L_{\theta_{k}}^{CLIP}=E\Big{[}\sum_{t=0}^{T}\big{[}min(r_{t}(\theta))A_{t}^{ \pi_{k}},clip(r_{t}(\theta),1\ -\ \varepsilon,1+\varepsilon)A_{t}^{\pi_{k}}\big{]}\Big{]}, \tag{7}\]  
where \(\theta\) refers to the policy parameter and \(\varepsilon\) refers to the clipping threshold. This algorithm considers the probability ratio between the new policy and the old policy. When this ratio is outside the variation between \((1-\varepsilon)\) and \((1+\varepsilon)\), the advantage function will be cut. Through the integration of the RL and MLP algorithms, the DRL method was applied to verify the efficiency of comprehensive AD maneuvers at a non-signalized intersection with different MPRs and real traffic volumes. This study focused on a closed-loop online optimization through a simulation environment (SUMO), the Flow tool, and DRL agents. Figure 1 expresses the research proposal flow. The DRL algorithm attempts to maximize the expected cumulative discounted reward, which is defined as follows:  
\[\theta*:=argmax_{\theta}\eta(\pi_{0}), \tag{8}\]  
where \(\eta(\pi_{0})\) expresses the expected cumulative discounted reward, defined as follows:  
\[\eta(\pi_{0})=\sum_{i=0}^{T}\gamma_{i}r_{i}, \tag{9}\]  
where \(\gamma_{i}\) expresses the discount factor and \(r_{i}\) expresses the reward. The SUMO simulator, which is an open-source microscopic simulation, uses a traffic control interface (TraCI) to enable the DRL method through Python programming. Figure 2 expresses a typical simulation in SUMO. In addition, the Flow tool, which was suggested by UC Berkeley, is a bridge between the simulation environment and DRL agents. The advantage of the Flow tool is that it is easy to implement across various types of roads. This process explores the important features of state information to generate the optimal acceleration policy.