## 2 Research Methodology
### Deep Reinforcement Learning Method Architecture  
RL, which is a subset of machine learning, learns to optimize a policy based on the trial-and-error method. The traditional type of RL is the MDP algorithm, and it is well-suited for full observations. Nevertheless, AVs operate in mixed-traffic conditions in which a dynamic environment consists of human-driven intentions and sensor noise. To address the limitations of MDP, a POMDP is well-suited for partial observations. The agent keeps a probability distribution in the possible states through a set of observations. The objective learning of the POMDP algorithm is to maximize the expected reward regarding the stochastic policy. A POMDP algorithm is a tuple \((B,A,T,R,Z,O,\gamma)\), where \(B\) indicates the set of belief states, \(A\) expresses the set of actions, \(T\) indicates the belief state transition, \(R\) defines the reward function through executing action, \(Z\) indicates the set of observations, \(O\) defines the observable function, and \(\gamma\) indicates the discount factor. DNN, which is a generalization of an artificial neural network (ANN), is able to extract features autonomously with respect to the representations of multi-hidden layers. To enhance the continuous tasks, the MLP, which was proposed by [PERSON] et al. [39], is applied to create the set of acceleration actions based on the updated states (the output of the SUMO simulation over every time step). Additionally, the PPO is used to improve the DRL performance. In this work, the advanced DRL method, which is the integration of RL and MLP, is used to evaluate the efficiency of leading AVs in the urban network along with nine non-signalized intersections. Firstly, the SUMO simulator conducts one time step. Secondly, the RL library (RLlib) trains and generates the set of acceleration policies according to the updated states of the SUMO simulator through the Flow tool. The objective learning of the MLP policy is to optimize the acceleration policy in terms of the set of observations. Figure 1 shows the DRL-based multi-agents in the urban network [35]. Figure 1: An advanced deep reinforcement learning (DRL) architecture for the urban network. RL: reinforcement learning; MLP: multilayer perceptron; PPO: proximal policy optimization; SUMO: simulation of urban mobility. In the DRL method, the policy is designed to communicate between perceptions and actions in a partial environment. Our proposed method refers to the MLP policy with multi-hidden layers as the controller of DRL agents. Based on the observations and states from the SUMO output, the controller iteratively updates parameters to maximize the discounted return. The expected cumulative discounted return is expressed as follows:  
\[\eta(\pi_{0})=\sum_{i=0}^{T}\gamma_{i}r_{i}, \tag{10}\]  
where \(\gamma_{i}\) indicates the discount factor and the \(r\) indicates the reward. The objective learning of the DRL agents is to optimize the stochastic policy, which is shown as follows. \[\theta^{*}:=argmax_{0}\eta(\pi_{0}). \tag{11}\]  
In this work, we apply the DRL method for closed-loop online optimization. This method integrates the SUMO simulator with DRL agents (RLlib library) by adopting the Flow framework. As shown in Figure 2, the advanced method's architecture consists of three parts: First, the SUMO is the environment simulator that executes realistic experiments in time steps. Second, the Flow tool integrates the SUMO environment and the DRL agents. Third, the RLlib library optimizes the cumulative reward based on the SUMO simulator's state. Finally, the simulation resets and iterates the DRL processing [35]. The SUMO simulator can apply DRL through TraCL, which is a python-based application programming interface (API). It can enable DRL agents to integrate with the traffic simulator. In this study, the SUMO simulator is applied to the urban network along with nine non-signalized intersections.