## 2 Research Methodology
### Deep Reinforcement Learning Method Architecture
A simulation of the urban network is expressed in Figure 3. Furthermore, the Flow tool, which was introduced by UC Berkeley, is the API for DRL agents and custom traffic simulators. The superiority of Flow consists of the easy implementation of numerous road types. In the proposed method's architecture, an environment simulation consists of six parts--namely, initialized simulation, state, action, observation, reward function, and controller--for the urban network, along with nine non-signalized intersections. This method accesses the state information of entire vehicles in an urban network and gives the state's advantage features for the DRL agents to obtain the appropriate policy. More details about these parts are shown in the following. Figure 2: The advanced method’s architecture. Firstly, the initialized simulation presents the initial environment settings--namely, the velocity, position, acceleration, deceleration, trajectories, the number of vehicles, and the PPO hyperparameters. In these trajectories, the SUMO simulator configures the number of points (nodes), the number of links (edges), and the directions of entire vehicles (routes). In addition, the SUMO also controls human-driven actions (e.g., accelerations). The RLlib library is applied to control the actions of AVs based on the MLP policy. Secondly, a state expresses the capable representation of AVs and their surrounding vehicles in terms of the current traffic condition. The state representation that exactly depicts its complex condition consists of multiple parameters--namely, the positions and speeds of the AVs, as well as the positions and relative distances of the leading and following AVs. In the state, the identifications of entire vehicles are obtained in the urban network, and the positions and speeds of entire vehicles are acquired to create the state. The state is illustrated as follows:  
\[S=\left( \begin{array}{c}x_{0}\\ v_{0}\\ v_{l}\\ d_{l}\\ v_{f}\\ d_{f}\\ \end{array} \right), \tag{12}\]  
where \(S\) indicates the specific state, \(x_{0}\) indicates the autonomous vehicle's coordinates, \(v_{0}\) indicates the autonomous vehicle's velocity, \(v_{l}\) indicates the leading autonomous vehicle's velocity, \(v_{f}\) indicates the following autonomous vehicle's velocity, \(d_{l}\) indicates the leading autonomous vehicle's bumper-to-bumper headway, and \(d_{f}\) denotes the following autonomous vehicle's bumper-to-bumper headway. Thirdly, the action that is provided by the OpenAI gym indicates the acceleration actions of the AVs in the simulation environment. The actions are discrete decisions corresponding to the specific autonomous agent. In this work, the bounds of acceleration actions are a variation between maximum deceleration and maximum acceleration. In the action, the specific action command is transferred to the actual control by using the apply_RL_actions function. Fourthly, the observation, which relies upon the output of the SUMO simulator, presents the features of observations, such as the velocity of the autonomous vehicles, the position of the autonomous vehicles, as well as the velocities and bumper-to-bumper  
Figure 3: A simulation of urban mobility (SUMO) simulator for the urban network. headways of the corresponding leading and following autonomous vehicles, as expressed in Figure 4. The observations are used as the updated state to train and choose the best action in the RLlib library. Fifthly, there is the reward function, which is the most critical factor of DRL, to converge the optimal policy. The purpose of the reward is to maximize discount returns. In addition, the higher average speed leads to reducing traffic congestion in an urban network. Thus, the average speed is used as a dominant metric to verify the expected reward. In this study, the DRL agent tried to achieve a higher average velocity while penalizing traffic collisions between vehicles in an urban network. The current speeds of entire vehicles were acquired by the get_speed function and then the average velocity was converted as the return.