## 2 Methods  
### Deep Reinforcement Learning (Deep RL)  
Reinforcement learning (RL) is a subarea of machine learning and is concerned with how agents interact with an environment and learn to take actions that maximize their cumulative reward. The typical form of the RL algorithm is a Markov decision process (MDP), which is a strong framework used to determine a proper action given a full set of observations [9]. An MDP is a tuple (\(S\), \(A\), \(P\), \(R\), \(\rho_{0}\), \(\gamma\), \(T\)), where \(S\) and \(A\) are states and actions of a participant, respectively; \(P(S^{\prime}\), \(S\), \(a)\) defines a probability for transition; \(R(a,S)\) defines the reward according to the selected action; \(\rho_{0}\) defines the initial state distribution; \(\gamma\) defines the discount factor, which ranges from 0 to 1; and \(T\) denotes the time horizon. However, automated vehicles maneuver in an uncertain environment that contains inaccuracy, intentions, and sensor noise. To solve this problem, a partially observable MDP (POMDP) was proposed that employs two more components, namely \(O\), which defines the set of observations, and \(Z\), which is an observation function. An objective learning agent in RL optimizes the policy \(\pi\) to maximize their expected cumulative discounted reward over some number of time steps.  
A deep neural network (DNN) has the ability to automatically perform feature extraction due to multiple hidden layers of representations. For continuous controllers, artificial neural networks (ANNs) are commonly used methods that employ multiple hidden layers to represent complex functions. In this work, we apply an MLP to generate a set of outputs (policy) from a set of inputs (states and observations). In addition, we apply a PPO based on a gradient descent optimization method to enhance the performance of the DNN. Our proposed deep RL framework, which fuses a MLP and RL, is designed to consider the effectiveness of AVs at a non-signalized intersection. First, the SUMO simulator executes one simulation step. Second, the Flow framework sends information on the SUMO simulator's state to the RL library. Then, the RL library (RLlib) computes the appropriate action according to SUMO simulator's state through MLP. The MLP policy is applied to maximize the cumulative reward for the RL algorithm based on the traffic data. Finally, the simulation resets and iterates the RL process. Figure 1 presents the deep reinforcement learning architecture in the context of a non-signalized intersection.  
Importantly, a 'policy' refers to a blueprint of the communication between perceptions and actions in an environment. In other words, a policy is similar to a controller of a traffic simulation. In this work, the controller is an MLPpolicy with multiple hidden layers. The parameters of the controller are iteratively updated by using the MLPpolicy to maximize the cumulative reward based on the traffic  
Figure 1: The deep reinforcement learning architecture in the context of a non-signalized intersection.  
data sampled from the SUMO simulator. The main goal of the agent is to learn how to optimize a stochastic policy as follows.  
\[\theta^{*}:=argmax_{\theta\eta}(\pi_{0}) \tag{1}\]  
where \(\eta(\pi_{0})\) is the expected cumulative discounted reward, which is calculated by the discount factor (\(\gamma_{i}\)) and the reward (\(r\)).  
\[\eta(\pi_{0})=\sum\
olimits_{i=0}^{T}\gamma_{i}r_{i} \tag{2}\]