## 4 Methodology
### MDP Model  
Similar to the RL-based ATSC, the MDP model is a prerequisite for applying the RL algorithm. Based on the assumptions we made in the previous section, we can specify the MDP according to the optimal control problem we build to make a mapping. The state, action, and reward of the MDP is defined as follows:  
1. _State_: The ego vehicle selects action in terms of the state, so the definition of the state is crucial. The state of this problem should include its dynamics, the state of its surrounding vehicles, and the state of the traffic lights. Therefore, a vector with multi-elements is built to serve as the state of the ego vehicle: \[s^{v}(t)=\left[d^{s}(t),v(t),\Delta d(t),\Delta v(t),\Delta a(t),\tilde{s}_{j },t_{j}^{c},s_{j}(t)\right]^{T},\] (14) where \(d^{s}(t)\) measures the distance from the vehicle to the stop line, whereas the agent needs to know how far to enter the intersection; \(\Delta d(t),\Delta v(t)\), and \(\Delta a(t)\) denotes the distance difference, speed difference, and acceleration difference between the potential leading vehicle and the ego vehicle. This may help the ego vehicle keep a safe distance between itself and the front vehicle or prevent itself from entering the tail of a queue. More precisely,  
Figure 5: The state description of each kind of agent. the front vehicle can be the tail of the queue, so the ego vehicle may enter the queue and keep the stop-go movement, which may increase the energy consumption significantly. However, a leading vehicle does not always exist, so we set the judgment threshold \(\rho=300\) m to measure the relative distance between the front vehicle and the ego one. Let the absolute position of the front vehicle be \(d^{f}(t)\), and the value of the ego vehicle be \(d^{e}(t)\), then we have:  
\[\Delta d=\left\{ \begin{array}{cc}d^{f}(t)-d^{e}(t),&if\ d^{f}(t)-d^{e}(t)< \varrho\\ \rho,&\text{otherwise}\end{array} \right., \tag{15}\]  
when \(d^{f}(t)-d^{e}(t)<\rho\), the value of \(\Delta v(t)\) and \(\Delta a(t)\) can be calculated in the same way. Otherwise, the two values will be set to the default values \(\Delta v_{d}\) and \(\Delta a_{d}\). Moreover, \(\xi_{j}\) in (9) is a boolean value that represents whether the lane where the vehicle is located is opened with the green light in the current phase; \(t_{j}^{c}\) denotes the time to act for the first downstream traffic signal agent. Finally, \(s_{j}(t)\) is the state of traffic light \(j\) (i.e., the traffic light can share its state with the ego vehicle through V2I communication). The three groups of the parameters here consist of the state related to the traffic signal agent, which may help the vehicle agent understand the performance of the traffic lights. This state definition applies regardless of whether the vehicle is on a road segment or inside an intersection. Hence, the framework can be used for a multi-intersection scenario naturally. _Action_: The action is consistent with the control item in the optimal control problem, that is, the acceleration of the ego vehicle. The action space should satisfy Equation (10) to generate an effective value. Meanwhile, the speed of the agent should also satisfy (9), which means that the speed change of the vehicle is denoted as:  
\[v(t)=max(min(v_{max},v(t-1)+\delta_{t}),\ 0), \tag{16}\]  
where \(a_{t}\) is the output of the DRL algorithm. However, the value \(a_{t}\) cannot be taken as the final acceleration to impose on the vehicle because \((11)\) should also be accomplished. Accordingly, the amended acceleration is:  
\[a_{t}=min\left(\delta_{t},\ a^{CF}(t)\right), \tag{17}\]  
where the \(a^{CF}(t)\) is determined by IDM [44] in this paper, and the parameters of the car-following model are set to default values in SUMO to enhance traffic safety. _Reward_: The definition of the reward function should be similar to the cost function (6) to get the same control effect.