# A Novel Deep Reinforcement Learning Approach to Traffic Signal Control with Connected Vehicles +
## 3 Methodology
### Scenario and Simulation Environment  
In this paper, we consider a traffic signal control problem for a typical four-way signalized intersection. As shown in Figure 2, the intersection has one left-turn lane, one through lane, and one right-turn lane in each direction. The isolated intersection and traffic are simulated by SUMO [40], a microscopic, space-continuous traffic simulation software, which allows to retrieve details of simulated objects and to adjust their parameters at every time step. The traffic signal at the intersection is managed by a DRL-based actor, which is also referred to as an agent. This agent continuously receives traffic states and a reward signal from the simulation environment and makes decisions based on the current traffic state.  
**Traffic States**: The traffic state is represented by discrete encoding of position and speed information of vehicles around the intersection [41]. The simulated intersection is divided into squared mesh grids with equal length \(c\), which can be represented by an \(N\times N\) matrix. Each grid in the matrix has two values: one binary value that indicates the presence of a vehicle, and the other that stores the speed of the existing vehicle. An example of a \(30\times 30\) traffic state matrix is shown in Figure 3, where the yellow grids denote vehicles and the numbers show their speeds in m/s. Blank grids indicate the absence of at those positions. In real-world implementations, vehicle mobility information can be obtained through a vehicular network or other devices [PERSON] et al. [42].  
Figure 1: Proposed data-driven optimization framework for traffic signal control.  
Figure 2: The simulated intersection.  
**Action Space**: This research focuses on two main types of discrete action spaces for RL-based traffic signal control. The first option involves selecting a signal phase from a fixed set of choices at predetermined time intervals, with the duration of each phase limited to a multiple of the time interval. The second option involves fixing the phase sequence of a signal cycle and adjusting the duration of each phase in the subsequent cycle at the end of the current cycle. As depicted in Figure 4, a typical four-phase signal cycle is considered as comprised of two straight and two left-turn phases. To discretize the action space, a combinatorial approach is used to choose the signal cycle length and phase splits.  
The selection of the signal cycle length plays a crucial role in handling traffic volumes. A long signal cycle increases road capacity and prevents loss of green time that could occur with delayed response to the green light [43]. However, excessively long cycle lengths can result in increased congestion and long waiting queues. There is a trade-off between road capacity and traffic delay to consider when selecting the cycle length. To balance this, the selection of cycle length is limited to {10 s, 20 s, 30 s, 40 s, 50 s, 60 s} to prevent extremely long cycle lengths from slowing down the traffic flow. The available selections for phase duration range from 0 to the maximum cycle length, in increments of 5 s. This results in a total of 1035 possible actions. Additionally, during the last 3 s of each phase, the green lights turn yellow.