# A Novel Deep Reinforcement Learning Approach to Traffic Signal Control with Connected Vehicles +
## 2 Related Work
### Data-Driven Approaches
[33] proposed a graph-based method that depicts the traffic state as graph-structured data, which were then input into a graph neural network to train the signal control policy. The study focused on an isolated intersection with only straight traffic flows and thus the SPaT was made up of two green-red phases and two yellow-red phases, and the action was defined as the ratio of the green time over a fixed signal cycle. To enhance the ability of the RL algorithm to generalize, [PERSON] et al. [34] incorporated prior traffic knowledge. They used a fully-connected network to classify the intersection's demand pattern, and combined the results with outputs from a convolutional network to produce joint Q-value approximations. The intersection state was represented by a discrete encoding matrix consisting of vehicle position, speed, and signal phase. The reward structure was a combination of the number of stopped vehicles and passing vehicles, phase changes, and total vehicle waiting time. In the scenario of network-wide traffic control, the application of a centralized RL method faces many challenges, such as an exponential increase in the dimension of the action space as the number of traffic lights increases, making it difficult to find an effective joint control policy. To address this issue, some studies, such as [35; 36], trained each intersection as an individual agent based on the local traffic states and information from neighboring intersections, while the overall system state and performance were determined by the joint actions of all intersections. However, the stationary reward distribution and environment dynamics are required by a [PERSON] decision process. It is difficult for the agent to converge to a stationary policy when its rewards are influenced by neighboring intersections. To resolve this issue, [PERSON]. [37] implemented a knowledge sharing mechanism to enhance cooperation and collaboration among traffic signals, where the \"knowledge\"was a collective representation of the traffic environment collected by all agents and used for learning individual policies. Similarly, to improve the convergence of the multi-agent RL algorithm, [PERSON]. [38] combined a group of traffic signals into a single agent through a \(k\)-nearest-neighbor-based joint state representation. In summary, the studies reviewed above have highlighted the potential of data-driven approaches in developing signal control strategies that outperform conventional controllers such as fix-timing and actuated controllers. However, the scenarios considered were significantly simplified. Some studies limited the number of lanes and traffic directions, resulting in a smaller state space, while others limited the action space by either selecting a continuous space with fixed phase sequence and phase split action, or choosing a discrete action space with limited options for phases and duration or fixed cycle/phase length with phase switching as the only actions. These simplifications were made to reduce the complexity of the data-driven models, indicating that the existing methods face challenges in learning a practical and truly optimal signal control strategy.