# A Novel Deep Reinforcement Learning Approach to Traffic Signal Control with Connected Vehicles +
## 2 Related Work
### Data-Driven Approaches  
The increasing availability of data and computation power has made machine learning-based approaches more and more popular in traffic control. These approaches offer the advantage of being model-free, eliminating the need to build complex mathematical models to describe the traffic states and solve nonlinear optimization problems. For instance, [PERSON] et al. [26] proposed the use of real-time high-precision vehicle trajectory and traffic flow data to better understand and control congestion, and suggested using deep learning algorithms and model predictive control theory to construct a congestion recognition and control optimization model for the urban roadway network. Without a need for prior knowledge of the traffic system, machine learning-based approaches also reduce the likelihood of introducing errors to the estimation of traffic states. Additionally, machine learning approaches are less computationally intensive and have great potential for real-time applications, making them more practical than traditional model-based optimization methods. Moreover, machine learning-based controllers have the ability to continuously learn and adapt to changes in the traffic pattern, leading to improved optimality. There have been efforts to use machine learning techniques to model the complicated relationship between the signal timing plans and traffic delays, as reported in [27]. Overall, machine learning-based approaches hold great potential for addressing the various challenges in the field of traffic signal control. The critical elements in designing a machine learning-based approach for traffic control include (1) capturing the traffic state effectively, (2) selecting the appropriate learning algorithm, (3) defining clear learning objectives, and (4) designing an inclusive action space. For instance, [PERSON] et al. [28] represented the traffic state of a single intersection as image-like grids, using an \(n\times n\times 2\) matrix to denote the position and speed of vehicles within the grids. This matrix was then used as input to a convolutional neural network that calculated expected future rewards for each possible action from the action space, which was adjusting the current signal phase duration. The reward was defined as the reduction in cumulative waiting time between two successive signal cycles. By using the double Q-learning method to maximize the expected reward, the neural network could learn how to reduce the average waiting time of vehicles at an intersection. [PERSON] et al. [29] employed a support vector machine (SVM) algorithm to model the relationship between the current traffic state and the optimal signal timings and an online learning algorithm to adjust the SVM model in real-time. In [30], the traffic state of an intersection was represented using a set of normalized queue lengths in each lane, which were discretized through the application of the \(k\)-means clustering algorithm. To optimize energy consumption and mobility simultaneously, they utilized an RL algorithm with three different reward functions. At each decision time step, the signal controller agent made a decision to either end or continue the current signal phase. Similarly, [31] proposed a reward function with respect to a fixed-time controller. The agent receives positive rewards for better performance than the fixed-time controller, and negative rewards for under-performance. In [32], the traffic state was characterized by a 2-D matrix consisting of the number of stopped vehicles in each direction and the average speed measured in each section. The action space comprised two options: selecting signal phases and adjusting phase offsets. The reward function was a combination of the total volume that passed through the arterial network and the difference in queue lengths between the two different directions. To increase the adaptability of the signal control model, [PERSON] et al.