# A Novel Deep Reinforcement Learning Approach to Traffic Signal Control with Connected Vehicles +
## 2 Related Work
### Optimization-Based Approaches  
Optimization-based approaches assume that the traffic model is known and the future traffic flow states could be predicted accordingly. Next, certain optimization problems are formulated and solved for the optimal SPaT control plans. The objectives of these optimization problems are usually to minimize traffic performance measures, such as traffic delay and queue length, which are estimated on the basis of predicted vehicle arrivals [8]. However, this approach requires accurate predictions of future traffic states, which can be challenging due to the complexity of the optimization problem that involves traffic flow models and couples with SPaT data. Therefore, the key challenges in CV-based traffic controls are to predict the future traffic states accurately, coordinate multiple intersections effectively by accounting for the conflicts of traffic flows, and efficiently solve the underlying large-scale optimization problem [8]. These challenges have led to the development of three different groups of optimization methods: centralized, decentralized, and hierarchical approaches. In comparison to conventional signal control methods, such as adaptive control and coordinated control, the biggest challenge in implementing optimization-based methods is the high complexity of optimization models. To address this issue, centralized approaches reformulate the optimization problem by reducing the number of variables. For instance, in [9], individual vehicles were grouped into pseudo-platoons based on the headways between them, and a mixed-integer linear program (MILP) was utilized to determine the optimal signal phase sequence and phase initialization in real-time using platoon request data and traffic controller status. This work also introduced a dynamic arterial coordination strategy to promote traffic progression by taking into account platoon queue delay, signal delay in the current intersection, and possible delay at downstream intersections. In [10], a real-time adaptive phase allocation algorithm was proposed that utilizes dynamic programming and optimization techniques to allocate signal phase sequences and duration based on predicted vehicle arrivals. [PERSON] et al. [11] adopted an interactive grid search method to solve an optimization problem, considering accumulated fuel consumption and travel time as the cost function, to determine the optimal traffic light timing of for each cycle at an intersection. [PERSON] and [PERSON] [12] proposed a new optimization method based on an improved genetic algorithm, which was compared with the Webster algorithm and the traditional genetic algorithm. The average vehicle delay was used as the control objective, and the green signal ratio and cycle time were used as the control variables. [PERSON] and [PERSON] [13] used a cell transmission model [14] to categorize the traffic network into cells and groups for higher-level representation and then formulated an MILP to maximize network throughput, which was solved using the Benders decomposition technique [15]. [PERSON] et al. [16] formulated an optimization problem to minimize network-level traffic delay, considering the energy consumption as a constraint, and solved the resulting non-convex problem using a stochastic gradient approximation algorithm. In [PERSON] et al. [17], a linear dynamic traffic system model was built for a large-scale traffic network and a linear-quadratic regulator was applied to minimize both traffic delay and control-input changes, allowing for an online update of the traffic model to be adaptive to signal control outcomes. Decentralized approaches aim to simplify the model and lower the computational cost of the traffic control problem by utilizing distributed control and optimization techniques. These methods optimize objective functions for each intersection individually and disregard coordination among neighboring intersections, leading to sub-optimal, local solutions instead of globally optimal solutions.