# A Novel Deep Reinforcement Learning Approach to Traffic Signal Control with Connected Vehicles +
###### Abstract  
The advent of connected vehicle (CV) technology offers new possibilities for a revolution in future transportation systems. With the availability of real-time traffic data from CVs, it is possible to more effectively optimize traffic signals to reduce congestion, increase fuel efficiency, and enhance road safety. The success of CV-based signal control depends on an accurate and computationally efficient model that accounts for the stochastic and nonlinear nature of the traffic flow. Without the necessity of prior knowledge of the traffic system's model architecture, reinforcement learning (RL) is a promising tool to acquire the control policy through observing the transition of the traffic states. In this paper, we propose a novel data-driven traffic signal control method that leverages the latest in deep learning and reinforcement learning techniques. By incorporating a compressed representation of the traffic states, the proposed method overcomes the limitations of the existing methods in defining the action space to include more practical and flexible signal phases. The simulation results demonstrate the convergence and robust performance of the proposed method against several existing benchmark methods in terms of average vehicle speeds, queue length, wait time, and traffic density. title: 10.3390/app13042750  
realistic driving behavior and multiple objectives, presents a significant challenge and remains an open research area, due to the NP-complete nature of the problem [2; 3] and the \"curse of dimensionality\" associated with an increasing number of vehicles and traffic lights in the network. With the large amounts of traffic data generated by CVs, it is possible to characterize the interactions between vehicles and traffic infrastructure components, thus enabling the development of data-driven traffic control strategies. Non-parametric learning approaches, particularly reinforcement learning (RL), are well suited for characterizing the stochastic and non-linear nature of traffic flow. These techniques allow the signal controller to learn policies by observing the transition of the traffic states, without the need for prior knowledge of the system's model structure [4]. In other words, RL-based signal control approaches eliminate the burden of building complex decision-making models for highly dynamic, nonlinear, stochastic traffic system. However, constructing and training a learning-based controller directly from raw data presents a major challenge. Without proper design of the learning models and training algorithms, the learning-based controller may not be able to effectively learn a control strategy. To tackle the issue of the \"curse of dimensionality\", many researchers have to simplify the training model by restricting the action and state space, which reduces the practicability and optimality of the resulting controllers. In this article, we aim to overcome these limitations by defining the action space in a way that allows for more practical and flexible signal timings, and by restructuring the state space to improve the learning performance. The main contributions of this paper are as follows:  
1. A new traffic signal control framework using deep reinforcement learning (DRL) is proposed, by incorporating a novel convolutional autoencoder network to reduce the dimensionality of the input traffic states. As a result, a concise representation of comprehensive traffic information is obtained and utilized to facilitate the learning of effective SPaT plans. 2. The action space is extended by including both phase duration and cycle length, allowing for increased adaptability to dynamic traffic flow. With the combinatorial action space of multiple phase durations and cycle lengths, our method can effectively handle unbalanced traffic flow with varying traffic volumes. 3. To improve the learning efficiency of the proposed DRL algorithm, several state-of-the-art techniques, such as target network [5], dueling network [6], and experience replay [7] are implemented. 4.