# A Novel Deep Reinforcement Learning Approach to Traffic Signal Control with Connected Vehicles +
## 3 Methodology
### Compressed Representation of Traffic States  
It is convenient to use the position and speed information of vehicles to build a traffic state matrix for input to the DRL training algorithm. However, the large space of the resulting traffic states makes it difficult for the DRL algorithm to identify a direct relationship between the traffic state and the signal control action. To address this issue, finding an appropriate traffic state representation is crucial. In this study, an autoencoder neural network is used to represent the complex traffic states of the whole intersection as a concise representation Baldi [44]. Having a state representation that contains rich information is vital for the control agent to make informed decisions, without being affected by the \"curse of dimensionality\". To achieve this, the dimension of state representation must be reduced while preserving as much information as possible. Additionally, the compressed state representation allows the underlying traffic pattern to be extracted as features by the autoencoder. Although it is hard to know the exact features learned by the autoencoder, feature extraction has been shown to be a useful approach in improving reinforcement learning in various applications [45]. Figure 3: An example of traffic state matrix. Autoencoder is a type of neural network that can learn a compact representation of the input data [46]. Convolutional neural networks, such as the Visual Geometry Group (VGG) neural networks [47], can extract inherent features from the spatial information in the input data. By organizing a convolutional neural network into an encoder-decoder architecture, the network can encode a static traffic state into a fixed-length vector, serving as input to the reinforcement learning model. The proposed convolutional autoencoder (CAE) network, shown in Figure 5, has a mirror structure of two functional components: the encoder and decoder. The task of the CAE is to reconstruct the original input to its output through a designated bottleneck layer \(\mathbf{h}\). As illustrated in the figure, the input and output of the CAE are the traffic state matrices with a shape of \(64\times 64\times 2\). The encoder part consists of two pairs of convolution-pooling layers followed by two fully-connected layers, while the decoder is the reverse of the encoder structure. The notation numbers define the dimensions of the outputs at each respective layer. The size of hidden layer \(\mathbf{h}\) can be determined through training experiments. Upon completion of training, the encoder network will be used as the state representation compressor to generate the input vector to the DRL neural network. The proposed CAE was implemented and trained using Tensorflow [48]. The optimization process employs the [PERSON] algorithm [49] with mean squared error (MSE) as the cost function. The hyper-parameters, such as number of filters and neurons in the CAE, were determined through cross-validation training experiments. The size of the reduced representation vector of the traffic states was selected as 8, resulting in a compression ratio of 1024 to 1. In addition, the input state matrix was normalized by scaling the vehicle speed to the range of 0 to 1 based on the maximum allowable speed of the road. Figure 4: A typical signal cycle with four phases. The proposed method was evaluated using a validation set, which constituted 10% of the total available dataset. The simulation of the entire traffic signal control system was carried out in SUMO, and the training dataset was generated by continuously running simulations in SUMO. All vehicles were randomly initialized with a specified flow rate. An example training session is shown in Figure 6 using a dataset of one million samples. The minimum reconstruction errors in this example are \(5.07\times 10^{-4}\) for the training error and \(6.54\times 10^{-4}\) for the validation error.