## 2 Reinforcement Learning
### General Deep Reinforcement Learning  
Recent publications [11; 13] propose an integration of deep learning methodologies with RL under the term DRL. The main purpose of this integration is the policy function approximation by the use of DNN based models. In early studies [14], DNNs models for DRL are represented by a deep Stacked-AutoEncoders (DeepSAE), while the recent models are based on Convolution layers (Conv), which are stacked along with the flatten one and several Fully Connected (FC) layers. The flatten layer converts two-dimensional features computed by convolution layers into the one-dimensional vector suitable as the input for FC layers. Those DNN models are known as the Convolutional Neural Networks (CNN) since they contain convolution layers that conduct feature extraction from the image-like inputs by simulating biological convolution processes in human visual cortex [15]. Additional Convolution layers in the CNN model allows the development of features from the features extracted in previous layers by the process of sub-sampling, transforming low-level features of the data to high-level ones. This process can potentially increase the overall CNN performance [16]. The FC layers in CNN have the role to provide a structural condition classification and eventually assess the probability for the execution of each possible action according to the passed inputs to the CNN. The configuration of the CNN model used in DRL does not contain pooling layers which are mandatory for example in image classification. Those layers are not needed in CNN setup for DRL since the exact positions of traffic fluctuation in the image-like inputs are essential for traffic control. The CNN model applied in DRL for traffic control along the all mentioned type of layers can contain Long Short-Term Memory (LSTM) layers based on Recurrent Neural Network (RNN) architecture. They have the ability to process data sequentially and keep hidden state through time. The LSTM layers are used in CNN for traffic control in [17], while the Residual Networks (ResNet) as the unique setup for CNN are used in [6]. The ResNet architecture for CNN models introduces a skip connection among a group of layers (ResNet blocks) in order to jump over some layers and reuse activations from a previous layer from which jump is made until the adjacent layer learn its weights. The addition of LSTM layers and the introduction of ResNet architecture in the CNN models is proposed in order to tackle _vanishing gradient_ problem and induce a more stable learning process. Furthermore, it is possible to introduce the Batch Normalization (BN) process in the form of an added Normalization layer between layers in DNN models. The mentioned normalization has to be done separately for each dimension of the input layer in order to adjust and scale its activations. The application of BN can improve the stability and speed of any DNN model, as proposed in [18]. The integration of complex DNN models with the RL can enhance its learning capacity in order to tackle complex tasks such as control over the stochastic traffic flows at the numerous intersections [9]. In general, the following RL approaches are extended by the use of DNN models for the sake of traffic control: value-based, policy-based, and Actor-Critic methods [19]. Deep Q-learning (DQL) is an augmentation of the vale-based Q-learning algorithm where \(Q^{*}(s_{t},a_{t})\) is approximated by the use of DNN model with weights \(\theta\). This setup is described as:  
\[Q(s_{t},a_{t};\theta)=Q^{*}(s_{t},a_{t}) \tag{2}\]  
The DNN digest input states arranged as the image-like matrix or one-dimensional vectors which may contain several channels if it is needed. If the input state contains several channels then it can be considered as the tensor--represented by the stack of several image-like matrices. Usually, each mentioned matrix is populated with different types of traffic parameters describing the same traffic scenario in the current action interval.