## 3 Adaptive Traffic Light Signal Control Based on DRL
### Traffic Signal Control on a Larger Scale  
The poor scalability of DRL frameworks dedicated to ATSC originates from their RL roots. The main problem is an extremely large discrete action space which emerges in the case when one wants to gain control over the larger number of intersections. Training a centralized DRL agent for control over several signalized intersection is still infeasible for the large scale traffic signal problems, and finding a feasible solution to that problem is still an open area for research [40; 41]. In the case of large scale traffic networks, it is computationally infeasible to train one DNN model for each signalized intersection. Furthermore, it is very unlikely that the resulting policy would be globally optimal without the coordination between local learning/optimization processes [34]. The various Shallow-ANN (SANN) models presented in [42] were the first non-DRL approaches used to address the aforementioned problem. Those models were based on a lower number of hidden layers and they used only macroscopic traffic variables such as a number of vehicles at each bound of the controlled intersection as its inputs. Each SANN model was assigned to one intersection. Those models had two shortcomings. The first addresses the need for manual identification of the general traffic features and arrangement of those features within fixed time frames. The second implies that the weights are updated according to the most usual daily traffic patterns. Thus, every sudden fluctuation in the traffic demand which diverges from those patterns can reduce its efficiency. The introduction of coordinated Multi-Agent RL (MARL) framework was the robust approach for holistic control over several intersection compared to previously used methodologies [43]. The MARL frameworks are mostly based on the Q-learning algorithm. MARL frameworks can be designed based on: (1) decentralized architecture that is based on the dynamical partitioning of the traffic networks into smaller regions where each RL agent control one of those regions; (2) transfer learning techniques; and (3) allocation of one simple RL agent for each intersection. A problem with decentralized architecture is the assumption that global network-level action-value function is simply a linear summation of regional action-value functions [34]. One of the first approaches for MARL design is described in [44] where the migration from single- to multi-intersection control was made by the use of transfer planning and the _max-plus_ coordination algorithm for inter-agent optimization. The DQL agent was firstly trained on two intersections; then, by the use of transfer learning, it was extended up to the four intersections. The shortcoming of that approach was the complex reward function which is hard to implement in the real world, and the inability of efficient scaling up to the larger number of intersections. In the third MARL approach, an agent conducts local observation and limited communication with other agents usually by distributing the global Q-function over the local agents. This is done by the use of coordination rule which governs _trade-off_ between the optimization level and scalability rate, or it is possible to introduce the Independent Q-learning (IQL) approach. In the IQL approach, each local agent learns its own policy independently and enable modeling of other agents as the parts of the environment dynamics [9]. The drawback of this approach is related to the appearance of the non-stationarity environment from the point of view of each agent since the controlled environment is affected by the actions of other agents which themselves also learning [45]. That approach can be extended by the use of DQN for RL in the form of Independent DQN (IDQN). In that approach, each agent observes its nearby environment (known as the partial state), then computes individual action, and finally receives a group reward that is shared among all agents.