## 3 Adaptive Traffic Light Signal Control Based on DRL
### Traffic Signal Control on a Larger Scale
Furthermore, the IDQN approach in [25] achieves learning over the multiple heterogeneous agents by the use of Dueling Double Deep Q-Network (3 DQN) for each agent. Approaches that are based on IDQN are prone to the RM relevance reduction because the data generation dynamics in the local agent's RM no longer reflect the current dynamics in which learning is done [45]. The problems with IDQN convergence stability induced due to the mentioned drawback can be alleviated by the introduction of two approaches proposed in [9]. The first is known as _fingerprinting_, which is based on information about neighborhood policies to improve the observability of each local agent thus each local agent has more information about the regional traffic distribution and usage of cooperative strategy. That approach consequently reduces the non-stationarity nature of the multi-agent environment [25]. The second is based on the inclusion of a spatial discount factor with the main goal to weaken the state and reward signals from other agents. The most advanced IDQN technique for the deep MARL control over the several traffic intersections is the Independent A2C (IA2C) approach which is based on the Actor-Critic architecture for DQN. The most prominent issue with the ATSC over the larger number of intersections is related to the feasible processing of input traffic data that is continuously sampled by the numerous traffic detectors. The Deep Deterministic Policy Gradient (DDPG) as the one type of off-policy DRL is proposed in [20] as the conceptual approach that can tackle the large scale ATSC problem which relies on the aggregated data from the road traffic sensors. It combines the Actor-Critic approach with the Deterministic Policy Gradient (DPG). Deterministic policy as opposed to stochastic is approximated by an ANN-based Actor \(\pi(s;\theta^{\pi})\) usually designed with a fully connected Multi-Layer _Perceptron_ (MLP) and _Leaky_ ReLU activation function. Thus, its actions depend on the state of the environment \(s\) and has weights \(\theta^{\pi}\). The size of the first layer in the Actor model is defined by the number of detectors in the traffic network and the number of available TSPs, while the final layer size corresponds to the number of actions. The DDPG off-policy nature combined with the aforementioned structure of the Actor model can provide a framework for continuous traffic data digestion from numerous sensors on a large traffic network. That structure enables the computation of actions in the form of the vector with \(N\) components. Each of those components is a real number that has a scaling effect on the duration of each TSP with the main purpose to preserve the duration of the total cycle. Those scaled values for TSP duration are stored in the phase adjustment matrix. They are coded in the final layer known as \"Phase adjustment\". The Actor network is updated according to the applied chain rule to the loss function, and the update of weights \(\theta^{\pi}\)) is based on gradient loss. The second separate model based on fully connected layers is implemented within the Critic part which computes the value \(Q(s,a;\theta^{\pi})\) according to the current state. It is updated by the means of the _Bellman_ equation as is the case with the DQN. The mentioned study does not provide results regarding the comparable traffic parameters but only shows favorable convergence of reward under the traffic scenario with 43 intersections. Furthermore, [PERSON] et al. [34] proposed a decentralized-to-centralized architecture for traffic signal control problems in a large traffic network. The main idea in this approach is to tackle large-scale control problems by transferring learned base models into the sub-regional grids. The first step is to conduct a partitioning of the large traffic grid into several sub-regional grids of the same topology. Each sub-region is paired with a single DRL agent for the local optimal policy learning within a sub-regional grid.