## 6 Discussion  
Fixed traffic signal plans can produce satisfying performance when traffic conditions are consistent in a temporal and spatial context. However, their performance degrades in cases when the traffic conditions are subject to the rapid fluctuations in mobility demand or in the case when intersection throughput is reduced. Those disruptions in traffic flows can occur due to the various types of traffic incidents, public events of great interest, or unanticipated road obstructions. The RL as one of the approaches for on-line machine learning provides an optimal ratio between the complexity and efficiency among various model-free data-driven traffic control methods. It introduces needed self-adaptation features to tackle control problems related to the new unforeseen traffic scenarios. Conventional RL based approaches store records that contain the described relationship between the states and actions by using a 2-dimensional matrix. In the case of a large traffic network, this matrix or table can become too vast to learn. An estimation of the return with respect to the mentioned table is introduced and it is known as the value function [28]. Those types of RL algorithms are known as Critic-only. They depend on the efficiency of value function approximation since they do not use the explicit function for the policy. The RL algorithms based on the policy function are known as  
Figure 6: The concept of Deep Open Traffic Data framework applied for intersection network. the Actor-only algorithms and they use a large continuous action spectrum. The drawback of those algorithms is high variance in the gradient estimation what makes their learning slow [28]. There are numerous methodologies that can reduce the high-dimensionality of Critic-only RL algorithms by conducting the value function approximation schemes in a form of an empirical balance between representational power and computational costs such as Tile Coding and Radial Basis Functions (RBF). However, they still require manual feature extraction from the raw traffic data. During this process, it is possible to lose a lot of useful information about traffic flow. The introduction of DNN models that utilize the feature extraction and estimation/classification has enabled direct learning from the information-dense input data. Furthermore, the use of the DNN model in the DRL framework as the function approximator within a Critics-only RL setting removed the need for searching vast tables what consequently mitigate the problem with dimensionality explosion. Currently, the most used architecture for the DRL framework is based on the Actor-Critic approach since the Critic part evaluates the quality of the used policy and the Actor uses the Critic's information to update its policy parameters. Those two integrated parts are capable of producing continuous actions while the high variance in the estimation of gradients of Actor-only methods is reduced by considering the value function of the Critic part as it is represented in [58]. The main efficiency measure for novel ATSC algorithms is their effectiveness in holistic traffic signal control since the poor scalability is the most prominent drawback of those types of algorithms [59]. During the last five years, according to Table 1, it is possible to notice an increase in the number of intersections that ATSC based on the DRL approach can handle. The scalability of the latest ATSC approaches which are based on DRL is tackled by the deep MARL frameworks. In most cases, those frameworks are based on the global Q-function distribution to the local DRL agents in the form of the IA2C, transfer learning, or introducing a hierarchical regional A2C/A3C approach. Simultaneously, over the past five years, it is possible to notice the growing complexity of DNN models which are used in the advanced Actor-Critic architectures. All mentioned advancements are proof of the growing interest of the scientific community in that field.