Improvement of trac signal control (TSC) eciency has been found to lead to improved urban transportation and enhanced quality of life. Recently, the use of reinforcement learning (RL) in various areas of TSC has gained signi cant traction; thus, we conducted a systematic literature review as a systematic, comprehensive, and reproducible review to dissect all the existing research that applied RL in the network-level TSC (NTSC) domain. The review only targeted the network-level articles that tested the proposed methods in networks with two or more intersections. We used natural language processing to de ne the search strings and searched Google Scholar, Web of Science, IEEE Xplore, ACM Digital Library, Springer Link, and Science Direct databases. This review covers 160 peer-reviewed articles from 30 countries published from 1994 to March 2020. The goal of this study is to provide the research community with statistical and conceptual knowledge, summarize existence evidence, characterize RL applications in NTSC domains, explore all applied methods and major rst events in the de ned scope, and identify areas for further research based on the explored research problems in current research. Reinforcement Learning; trac light control; urban network; multi-agent system; intelligent transportation sys- tem; arti cial intelligence. 1. Introduction With an explosion in urban and rural population rates, city transportation systems become less ecient at han- dling the ever-growing numbers of commuters. A lack of space and resources with which to improve infrastructure poses problems in accommodating the increasing urban population. The resulting congestion further leads to in- creased pollution caused by sitting idle in trac jams, trac delays and bottlenecks, and a rise in accidents. The secondary issues that arise are just as severe, including economic loss and an overall decrease in quality of life. This presents the problem of improving trac ow and trac signal control (TSC) within the already existing infrastructure. Trac signals are most often regularised through xed time, actuated, or adaptive control methods, whether the state-of-the-art methods or the methods deployed in the real-world, such as SCATS (Sims et al., 1981), SCOOT (Hunt et al., 1981), and TUC (Diakaki et al., 2002). Fixed time signal control involves a repeating pattern that does not change with the live trac situation and which continues through its cycles regardless of dynamic trac changes * Corresponding author. E-mail: m.noaeen@utoronto.ca in that area. The actuated control method operates trac signals based on real-time data of loop detectors. Despite being trac responsive, the actuated control method is not designed to fully address uctuating trac demands, thereby rendering it less than optimal, speci cally in highly saturated volumes. Conversely, an adaptive signal is a more ecient solution as it has the built-in capacity to adapt to trac changes without the restrictions that plague the actuated method. Reinforcement Learning (RL) (Sutton, Barto, et al., 1998) in TSC is generally employed to advance the category of the adaptive methods. Derived from the natural learning processes observed in animals, RL allows a TSC system to learn and adapt to their environment. In a trac environment, several components, such as pedestrians, drivers, vehicles and trac signals may interact with each other. In TSC, trac signals are the most common agents. Some TSC systems have a single agent in the RL environment, however it is common to have multiple agents work either cooperatively or competitively, in what is called Multi-Agent Reinforcement Learning (MARL). The bene t here is that agents work across a large environment while still having the precision of a single agent or close to it. The agents interact in a simulated trac environment in di erent situations to learn the optimal way of interacting with the environment in a real-world setup. RL works based on a reward system that promotes long-term goals in an environment. The learning process is a feedback cycle of state, action and reward, where RL learns how an agent should map the states to actions to maximize a reward (this is discussed in detail in Section 1.1.). See Figure 1. The action often involves setting the phase duration. However, there are other action types, including setting the phase order, cycle time, o set, etc. We de ne these TSC-related terms in the following. A phase is de ned as a period of time during which a set of non-con icting trac movements receive a green signal. A cycle is composed of several phases and cycle time is the time required to complete a full sequence of the phases. The proportion of the cycle that is green is called split. Moreover, in a coordinated system, o set is de ned as the time that the green phase at an intersection begins after the beginning of green time of the reference signal. The main goal of TSC is to improve the environment or network performance (e.g. delay time, travel time, queue length, and speed) by controlling the actions of the agents. The main focus of this paper is on controlling the timing of trac signal agents, although this control can be integrated with the control of actions of other types of agents, like vehicles in a connected vehicle environment. Due to the rising popularity of RL in TSC recently, speci cally in NTSC, we aim to thoroughly characterize the existing research in the area of urban trac networks where RL is applied and to provide a complete account of what has already been explored. To this end, we exclude the research that has only proposed or tested for single isolated intersection control. Thus, we only concentrate on the application of RL in the network-level TSC, called as RL in NTSC or RL-NTSC for brevity, that tested the proposed methods in networks with two or more intersections. It is worth noting, however, that there are several surveys and review papers that do cover this area. For instance, (Wei et al., 2019c; Yau et al., 2017; Mannion et al., 2016; Bazzan and Kl ugl, 2014; Bazzan, 2009; Abdulhai and Kattan, 2003; Wei et al., 2021) all present general reviews or surveys on TSC methods, compiling a list of the most recent methods and algorithms related to RL in TSC. Additionally, two very recent papers, (Greguri c et al., 2020; Haydari and Yilmaz, 2020), discuss the applications and opportunities regarding Deep Reinforcement Learning (Deep RL) in TSC, while a number of relevant studies that do not exclusively focus on RL in TSC, e.g. (Yuan Wang et al., 2019; Nguyen et al., 2018; Tahilyani et al., 2013; Z. Liu, 2007; J acome et al., 2018; Yizhe Wang et al., 2018a; D. Zhao et al., 2011; Eom and B.-I. Kim, 2020) were found to exist. Nonetheless, to the best of our knowledge, there is no such systematic literature review aimed at examining the existing research of RL-NTSC. This research aims to: (i) collect all the existing relevant papers in the de ned area and present a systematic, explicit, comprehensive, and reproducible review for identifying, evaluating, and synthesizing the existing body of the literature (based on the de nition of a systematic literature review given by Fink (Fink, 2019) and Okoli et al. (Okoli and Schabram, 2010)), (ii) provide statistical and conceptual knowledge based on the qualitative and descriptive data analysis on data extracted from the included articles to investigate what has been done in the area, which methods and techniques were used (alone or as a core or combined method in integration with other methods), and which patterns, trends, and information can be extracted using data analysis techniques from the data reviewed, (iii) show which methods and which NTSC application domains have still room to be more elaborated in further research, (iv) identify the major rst events in RL-NTSC to get familiar with how the research novelties and contributions are temporally located in 2 RL controllerActionRewardRL (Training)State elements Reward elementsState Demand traffic networkTrained RL controllerActionRL (Test)State elementsState Demand traffic network(a) (c)(b)Figure 1: A general RL process: a) RL training, b) RL testing, and c) a sample intersection. the course of research in the area, which is speci cally well to nd the very recent research problems, (v) explore the recent research problems and domains with their frequencies that helps identify potential future research, (vi) provide common future directions based on what the included papers recommended, and (vii) summarize existence evidence, (viii) identify and summarize areas for further research. For convenience, Tables 1 summarize the main acronyms in RL and NTSC domains. 1.1. Reinforcement Learning in Trac Signal Control A Markov Decision Process (MDP) is a mathematical framework well suited to optimize decision-making processes under uncertainty. An MDP is a four-tuple hS,A,R,Ti, including, respectively, state space S, action spaceA, reward functionR, and transition function T. An MDP satis es the Markov Property if the transition function, whether known or unknown, depends only upon the current state and the action taken, not on the sequence of events that preceded it. If the reward and transition functions are known, the optimal policy can be found using dynamic programming (DP) methods via the recursive de nitions of the value function. However, when the environmental dynamics are not known, i.e. reward and transition functions are unknown, the agent has to estimate the value of taking action in a state without using knowledge about the reward function and transition probabilities. In this situation, RL is suitable. RL can be model-based where the agent samples from the environment to estimate the reward and transition functions and nd an optimal policy. Unlike model-based RL, in model-free RL algorithms the agent directly estimates the Q-function (Q-function is discussed later in this section) from experience while the reward and transition functions are unknown beforehand. 3 Table 1: Summary of abbreviations in RL and NTSC. Acronym Description AC Actor-Critic AV Autonomous Vehicle ADP Approximate Dynamic Programming BPNN Back Propagation Neural Networks CAV Connected and Autonomous Vehicles CBR Case-Based Reasoning CMAC Cerebellar Model Articulated Controller CMOP Correlated Multi-Objective Problem CNN Convolutional Neural Networks CPT Cumulative Prospect Theory CQF Connectionist Q-learning Framework CRFs Conditional Random Fields CTM Cell Transmission Model DAE Denoising Auto Encoder DDQ Double Deep Q-learning Deep CNN Deep Convolutional Neural Networks Deep NN deep Neural Networks Deep RL Deep Reinforcement Learning DNN Dueling Neural Networks DP Dynamic Programming DQF Decomposable Q-function FFNN Feed Forward Neural Networks FNN Fuzzy Neural Networks GA Genetic Algorithms GAN Graph Attentional Network GCN Graph Convolutional Network GT Game Theory I2V Infrastructure-to-Vehicle IGMN Incremental Gaussian Mixture Network KNN K-Nearest Neighbors LA Learning Automata LC Linear Combination LCS Learning Classi er System LIME Local Interpretable Model-Agnostic Explanations Linear FA Linear Function Approximation LMT Linear Model Tree LR Linear Regression LSTM Long Short-Term Memory LTL Linear Temporal Logic MARL Multi-Agent Reinforcement Learning MDP Markov Decision Process ML Machine Learning MOPSO Multi-Objective Particle Swarm Optimization MP Max Pressure NFQI Neural Fitted Q-iteration NLP Natural Language Processing NN Neural Network Pro-CEP Proactive Complex Event Processing RBF Radial Basis Functions RL-CD Reinforcement Learning with Context Detection RM Ramp Metering RNN Recurrent Neural Networks SF Smoothed Functional SHAP Shapley Additive exPlanation SI Signalized Intersection SPSA Simultaneous Perturbation Stochastic Approximation SR Sigmoid Regression TD Temporal Di erence TF Triangular-shaped Functions V2I Vehicle-to-Infrastructure V2V Vehicle-to-Vehicle VMS Variable Message Signs WNN Wavelet Neural Networks 4 In model-free RL, an agent tries to learn the optimal way of interacting with an environment. RL learns how an agent should map the states to actions to maximize a numerical reward. At each time step (or decision point) k, based on a policy that is intended to be optimized during the learning process towards reaching an optimal policy, the agent takes action akfrom a set of possible actions Ain response to the current state skfrom a set of possible states S; i.e.ak=(sk). Simply put, a policy is a rule that the agent follows in selecting actions based on its current state. At the end of step k, the agent receives a reward rkfrom the environment based on a reward functionRwhere the elements of the reward can be collected through sensors. A sequence of state, action, and reward is a history of an agent saved in memory. At each time step, the RL agent tries to learn an optimal policy from its history of interactions with the environment that maximizes the discounted cumulative reward: Rk=1X =k ( k)r =rk+ rk+1+ 2rk+2+:::=rk+ (rk+1+ rk+2+:::) =rk+ Rk+1; (1) where 2[0,1] is a discount factor. The discount factor is associated with time horizons and is used to balance immediate and future rewards. It determines how much the RL agent cares about rewards in the distant future compared to those in the immediate future. If is 0, the agent only cares about the most immediate reward ( Rk=rk). If is 1, the reward is not discounted and the distant future reward is considered ( Rk=rk+rk+1+rk+2+...). As we set closer to 1, future rewards are given greater emphasis relative to the immediate reward. For more details, see (Sutton and Barto, 2018). One of the most frequently used and successful RL methods in trac signal control is Q-learning (Sutton and Barto, 2018), which was rst investigated in 1989. Q-learning is a model-free RL. It is also an o -policy RL algorithm that uses a di erent policy for estimating Q-values than for action-selection. It updates the Q-values of the current state-action pair using the greedy policy to estimate the Q-value of the optimal policy of the next state-action pair. In other words, the optimal policy is learned by estimating a second function, called Q-function, that speci es the value of an action (following a given policy ) given the current state. Q-function calculates the quality of a state-action combination. Assuming the agent continues to follow the optimal policy, the Q-value is de ned as the expected discounted future reward of taking action akin statesk: Q(sk;ak) =E[Rkjsk;ak] =E[1X =k ( k)r jsk;ak]: (2) The Q-value is estimated by iterative Bellman updates: Q(sk+1;ak+1) =Q(sk;ak) + ( kQ(sk;ak)); (3) where 2[0,1] is the learning rate that is set through experimentation, and k=rk+ max ak+1Q(sk+1;ak+1) is the target. Hence: Q(sk+1;ak+1) =Q(sk;ak) + [rk+ max ak+1[Q(sk+1;ak+1)]Q(sk;ak)]; (4) Q-learning uses Q-table to decide which action to take. A simply put summary of Q-learning (Sutton and Barto, 1998) shows the algorithm to follow these steps: 1.skis the current state in which the agent resides. 2. The agent chooses action akfrom the available or acceptable actions. 3. In response, the agent receives a reward rkfor actionakand the next state sk+1. As already discussed, this statesk+1is merely a projection or estimation of the next state, rather than a known value. 4.Q(sk+1;ak+1)is then updated using equation 4. 5. The entire process is repeated. 5 Table 2: Details of the search process Journal/Conference #QGS (20172019) IET Intelligent Transport Systems 2 Journal of Intelligent Transportation Systems 3 Engineering Applications of Arti cial Intelligence 2 IEEE Transactions on Intelligent Transportation Systems 2 IEEE Transactions on Vehicular Technology 1 International Conference on Intelligent Transportation Systems (ITSC) 2 ACM International Conference on Information Knowledge Management (CIKM) 3 Total 15 Here, we should note that Rk+1in Equation 1 is the reward that is obtained at the next time step (after taking action), but it is not necessarily known upfront. That is why Q-learning is a model-free RL that does not know about the model of the environment, i.e. the reward and transition functions. In the learning process, there is the exploration-exploitation dilemma. The agent tries to exploit based on what it already learned to achieve the reward, and at the same time, it also must explore possible actions for each state to nd the one that has received the highest reward for exploiting it. The rest of this paper is organized as follows: Section two will explore the methodology used to conduct this review, including the search strategy, selection criteria, and data extraction; Section three delves into the results and ndings presented in section one; Section four discusses common future works and key ndings; Section ve covers threats to the validity of our work; and Section six wraps up with a look at future implications of the current paper. 2. Review Method In this study, search strategy, inclusion and exclusion criteria, and data extraction are intertwined, and the selection criteria and data extraction are performed within the search and search evaluation steps; hence, we explore this as a whole rather than separately. It should be noted that to identify the relevant literature we conducted both manual and automated searches, and included all articles published up to the end of March 2020. We designed and implemented our review based on the guidelines provided by (Tricco et al., 2018). 2.1. Search strategy, selection criteria, and data extraction To identify the most appropriate search terms and strings for the automated search, we used literature review, manual content analysis, and Natural Language Processing (NLP) in the steps that follow. 1:Based on knowledge in the area of the study, we identi ed seven venues, including ve journals and two conferences as listed in Table 2, and manually searched for and reviewed the relevant published studies from 2017 to 2019 based on their title, abstract, and keywords. In very limited cases, we also examined the conclusion and searched for speci c, relevant key terms to help with the inclusion decision. Fifteen pertinent articles were found during this rst stage search. These papers were also used in the fourth step to form the Quasi-Gold Standard (QGS) (Abad et al., 2016) to evaluate the quality of the search strings. 2:We performed NLP, including language modelling and lexical association analysis (Abad et al., 2019), to extract the most frequently used terms in the 15 retrieved articles for the purpose of identifying search strings. Figure 2 shows the directed graph of common bi-grams formed from the QGS set, based on the frequency analysis on the pre-processed texts collected from the title, abstract, introduction, section/subsection headings, and conclusion of the 15 articles. Based on the results from the NLP, several inspections and investigations, and using various combinations of the most related search terms, the following single search string was chosen: hreward AND action AND trac light OR trac signal AND reinforcement learning i. 3:Using the identi ed search string we queried Google Scholar, which served as our main search base, yielding 2,887 articles. To increase the reliability of our ndings, the search was complemented by searching within Web of Science, IEEE Xplore, ACM Digital Library, Springer Link, and Science Direct databases after the data extraction. 6 ●●● ●●● ●● ● ●●● ●● ●●● ●● ●●● ● ●● ● ● ●● ●● ●●● ● ● ● ●●●●●●●● ●●● ●● ●● ●● ●● ●● ●● ● ● ● ●● ●● ● ●●●●● ●●● signaltrafficreinforc waitneuralqueue realcontrolrl averaglightnqn multiadapt cumulfixedpropos rewarddiscount learnoptimaction deep fixlearning rushsimul timemotor roadsocial totaltransportactuat artifici complexdecis functionfuturinformintelligpoliciredsuperiortim valunetworklength worldsystem algorithmtsc agflow delaymodel scenariofactor methodspacebashour result planmethods basemedia data approximresearchsharederivsolut structurhoursperform ●●● ●●● ●●● ●●● ●● ●●● ●● ●●● ● ●● ● ● ●● ●● ●●● ● ● ● ●●●●●●●● ●●● ●● ●● ●● ●● ●● ●● ● ● ● ●● ●● ● ●●●●● ●●● signaltrafficreinforc waitneuralqueue realcontrolrl averaglightnqn multiadapt cumulfixedpropos rewarddiscount learnoptimaction deep fixlearning rushsimul timemotor roadsocial totaltransportactuat artifici complexdecis functionfuturinformintelligpoliciredsuperiortim valunetworklength worldsystem algorithmtsc agflow delaymodel scenariofactor methodspace bashour result planmethods basemedia data approximresearchsharederivsolut structurhoursperform ●●● ●●● ●● ● ●●● ●● ●●● ●● ●●● ● ●● ● ● ●● ●● ●●● ● ● ● ●●●●●●●● ●●● ●● ●● ●● ●● ●● ●● ● ● ● ●● ●● ● ●●●●● ●●● signaltrafficreinforc waitneuralqueue realcontrolrl averaglightnqn multiadapt cumulfixedpropos rewarddiscount learnoptimaction deep fixlearning rushsimul timemotor roadsocial totaltransportactuat artifici complexdecis functionfuturinformintelligpoliciredsuperiortim valunetworklength worldsystem algorithmtsc agflow delaymodel scenariofactor methodspacebashour result planmethods basemedia data approximresearchsharederivsolut structurhoursperform●●● ●●● ●● ● ●●● ●● ●●● ●● ●●● ● ●● ● ● ●● ●● ●●● ● ● ● ●●●●●●●● ●●● ●● ●● ●● ●● ●● ●● ● ● ● ●● ●● ● ●●●●● ●●● signaltrafficreinforc waitneuralqueue realcontrolrl averaglightnqn multiadapt cumulfixedpropos rewarddiscount learnoptimaction deep fixlearning rushsimul timemotor roadsocial totaltransportactuat artifici complexdecis functionfuturinformintelligpoliciredsuperiortim valunetworklength worldsystem algorithmtsc agflow delaymodel scenariofactor methodspacebashour result planmethods basemedia data approximresearchsharederivsolut structurhoursperformFigure 2: Directed graph of common bi-grams resulted from the NLP that is used to de ne the search strings. This accelerated our search process by providing a view of those papers that were included and excluded, based on the de ned criteria. 4:We formed the Quasi-Gold Standard. Our automated search retrieved all 15 articles, indicating the quasi- sensitivity of 100%. 5:We began the process of inclusion and exclusion by carefully de ning the selection criteria. This iterative process continued even during data extraction, (i.e. the last step) to ensure the correctness of the included and excluded papers. Figure 3 depicts the owchart of the article selection process in this paper. Publications that meet the following criteria were excluded: (1) those articles not related to RL-NTSC, (2) duplicate papers, (3) review and survey papers, (4) presentations, abstracts, extended abstracts, viewpoints, letters, reports, technical reports, projects, table of contents, any papers that have not been peer reviewed, theses and dissertations, books, and book chapters, (5) publications in any language other than English, (6) those papers whose better or updated version was found in another journal or publication and used, and (7) unavailability of the full-text of the paper on the internet. Since the scope of our research is network-scale, it is henceforth referred to as NTSC (i.e. any scales involving two or more intersections), and another important exclusion criterion used was those papers that propose or test a proposed method in a single isolated intersection only. We included those papers containing an RL method, whether it was the core or combined method applied in the context of NTSC. These were included whether or not they provided evaluation and simulation, and if they provided only a framework. 7 105 full-text records excluded 105 single isolated intersection34 full-text records excluded 0 duplicate papers 3 irrelevant/review papers 25 2 non-English papers 1 a better/journal version exists 3 unavailable full-text records 7 full-text records excluded 7 single isolated intersection2640 full-text records excluded 39 duplicate papers 2085 irrelevant/review papers 473 22 non-English papers 11 a better/journal version exists 3 unavailable full-text records2887 records identiﬁed through systematic database search for title, abstract, and conclusion screen 52 records identiﬁed through the snowballing process254 TSC-related records identiﬁed149 records included in qualitative synthesis 167 TSC-related records identiﬁed (149 already included records and 19 newly identiﬁed TSC-related records)160 records included in qualitative synthesiseditorial/viewpoints/letters/non-peer reviewed records/books/book chapters/theses editorial/viewpoints/letters/non-peer reviewed records/books/book chapters/thesesFigure 3: The owchart of systematic literature review process. This investigation is designed to cover the papers that apply RL in controlling trac signals, and as such, if RL is used to control only vehicles, it is out of scope for our paper. In essence, as long as RL is being applied in NTSC, whether vehicle control is involved or not, the paper was included in the current study. Hence, the connected and autonomous vehicle (CAV) environment is covered in our paper as long as NTSC is involved. The CAV in the current study is inclusive of all environments of cases of CAV, including: vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), and infrastructure-to-vehicle (I2V). Despite the fact that below-mentioned research areas have trac components (including: public transit, bikes, and pedestrians) in common with our research, they are out of scope since TSC is not their main focus as it is in the current study. The research areas include: ramp metering, freeway, trac control (not TSC), public transit (where the focus is on bus control and TSC is excluded), route choice, routing systems, pedestrian routing, reactions of cyclists to speed advice, ride-sharing, best path selection, lane changing, autonomous intersection, trac congestion detection, driver behaviour, trac signal control simulation, simulator, simulation environment, online calibration, trac assignment problems, couriers management in express systems, eet management, toll plaza, trac analytics, trac control architecture based on fog computing paradigm where the focus is on fog paradigm (not TSC), image- based learning, image processing, and optimizing the sensor installation locations in a trac network. We only found the topics, including ramp metering, public transit, emergency vehicles, and fog computing in NTSC, in which NTSC is included; thus we included these studies. 6:After applying the selection criteria and identifying the included papers, we used both, backward snowballing by screening the reference list of the included papers, and forward snowballing (Greenhalgh and Peacock, 2005) by 8 Figure 4: Distribution (solid line) and trend (dashed line) of number of studies over time. To make the comparison consistent, the 6 included papers published in 2020, until the end of March, are not included in this chart. scanning the citation of the included papers to reduce the probability of missing some papers in our inclusion process; see Figure 3. 7:In addition to Google Scholar, ve other electronic databases were searched, yielding no new articles. All articles found here had already been covered in the search using Google Scholar and the snowballing method. 8:From the included papers, data were extracted and analyzed. In the following section, we provide an analysis of the data we extracted. In Table 3, we present an overall comparison along with details of some of the features extracted for all 160 papers included in our study. The papers that were reviewed are listed in the section entitled Included Articles. Other resources cited in this paper are listed in the reference section. It should also be noted that RL methods in Table 3 in the second column are classi ed into three main categories: model-free, model-based and RL. In this table, For model-free RL methods we present the speci c model-free RL method that is used, such as Q-Learning, SARSA, etc. However, the model-based RL methods are shown as \Model-based RL". Also, in this table, by \RL" we mean to represent RL in general, where no speci c RL method is mentioned in the paper. Moreover, there are articles that do not provide any speci c RL method rather they mention the temporal di erence (TD) update strategy. In this case, we report the method as TD as presented in these papers. 3. Results As shown in Figure 4, the number of studies regarding RL-NTSC is increasing. This topic continues to gain momentum, and hence, importance as the world and speci cally, urban populations increase and this is demonstrated in the large number of papers published recently. During the rst 15 years, from 1994 to 2008, a total of 22 papers on the subject were published, compared to 27 papers in the year 2019 alone. These statistics further attest to the fact that an in-depth review like ours is important at this time as it provides future works a comprehensive view of the past 25 years. The 160 included articles have been published in 104 di erent publication venues, including conferences (57%), journals (39%), and workshops and symposia (4%), see Figure 5. The rst (Mikami and Kakazu, 1994) and second (Cao et al., 1999) papers in RL-NTSC were published in 1994 and 1999, respectively. These papers proposed model- free RL methods, while the rst model-based RL method in RL-NTSC (M. A. Wiering, 2000) was proposed in 2000. Among the 160 included papers, 6 papers (M. Wiering et al., 2004; Kuyer et al., 2008; Houli et al., 2010; Khamis et al., 2012; Khamis and Gomaa, 2012, 2014) used or extended this model-based method, called as Wiering method. In addition to these Wiering method based research works, (Da Silva et al., 2006) proposed a model-based method that is based on a mechanism for creating, updating and selecting one among several partial models of the environment. 9 Table 3: Overall comparison of the included articles. [ 3/7]: yes for discrete methods and no for continuous methods. Citation RL Method Combined Method Approx. Method Action Selection Action Type Discretization Spillback or Sat. Mikami and Kakazu, 1994 LA Genetic Alg. { { 7 7 Sat Cao et al., 1999 LCS Fuzzy { Credit Assignment 7 7 Neither Cao et al., 2000 LCS { { Credit Assignment 7 7 Sat M. A. Wiering, 2000 Model-based RL { { -greedy 7 7 Sat Camponogara and Kraus, 2003 Q-learning { { -greedy 7 7 Sat Choy et al., 2003 RL Fuzzy, Evol. Alg., NN FNN { 5 3 Sat M. Wiering et al., 2004 Model-based RL { { { 5 7 Neither Dowling et al., 2004 RL { { Greedy 7 { No evaluation Ling and Shalaby, 2005 Q-learning { IA -greedy 6 3 Neither Da Silva et al., 2006 Model-based RL { { { 3 3 Sat Srinivasan et al., 2006 RL Fuzzy, Evol. Alg., NN SPSA, FNN { 7 3 Sat Oliveira et al., 2006 RL { { { 3 3 Sat Richter et al., 2007 Actor-Critic { Linear and Non-linear FAs Softmax 7 3 Spillback Su and Tham, 2007 Q-learning SensorGrid CMAC Softmax 6 7 Neither Zhang et al., 2007 Policy-Gradient CRF { { 7 7 Sat Bazzan et al., 2007 LA { { { Multiple 7 Sat Ritcher, 2007 Policy-Gradient { Linear FA { 7 3 Neither Kuyer et al., 2008 Model-based RL Max-plus { -greedy 7 3 Sat Salkham et al., 2008 Q-learning { { Softmax 5 3 Neither T. Li et al., 2008b ADP { FFNN { 5 7 Neither S. Lu et al., 2008 Q-learning Multiband { -greedy 6 7 Sat T. Li et al., 2008a ADP Neuro-fuzzy { { 3 7 Sat Xinhai and Lunhui, 2009 Q-learning { { { 7 { No evaluation Cai et al., 2009 ADP { Linear FA { 7 7 Sat Dusparic and Cahill, 2009a W-learning { { DWL 7 7 Sat X. Zhao et al., 2009 Q-learning GT BPNN { 7 7 Sat Dusparic and Cahill, 2009c W-learning { { DWL 7 3 Sat W. Wu et al., 2009 Q-learning { { { Not clear 7 Neither Dusparic and Cahill, 2009b W-learning { { { 7 { No evaluation Prothmann et al., 2009 LCS Evolutionary Alg. { { 5 7 Sat Medina et al., 2010 Q-learning { { Multiple 7 3 Spillback Prashanth and Bhatnagar, 2010 Q-learning { Proposed Method { 7 3 Sat Bazzan et al., 2010 Q-learning { { Softmax 3 3 Sat Arel et al., 2010 Q-learning { FFNN 6 7 Sat Davarynejad et al., 2010 Q-learning Fuzzy Granulation { { 7 { No evaluation Dai et al., 2010 ADP { NN -greedy 7 3 Sat Balaji et al., 2010 Q-learning { { Greedy 5 3 Spillback Natarajan et al., 2010 Inverse RL { { Softmax 7 3 Neither Waskow and Bazzan, 2010 Q-learning { Tile Coding -greedy 3 3 Sat El-Tantawy and Abdulhai, 2010 Q-learning GT (Coalition) { { 7 { No evaluation Salkham and Cahill, 2010 Q-learning { { -greedy 5 3 Sat Houli et al., 2010 Model-based RL { { { 7 7 Spillback Prashanth and Bhatnagar, 2011 Multiple RL { Average Cost Alg. { 6 3 Sat Song et al., 2011 Q-learning { { { 7 7 Neither 10 Citation RL Method Combined Method Approx. Method Act. Selection Action Type Quantization Spill or Sat. C.-g. Li et al., 2011 Q-learning { { { Not clear 7 Sat W. Lu et al., 2011 Actor-Critic Swarm Optimization { { 7 7 Neither Natarajan et al., 2011 Inverse RL { { { 7 3 Neither Heinen et al., 2011 Q-learning { IGMN -greedy 3 7 Sat Abdoos et al., 2011 Q-learning { { -greedy 4 3 Sat Dai et al., 2011 RL { FFNN { 7 7 Sat Dusparic and Cahill, 2012 W-learning { { Softmax 7 7 Sat Medina and Benekohal, 2012 Q-learning Max-plus { { 7 7 Spillback Khamis et al., 2012 Model-based RL { { { 7 7 Sat Khamis and Gomaa, 2012 Model-based RL { { Softmax 7 3 Sat M. N. S. Jadhao and M. P. A. Kulkarni, 2012 Q-learning { { { 7 3 Spillback L.-H. Xu et al., 2013 Q-learning { { Softmax 3 7 Sat Moghadam and Mozayani, 2013 Q-learning { { Random 3 7 Neither Y. K. Chin et al., n.d. Q-learning { { -greedy 5 3 Sat Nuli and Mathew, 2013 Actor-Critic { { -greedy 5 7 Sat El-Tantawy et al., 2013 Q-learning { { { 7 7 Sat Abdoos et al., 2013 Q-learning { { -greedy 4 3 Sat Box and Waterson, 2013 Q-learning { NN -greedy 7 7 Sat Pham et al., 2013 Q-learning { Tile Coding -greedy 7 7 Sat Brys et al., 2014 Q-learning { Tile Coding -greedy 7 3 Neither Prabuchandran et al., 2014 Q-learning { { -greedy 4 3 Sat Chanloha et al., 2014 Q-learning CTM { -greedy 7 3 Sat Fagan and Meier, 2014 Actor-Critic { { -greedy 7 7 Neither Marseti c et al., 2014 Q-learning { { -greedy 7 3 Sat Khamis and Gomaa, 2014 Model-based RL { { Multiple 7 7 Sat El-Tantawy et al., 2014 Multiple RL { { Comparison Multiple 3 Sat Sadigh et al., 2014 TD { { { 7 3 Neither Abdoos et al., 2014 Q-learning { Tile Coding { 3 3 Sat W. Liu et al., 2014 Q-learning { Gradient-descent { Not clear 7 Sat N. S. Jadhao and A. S. Jadhao, 2014 Q-learning Edge Detection Alg. { Random 7 7 Sat Teo et al., 2014 Q-learning { { { 5 3 Sat Xiang and Z. Chen, 2015 Q-learning Grey Theory BPNN Greedy 7 7 Spillback W. Xu et al., 2015 RL { { { Not clear { No evaluation Yin et al., 2015 ADP { Gradient-descent { 7 7 Sat Araghi et al., 2015 Q-learning { { -greedy 7 3 Sat Ajorlou et al., 2015 Q-learning CTM { -greedy 7 7 Neither Ozan et al., 2015 Q-learning { { Random 6 7 Sat Zhu et al., 2015 RL Junction Tree Alg. { -greedy 7 3 Spillback El-Tantawy et al., 2015 Q-learning { Linear Model Tree { 7 7 Sat Prabuchandran et al., 2015 Q-learning { { -greedy 2 3 Sat Abdelgawad et al., 2015 Q-learning { { -softmax 7 7 Spillback Mashayekhi and List, 2015 Q-learning Auction Theory { -greedy 7 7 Sat Abdoos et al., 2015 Q-learning { { { 5 7 Neither Tahifa et al., 2015 Q-learning Swarm Optimization { -greedy 3 7 Neither Van der Pol and Oliehoek, 2016 Q-learning { Deep NN Greedy 7 7 Neither 11 Citation RL Method Combined Method Approx. Method Act. Selection Action Type Quantization Spill or Sat. Dusparic et al., 2016 W-learning { { Softmax 7 7 Sat Gaikwad et al., 2016 Q-learning { { -greedy 3 3 Sat Prashanth and Ghavamzadeh, 2016 Actor-Critic { SPSA, SF Softmax 7 3 Neither Iyer et al., 2016 Q-learning Fuzzy { Greedy 7 3 Neither Yongheng Wang et al., 2016 Q-learning Pro-CEP Model DQF { Not clear 3 Sat Prashanth et al., 2016 RS RL CPT Paradigm SPSA { 7 7 Neither Chu et al., 2016a Q-learning { CNN UCB 7 7 Neither Shen, 2016 Q-learning Cellular Automata WNN { 7 { Spillback Chu et al., 2016b Q-learning { Linear Q-function -greedy 7 3 Neither Rosyadi et al., 2016 Q-learning { { { 7 3 Sat Yin et al., 2016 Q-learning { Linear FA { 7 7 Sat Y. Chen et al., 2017 Q-learning { { { 7 7 Sat Kristensen and Ezeora, 2017 Q-learning JADE { Greedy 7 3 Neither Darmoul et al., 2017 Q-learning Immune Network { { 6 3 Spillback Jin and Ma, 2017 SARSA { { { 7 7 Neither W. Liu et al., 2017 Q-learning Dynamic Clustering Gradient-descent, Tile Coding { 7 3 Neither El Hatri and Boumhidi, 2017 Q-learning { { { 6 7 Incident C. Lu et al., 2017 Q-learning { { Softmax 7 3 Sat Chu and J. Wang, 2017 Q-learning Max-sum LR, SR -greedy 7 7 Sat Y. Liu et al., 2017 Q-learning { { -greedy 7 7 Sat Vidhate and P. Kulkarni, 2017 Q-learning { { { 7 3 Neither Aslani et al., 2017 Actor-Critic { Tile Coding, RBF -greedy 5 3/7 Sat Al Islam et al., 2018 Q-learning { { -greedy 7 7 Sat Daeichian and Haghani, 2018 Q-learning Fuzzy, GT { { 3 7 Sat Wei et al., 2018 Q-learning { Phase Gate -greedy 7 7 Sat Aslani et al., 2018a Multiple RL { Tile Coding -greedy 5 3/7 Spillback Aziz et al., 2018 TD { { -greedy 7 3 Spillback Aslani et al., 2018b CR RL { Tile Coding, RBF, TF { 5 3/7 Spillback C. Li et al., 2018 Q-learning { DAE -greedy 7 7 Sat Kao and C.-W. Wu, 2018 Q-learning Self-Organizing Map { { 5 7 Neither X.-Y. Liu et al., 2018 Q-learning { NN -greedy 5 7 Neither Torabi et al., 2018a RL Self-adaptive NTSC { { Act 1 3 Sat Yizhe Wang et al., 2018b Q-learning Bayesian Optimization { { 6 7 Sat Shi and F. Chen, 2018 Q-learning Transfer Learning RNN -greedy 7 7 Sat Jin and Ma, 2018 SARSA { KNN -greedy 7 7 Sat Vinitsky et al., 2018 RL { NN { Multiple 7 Sat Lemos et al., 2018 Q-learning { { -greedy 7 3 Sat Nishi et al., 2018 Q-learning { GCN { 7 3 Sat Jin and Ma, 2019 SARSA { LC,KNN,CMAC,FFNN,CQF -greedy 3 7 Sat Wei et al., 2019b Q-learning { NN, GAN Softmax 7 7 Sat Kitagawa et al., 2019 Q-learning { Deep NN { 6 7 Sat Reda et al., 2019 Q-learning { { { 7 7 No evaluation Shabestray and Abdulhai, 2019 Q-learning { CNN -greedy 7 3 Spillback Q. Wu et al., 2019 RL Fog Computing { { 4 7 Neither P. Chen et al., 2019 Q-learning { CNN Greedy 7 7 Sat 12 Citation RL Method Combined Method Approx. Method Act. Selection Action Type Quantization Spill or Sat. Rizzo et al., 2019a Policy Gradient { Deep NN Softmax 7 7 Sat Aslani et al., 2019 Actor-Critic { { Comparison 7 3 Neither Huang et al., 2019 Q-learning { Deep NN { 5 3 Sat R. Gao et al., 2019 Q-learning Fuzzy, Edge Computing { -greedy 6 3 Sat T. Tan et al., 2019 Q-learning { { -greedy 7 7 Sat Horsuwan and Aswakul, 2019 Q-learning { Deep NN { 7 7 Spillback Chu et al., 2019 Actor-Critic { Deep NN -greedy 7 7 Sat Wei et al., 2019a Q-learning { Deep NN { 7 7 Spillback N. Xu et al., 2019 Q-learning Transfer Learning LSTM { 7 7 Sat Rizzo et al., 2019b Policy-Gradient { Deep NN Softmax 7 7 Sat Zheng et al., 2019a Q-learning FRAP Model Design Deep NN { 7 7 Sat Gong et al., 2019 Q-learning { Deep CNN { 7 7 Sat Zhou et al., 2019 Q-learning Edge Computing Deep NN -greedy Act 1 7 Sat Bouderba and Moussa, 2019 Q-learning Cellular Automata { { Not clear 3 Sat Higuera et al., 2019 Q-learning { { -greedy 7 7 Neither Ni and Cassidy, 2019 Actor-Critic { GCN -greedy Act 2 7 Sat Ge et al., 2019 Q-learning { Deep CNN -greedy 7 3 Sat Shu et al., 2019 RL { { { 7 7 Sat Gan et al., 2019 Q-learning { { Softmax 7 7 Sat Yang et al., 2019 Actor-Critic { Deep NN,CNN Softmax 7 7 Sat J. Lee et al., 2020 Q-learning { Deep CNN { 2 7 Sat Genders and Razavi, 2020 Multiple RL { FFNN Random 7 7 Sat M. Xu et al., 2020 Q-learning { RNN -greedy 7 7 Neither Qu et al., 2020 RL RMS-NE, CTM { Softmax 5 3 Sat Y. Zhao et al., 2020 RL Junction Tree Alg. { { 7 3 Sat D. Kim and Jeong, 2020 Q-learning { DNN -greedy 7 7 Sat 13 Number of publications0369121518 Publication venue 4 4 5 6 7 18International Conference on ITS (ITSC)IEEE Transactions on ITSTransportation Research Part CEngineering Applications of Artiﬁcial IntelligenceInternational Conference on Machine LearningJournal of Intelligent Transportation Systems 4% 39% 57% ConferenceJournalworkshop and symposium 1% 4% 7% 26% 62%Computer related departmentsTransportation related departmentsOther engineering departmentsOther departments (Science, business, etc)N/A(a) (b)(c)Figure 5: a) The frequency of publications in RL-NTSC in di erent publication venues, b) distribution of university department of authors in RL-NTSC, and c) distribution of publication venue types in RL-NTSC. 3.1. Countries, Departments and Aliations Figure 6 exhibits our data of the distribution of research papers among each country. 160 papers from a total of 30 di erent countries were included in this review, with most of the research papers coming from 7 countries: China, USA, India, Iran, Ireland, Canada, and Brazil. These papers represent 67% of our pool of papers, strongly suggesting that as of 2020, these countries are the global leaders in the research of RL-NTSC. An interesting fact to note is that all seven of the countries mentioned have a trac index1above 140.45 according to (Index, 2014), with Iran having the highest trac index of the group, at 216.09. The motor data company, INRIX (Inrix, 2020), nds Ireland's capital city of Dublin to be the 7th worst city in the world in terms of hours lost due to trac congestion based on 2019 data (154 hours). And, according to the Central Intelligence Agency (CIA) World Factbook (Central Intelligence Agency, 2020), the USA, China, India, and Brazil rank in the top four countries with the longest road network, respectively, with Canada at eighth. This study also took into consideration the department and aliations of authors, and identi ed four groups of departments, as follows: (1) computer related departments, such as computer engineering/science, information technology, and electrical engineering, (2) transportation related departments, including civil and transportation 1Metric that is a composite index of time consumed in trac due to job commute, estimation of time consumption dissatisfaction, CO2 consumption estimation in trac and overall ineciencies in the trac system. 14 Figure 6: Left) the distribution of studies based on country of the authors, and right) the number of authors. The number of authors of the papers published on RL in NTSC ranges from 1 to 10. engineering departments, (3) other engineering departments, such as industrial, mechanical, material, and geomat- ics, and (4) other departments, like science, business, management, astronautics, and English. It was found that 62% of the authors were from computer related departments (group 1), and 26% were from transportation related departments (group 2), while researchers from other departments (groups 3 and 4) accounted for 12% of the total. Notably, authors from these four groups of departments had very low research collaboration with each other, and had conducted research in only 20 out of 160 articles, generally. The computer related and transportation related departments collaborated in only 11 articles, while they contributed independently in 98 and 36 papers, respectively. This investigation also uncovered that academia and research institutes have low research collaborations with in- dustry and government, with 11 instances of collaboration between academia and industry and only one appearance of government in research papers that collaborated with academia in a paper in 2018. Suce it to say that research exploring the potential bene ts of the collaborations between these three parts, i.e. academia, industry, and govern- ment is needed, as the ndings in the literature (Anderson and Odei, 2018) suggest that increased collaboration in these domains may boost the eciency of the proposed methods in real-time, real-world applications. 3.2. Method identi cation and analysis In this section, we elaborate on the proposed methods from di erent angles. We start with the categorization of the proposed methods in terms of the method and environment attributes, respectively, in Figures 7 and 8. 3.2.1. Methods' contribution and combination The proposed methods in the included articles consist of (M1) the applied RL methods in NTSC (154 papers) or (M2) theoretical developments in RL with feasibility or applicability assessment of the proposed methods in di erent contexts, one of which is the NTSC environment (6 papers). To get familiar with the variations of application of RL in solving di erent research problems in NTSC and the way that RL is applied, we identi ed two main groups: (M1.1) RL is used alone or in combination with methods 15 Regular network controlPerimeter controlCar bunching controlRL Method proposal in NTSCTheoretical contribution in RL with feasibility assessment in NTSCRL alone, RL + ML/GT/DPRL + other methodsMETHOD ATTRIBUTES DATA SOURCENETWORKVEHICLE CLASSENVIRONMENT ATTRIBUTESCOMMUNICATION PROCESSNetwork of intersectionsArterial networkSignalized roundaboutPrivate vehicles (cars)Public transitEmergency vehiclesGeneral detection devicesLoop detectorsCAVs, V2V, V2I, I2ICamera/imageNo process on imageImage representationImage from simulationEdge, Fog, Cloud computingCONTROL METHOD SCHEMEMETHOD CONTRIBUTIONRL METHOD APPLICATIONNTSC METHODCentralizedFully centralizedHierarchicalDecentralizedDistributed(a) (b)Methods from other fields in RL RL in methods from other fields or in a specific design for NTSCRL in optimization, optimization in RLStreetcar bunching control/Distributed Figure 7: Method attributes in RL-NTSC. from the same eld (i.e. machine learning (ML), game theory (GT), and DP methods) in NTSC (134 papers), and (M1.2) RL is used in combination with other methods from other elds in NTSC (20 papers). hM1.1i: The vast majority of the studies (134) are classi ed in this category where RL is the only applied method, or is used in combination with other ML, GT, and DP methods. Di erent innovative designs were proposed to tackle the problems of NTSC. The NTSC problem has a continuous state space, in nite horizon, and is only partially observable and dicult to model. In this context, most of the papers try to improve the performance (e.g. (Moghadam and Mozayani, 2013)), dimensionality (e.g. by means of function approximation (Waskow and Bazzan, 2010; Prashanth and Bhatnagar, 2010)), complexity (e.g. by organizing agents in groups of limited size (Bazzan et al., 2010), and using holonic RL methods (Abdoos et al., 2013)), scalability (e.g. (Nuli and Mathew, 2013)), stability (e.g. (Aslani et al., 2018b)), speed of optimization (e.g. in Transfer Learning models (N. Xu et al., 2019)), state and/or action space manageability or generalizability (e.g. (Araghi et al., 2015; Gaikwad et al., 2016)), and centrality problem in applying RL in NTSC. Some studies invest in solving convergence and oscillation problems that commonly appear in the multi-agent context. For instance, (Reda et al., 2019) proposed a model based on Double Deep Q-learning (DDQ), with Experience Replay and cooperation between agents. They used NN to reduce the correlation between agents and improve performance. Improving the accuracy, optimum functioning, and eciency of the results is also of high interest to the researchers, through hierarchical methods (Yizhe Wang et al., 2018b; T. Tan et al., 2019), for example. There are several papers where the main focus of research is on studying the coordination between agents (e.g. (Higuera et al., 2019)), and the integrated network, speci cally signalized intersections and ramp metering (e.g. (El-Tantawy and Abdulhai, 2010; El-Tantawy et al., 2013, 2015)). Famous methods proposed in trac theory context, such as CTM (Chanloha et al., 2014; Ajorlou et al., 2015; Qu et al., 2020), Max-plus (Kuyer et al., 2008; Medina and Benekohal, 2012), and Max Pressure (MP) or back pressure (brought in NTSC from the communications networks theory) (Wei et al., 2019a), are also applied in some research, and in other studies multiple trac optimization goals are simultaneously optimized (multi-policy RL), e.g. (Dusparic et al., 2016). Considering both vehicular and pedestrian trac in the network-scale is a recent application of RL, where a distributed multi-agent RL method is proposed by (Y. Liu et al., 2017) for the rst time in 2017. By the increasing use of deep RL methods, the number of papers that focused on improving deep RL models in NTSC has increased since 2018 (e.g. (Wei et al., 2018; C. Li et al., 2018; X.-Y. Liu et al., 2018). The co-learning problem of both classes of learning agents, trac signals and drivers, with di erent goals (minimizing individual travel times vs minimizing the queues locally), di erent nature (driver agents learn in episodes that are asynchronous, while trac light agents learn continuously (non-episodic)), and the nontrivial task of microscopic modelling and simulation (whose actions are highly coupled) is another area of research that was addressed (Lemos et al., 2018). In addition to these lines of research in this category, analyzing what speci cally RL does di erently (i.e. analysis of the learned policies) than 16 other TSC methods is another research e ort that is motivated and conducted by (Genders and Razavi, 2020). hM1.2i: In category M1.2, the articles applied methods from other elds (rather than ML, GT, and DP) in RL or RL framework (M1.2.1), or used RL in the methods from other elds or in a speci c design/model/framework for NTSC context (M1.2.2). Out of 20 articles in category M1.2, there are 8 articles that either applied RL in optimization problems (e.g. Swarm optimization) or an optimization algorithm is used in RL methods/frameworks in the context of NTSC. hM1.2.1i: To cope with non-stationary environments, (Da Silva et al., 2006) proposed, formalized and showed the eciency of a method called the RL-CD, or Reinforcement Learning with Context Detection, which performs well in non-stationary environments, and better than classic RL algorithms (Q-learning and Prioritized Sweeping). In a similar work, (Oliveira et al., 2006) assessed the feasibility of applying RL-CD approach in a more realistic scenario, implemented by means of a microscopic trac simulator. (Zhang et al., 2007) showed how to use Condi- tional Random Fields (CRFs) to model control processes, where CRFs model joint actions in a decentralized Markov decision process and de ne how agents can communicate with each other to choose the best joint action. The CRF model clearly outperformed the independent agents approach. (Khamis et al., 2012) enhanced the single-objective controller by developing a multi-agent NTSC system based on a multi-objective sequential decision-making frame- work using Bayesian interpretation and some innovative reward design. (Zhu et al., 2015) proposed a Junction Tree Algorithm (JTA) based approach to obtain an exact inference of the best joint actions in trac signal coordination that outperformed independent learning (Q-learning), real-time adaptive learning, and xed timing plans. To predict future system states and avoid unwanted states, (Yongheng Wang et al., 2016) applied Proactive Complex Event Processing (Pro-CEP) method (in processing proactive trac congestion control) that uses RL to nd the optimal joint policy. This method works well when used to control congestion. To develop learning and adaptation mecha- nisms to deal with disturbances, (Darmoul et al., 2017) proposed a distributed TSC system based on hybridization between Case-Based Reasoning (CBR) and an adaptation of the reinforcement principle within arti cial immune networks (a mechanism inspired by biological immunity). The method controls interrupted ow at signalized in- tersections. (Wei et al., 2019b) studied how the attention mechanism helps cooperation (to minimize the average queue length) via using graph attentional networks to facilitate communication, incorporating the temporal and spatial in uences of neighbouring intersections to the target intersection, and building up index-free modelling of neighbouring intersections. hM1.2.2i: (Su and Tham, 2007) integrated sensor networks and grid computing and the usage of web services to implement this integration. They used Q-learning algorithms in distributed Stargates to NTSC. Stargates is a computer with sensor signal processing capabilities. (Abdoos et al., 2015) modelled a holonic multi-agent system and proposed a holonic RL multi-agent system method that improves the performance of individual Q-learning in NTSC. (Mashayekhi and List, 2015) showed the applicability and ecacy of using auction theory combined with an RL in a multi-agent system. In low trac volume the proposed method outperforms actuated and pre-timed control strategies, but in heavy volume the pre-timed control strategy outperforms the proposed method. (Iyer et al., 2016) designed a distributed multi-agent method with coordination between agents through the communication of decision data. In this paper, the e ectiveness of integrating fuzzy logic controller (to deal with continuous states and actions) and Q-learning (for learning during the process) is studied. With regards to the optimization-related articles in category M1.2, all of the proposed methods in this category are designed to handle the growing complexity and curse of dimensionality and to increase the speed of TSC by using RL in the optimization problems or applying the optimization algorithms in an RL method or framework. To achieve cooperation in the long term, as stated earlier, (Mikami and Kakazu, 1994) combined RL of a local agent with global optimization through Genetic Algorithms (GA) by which the RL parameters are modi ed. (Cao et al., 1999) used a classi er system with a fuzzy rule representation, with both evolutionary and RL methods to provide a faster method than hierarchical control. (Cao et al., 2000) incorporated Learning Classi er System (LCS) and TCP/IP (Transmission Control Protocol/Internet Protocol) based communication server into a distributed learning control strategy to increase the speed of control. (Prothmann et al., 2009) managed the complexity by using an organic approach to NTSC and proved the feasibility of the proposed approach. (W. Lu et al., 2011) proposed 17 a multi-agent NTSC by using Swarm Intelligence and Neuro-Fuzzy RL to combine the better attributes of both with improving the learning speed and performance. (W. Liu et al., 2014) optimized the TSC for V2I networks by proposing a cooperative distributed Q-learning algorithm with a fast gradient-descent function approximation. (Ozan et al., 2015) applied an RL method in the optimization problem to reach good solutions for NTSC. The results were better than the genetic algorithm and hill-climbing methods in low demand but could not outperform them in medium and high demands. (Tahifa et al., 2015) showed that Swarm Q-learning performs better than standard Q-learning in increasing the speed of TSC. To alleviate trac congestion and limit the e ects of incidents on trac ow, (El Hatri and Boumhidi, 2017) proposed a Q-learning based trac management model, which simultaneously optimizes vehicle re-routing and TSC based on the Multi-Objective Particle Swarm Optimization (MOPSO) method. hM2i: To provide a simple and ecient method to implement, (Natarajan et al., 2011) put into operation a func- tional gradient boosting approach to imitation learning in relational domains. The proposed approach outperforms both learning a single relational regression tree and performing a propositional functional gradient to represent the policy in all domains. To provide a solution to multi-objective problems with correlated objectives rather than typical multi-objective problems, (Brys et al., 2014) proposed an RL-based method combining multiple correlated rewards and shaping signals by measuring con dence (i.e. combining the feedback from all objectives, instead of only looking at a single one), called adaptive objective selection. They formally de ned a new class of multi-objective problems called correlated multi-objective problems (CMOP), whose set of solutions being optimal for at least one objective is so restricted that the decision-maker is least concerned about which of these is found, and more so about how fast one is found, or how well one is approximated. (Sadigh et al., 2014) proposed a method for synthesizing a control policy for an MDP such that traces of the MDP satisfy a control objective expressed as a linear temporal logic (LTL) formula through using an RL algorithm that nds the policy optimizing the expected utility of every state in the Rabin-weighted product MDP. They prove that the method is guaranteed to nd a controller that satis es the LTL property with probability one if such a policy exists, and they suggest empirically with a case study in trac control that their method produces reasonable control strategies even when the LTL property cannot be satis ed with probability one. (Prashanth and Ghavamzadeh, 2016) optimized variance-related risk measures in rewards and demonstrated its usefulness in an NTSC application. The risk-sensitive algorithms result in lower variance but higher long-term cost compared to their risk-neutral counterparts. (Prashanth et al., 2016) illustrated the usefulness of modeling human decisions by Cumulative Prospect Theory (CPT) paradigm in RL and suggested that CPT-based criteria is useful in a NTSC application. (Gan et al., 2019) proposed a dynamic correlation matrix based MARL approach where the meta-parameters are evolved using an evolutionary algorithm in a distributed manner. This was done to provide meaningful theoretical veri cation by using both agent-level implementation and system-level convergence veri cation. Agents using the proposed learning algorithm reach optimal behaviours faster than other canonical learning techniques. In the following, we introduce the types of RL methods used in the articles we included in this paper. Furthermore, we extracted data about other methods that an RL method has been integrated with to provide a solution, whether as a core or combined method. 3.2.2. RL methods Q-learning (Watkins and Dayan, 1992) ranks rst on the list of used RL methods in 96 (60%) studies. 13 papers do not provide any speci c RL method, especially when the goal of the paper is the proposal of a framework or when an RL concept is used. In this case, we used \RL" in Table 3. The remaining papers include methods that are listed (along with their frequency) in the rst column of Table 4. Furthermore, RL is the core method in 149 (93%) of the studies. In 11 studies, other methods were employed as the core method in which RL is used as a combined method, see the second column of Table 4. There are also other non-RL methods that were used in combination with the RL methods or frameworks, see the third column of Table 4. Below is a synopsis of the RL methods. •Q-learning and SARSA : Both Q-learning and SARSA (State, Action, Reward, State, Action) methods are critic-only where they use Q-tables to decide which action to take. The biggest di erence between the two is 18 Table 4: RL, core non-RL, and combined non-RL methods. RL method Core non-RL method Combined non-RL method Q-learning (96) Auction theory JTA No speci c RL (13) Fuzzy CPT paradigm Actor-critic (10) Fog Computing Self-Organizing Map Model-based RL (8) Pro-CEP Model-based and Bayesian optimization algorithms ADP (6) Self-adaptive NTSC system Transfer Learning W-learning (5) Cellular Automaton FRAP model design Multiple RL (4) Immune Network algorithm CTM Policy-Gradient (4) NN RMS-NE LCS (3) Multiband Max-plus SARSA (3) SensorGrid CRF TD (2) GA LA (2) Grey theory model Inverse RL (2) Qualitative method Continuous Residual RL (1) Evolutionary Alg. Risk-sensitive RL (1) Dynamic Clustering Alg. Neuro-Fuzzy JADE Coalition Fuzzy granulation Edge computing framework Max-sum message passing Alg. Swarm optimization that Q-learning is an o -policy RL method, while SARSA is an on-policy method. Q-learning and SARSA Q-table update equations are respectively presented below, where the policy is TD. •Critic-only, Actor-only, and Actor-Critic : The critic uses the calculated Q-values (or function values) to choose its action while the actor uses the policy to decide. Actor-only methods work to improve the policy. Actor-critic being a combination of both, allows both calculations of the Q-values and the policy to choose appropriate action. The papers that used an actor-critic method are identi ed in Table 5. •W-learning: W-learning is a multi-policy self-organising action-selection technique proposed in (Humphrys, 1996) that builds on Q-learning. In W-learning, there is a competition among sel sh Q-learners where agents learn Q-values for state-action pairs for each policy and W-values for each of the states of each of their policies to explore what happens if the nominated action is not followed. •Approximate Dynamic Programming (ADP): The computational complexity of DP algorithms due to excessive system state and their need for an exact algorithm and true value function makes the algorithm impractical in solving large-scale TSC problems. ADP aims not to fall into the predicament of computational complexity by replacing the true value function of the DP with an approximation function. In other words, it is similar to the model-based RL with function approximation. The research papers that used ADP method are identi ed in Table 5. •Policy-Gradient: The policy gradient method does not need to estimate the state or action value functions. It learns parameterized policy functions directly by searching policy space to maximize a measure based on the accumulated reward. In this way, it averts the convergence problems of estimating value functions. •Inverse RL : In Inverse RL (Ng, Russell, et al., 2000), the reward function of an agent (that the agent tries to optimize) is learned and determined by observing the agent's behaviour over time, the environment model, and the environment measurements. This approach is akin to learning from an expert and is helpful in the domains where the reward may not be easily accessible, like TSC (Natarajan et al., 2010). This method has its 19 origins in Imitation learning (also called as apprenticeship learning, learning by observation, or learning from demonstrations). It is comparable to supervised learning, with the key di erence being that the examples are not i.i.d, but instead, follow a meaningful trajectory (Bagnell et al., 2006). •Learning Classi er Systems: A learning classi er system (LCS) (Butz et al., 2005) is a rule-based RL system in which each rule (or classi er) is composed of a condition, an action, and a reward (or evaluation). LCS combines an evolutionary process (e.g. a genetic algorithm), with a learning process (e.g. RL), wherein a rule is constructed as fIF 'condition' THEN 'action' g. A genetic algorithm tries to improve condition-action rule space by generating new classi ers from current strong classi ers and removing the weak ones. RL is responsible for selecting the action with the best-rewarded response or evaluation to be executed. •Learning Automata: The action selection in learning automata is performed based on the last selected action and the received reward. A learning automata method forms from a vector of probabilities over the set of actions, which are updated (i.e. increase or remain the same) based on the reward. •Model-based and Model-Free : Model-based methods provide the agent with part or all of a model in which the agent must work. In model-free methods the agent develops its own model in which to work and has fewer restrictions. Essentially, in model-based methods the transition and reward functions are assumed to be available to construct a model, unlike in model-free methods where the agents do not need to have access to information regarding how the environment works. The papers that used model-based methods are identi ed in Table 5. 3.2.3. Control method scheme By analyzing the proposed methods in the included articles, we identi ed three levels of control, including (i) regular network control, which covers the general idea of controlling all trac signals in a regular network, (ii) perimeter control, which improves the performance of the entire network by controlling a part of trac signals that located on the boundary of a region in the network, and (iii) streetcar2bunching control, which mitigates the e ect of streetcar bunching along transit routes by controlling successive signalized intersections. The regular network control has been the focus of 158 out of 160 included articles. On the other hand, although there are numerous research works in the domain of perimeter or cordon control, the application of RL for perimeter control is studied, for the rst time, in 2019. (Ni and Cassidy, 2019) explored how RL can be used to re-time trac signals on the perimeter by developing an RL based controller with NN architectures that controls perimeter with spatially-varying metering rates. With regard to the third level of control, (Ling and Shalaby, 2005) conducted research to mitigate the e ects of streetcar bunching along transit routes through automating streetcar bunching control by means of multiple RL agents that act on a series of successive signalized intersections. The proposed method was able to e ectively split up a streetcar bunch and prevent it from forming again. 3.2.4. NTSC method: Centralization (centralized, hierarchical, and decentralized/distributed methods) When tackling multi-agent problems, there is a spectrum from centralized to decentralized decision making. At large scale implementation, multiple agents tackle the task given while communicating with each other. This usually results in quicker optimization as each agent learns from its neighbours as well as from itself (Tahifa et al., 2015). With multiple agents, the collected data and actions can be stored centrally in a location that all agents can access to function as one agent. In this setup, the central agent often makes all decisions for the system, and this can slow down the learning process while coordinating the unit (OroojlooyJadid and Hajinezhad, 2019). Although DNNs helps enhance the scalability of RL, training a centralized RL agent is still infeasible for large-scale NTSC. Conversely, a 2street railway or tram 20 Table 5: Identi cation of the included papers in terms of method identi cation, code, simulation, and evaluation. Category citation Method Identi cation Actor-critic Prashanth and Ghavamzadeh, 2016; Aslani et al., 2018a, 2017; Gen- ders and Razavi, 2020; Chu et al., 2019; Yang et al., 2019; Ni and Cassidy, 2019; Fagan and Meier, 2014; Nuli and Mathew, 2013; W. Lu et al., 2011; Zhang et al., 2007; Richter et al., 2007; Aslani et al., 2019; Dai et al., 2010 Deep Learning Wei et al., 2018; C. Li et al., 2018; X.-Y. Liu et al., 2018; Zhou et al., 2019; M. Xu et al., 2020; Shi and F. Chen, 2018; J. Lee et al., 2020; T. Tan et al., 2019; Horsuwan and Aswakul, 2019; Chu et al., 2019; Wei et al., 2019a; N. Xu et al., 2019; Rizzo et al., 2019b; Zheng et al., 2019a; Gong et al., 2019; D. Kim and Jeong, 2020; Yang et al., 2019; Ni and Cassidy, 2019; Ge et al., 2019; Shabestray and Abdulhai, 2019; Kitagawa et al., 2019; Van der Pol and Oliehoek, 2016; Rizzo et al., 2019a; P. Chen et al., 2019; Chu et al., 2016a; Vinitsky et al., 2018; Reda et al., 2019; Huang et al., 2019 Model-based methods Khamis and Gomaa, 2014; Kuyer et al., 2008; M. Wiering et al., 2004; Houli et al., 2010; Khamis et al., 2012; M. A. Wiering, 2000; Khamis and Gomaa, 2012; Da Silva et al., 2006 RL and GT Qu et al., 2020; L.-H. Xu et al., 2013; X. Zhao et al., 2009; Dae- ichian and Haghani, 2018; El-Tantawy et al., 2013; El-Tantawy and Abdulhai, 2010; Xinhai and Lunhui, 2009 ADP Fagan and Meier, 2014; T. Li et al., 2008b; Yin et al., 2015; Dai et al., 2010; Reda et al., 2019 Code, Simulation, and Evaluation Code is available Wei et al., 2018; Prashanth et al., 2016; Zhou et al., 2019; Genders and Razavi, 2020; Chu et al., 2019; Wei et al., 2019a; Zheng et al., 2019a; Wei et al., 2019b; Brys et al., 2014; Vinitsky et al., 2018 No Simulation W. Xu et al., 2015; Reda et al., 2019; Dowling et al., 2004; El-Tantawy and Abdulhai, 2010; Davarynejad et al., 2010; Xinhai and Lunhui, 2009 No Evaluation/Self-comparison W. Xu et al., 2015; Reda et al., 2019; Dowling et al., 2004; El-Tantawy and Abdulhai, 2010; Davarynejad et al., 2010; Xinhai and Lunhui, 2009; Dusparic and Cahill, 2009b; Prashanth et al., 2016; X.-Y. Liu et al., 2018; Prothmann et al., 2009 distributed approach can be used to store the local information around multiple agents, allowing each one to make its own decision while still communicating with its neighbours (H uttenrauch et al., 2019). In this approach, all agents would be considered \equal" (Baldazo et al., 2019). If the local agent does not communicate with other neighbor agents the system is called a decentralized system. Another setup that uses a combination of both centralized and decentralized/distributed methods is a hierarchical system, which can be categorized as a centralized method since it involves centralization. It forms a hierarchy of sorts, where the lower agents may have limited to no ability to enact upon the environment without permission from the \leader". A hierarchical control allows agents to perform micro-actions between tasks to improve the nesse of the agents. Most (65%) of the proposed methods in NTSC are designed in a decentralized way. There are 7 papers proposing holonic or hierarchical methods while the rest are centralized methods that might be rarely applicable in real-world scenarios in a real-time process. 3.2.5. RL methods' components and types State, action, and reward. There are normally three main components in RL: state, action, and reward. There are several elements that can de ne state and action in various papers. The elements of the state can be similar or di erent from those of the rewards. Based on our collected data of 160 studies, we identi ed 35 distinct elements of state and 30 of reward. The top 5 frequently used elements in state are queue size with 73 occurrences (38%), phase 21 Table 6: Categorization of types of trac signal based actions in the RL methods. The numbers in parentheses indicate the number of publications in which an action type is used. class cycle length phase duration phases order frequency cycle-/phase-based (#) 2 xed xed variable (2) cycle-based (48) 3 xed variable xed (13) 4 xed variable variable (4) 5 variable variable xed (18) 6 variable variable variable (11) 7 - variable - (100) phase-based (100) other action types class description frequency Act1 set the value of a threshold metric for each trac signal (2) Act2 set a link speci c metering rate in the perimeter control (1) Multiple Actionsincluding: ( Act3 ) set the value of a threshold metric for each traf- c signal, ( Act4 ) set a link speci c metering rate in the perimeter control, and ( Act5 ) select a route as a driver's action(3) state (11%), number of vehicles (10%), the position of the vehicles (6%), and speed (6%). In reward, the top 5 are queue size with 71 occurrences (30%), delay (13%), waiting time (9%), the number of vehicles (6%), and number of vehicles passed the intersection (or generally throughput) (4%). The elements of the state in 16 (5%) research papers and those of reward in 18 (7%) papers are not available. The list of all the elements of the state (38 unique elements) and reward (39 unique elements) found in the papers are available in Appendix A and Appendix B. This might help the new researchers get some idea in de ning these components. These two appendices also depict the most frequently used elements in states and reward, respectively. In addition to state and reward, action needs to be de ned. In the majority of the research papers action is de ned as a trac signal or phase switch. However, there are some that will use a di erent de nition than the signal switch. These actions include (Act 1) set the value of a threshold metric for each trac signal, (Act 2) a link speci c metering rate in the perimeter (or cordon) control, (Act 3) select a route as a driver's action, (Act 4) set the acceleration of the vehicles, and (Act 5) set the maximum speed of the vehicles. The last three options are used in a mixed environment where the vehicles are also considered as well as trac signals. When two or three types of actions were used, it was reported as \Multiple" in Table 3. The actions directly related to control trac signals are categorized into two main groups (phase-based and cycle-based), and seven classes where each class is de ned based on cycle length, phase duration, and phase order. Each of these three elements can be xed or variable. See Table 6. The decision point in cycle-based methods is the end of the cycle where cycle length, phase duration, or phase order are determined. In phase-based methods, the decision is made at the end of a phase, which includes phase duration determination and phase selection. In this case, phase duration can be set as xed for the entire phase or can be allowed to be extended at the end of the phase. Note that in phase-based methods, the cycle and phase order are not applicable. In Table 6, we removed class 1 in which all the three elements (cycle length, phase duration, and phase order) are xed. This is not applicable in RL design and is counter-intuitive to optimize. 62% of the papers proposed phase-based methods (i.e. class 7), while 32% constructed their methods on cycle- based methods. Only 2 papers focused on using RL, assuming that the phase duration is xed. Moreover, only 20 (12%) papers worked on applying RL in a xed cycle-length setup. There are also 3 papers with multiple actions or sets of actions, and 6 papers in which the action is not clearly or completely de ned. The details related to each paper are given in Table 3. 22 Action selection methods and parameters. To select the actions, various action selection methods can be used. Based on the data we retrieved, 71 (44%) papers did not state which action selection they used, which is a signi cant number.-greedy has the highest usage and was observed in 51 (32%) papers. Softmax (or Boltzmann) and greedy methods are used in 17 (11%) and 7 (5%) papers, respectively. Other action selection methods with a frequency of 4 or less are Random, Distributed W-learning (DWL), credit assignment algorithms, -softmax, and Upper Con dence Bound (UCB). Two papers employed multiple or combined action selection methods, while two others compared various action selection methods. •-greedy strategy and SoftMax :-greedy uses the epsilon term to balance exploration and exploitation of the environment, encouraging the former early on and switching to the later as the algorithm learns. It randomly selects actions for the next round based on the values of the exploration rate ( ), discount factor ( ) and learning rate ( ). SoftMax behaves similarly but with weight parameters either assigned or learned to each action. They can be quite sensitive to changes and encourage an outcome where important parameters have more value. The sensitive nature of the weights makes them tricky to learn/ nd and can a ect the performance of the algorithm signi cantly. We collected data about the RL parameters, including: exploration rate, discount factor and learning rate. We found that the information was not displayed in a number of cases. 121, 60, and 86 out of 160 papers did not reveal the information about these parameters, respectively. In papers where the data are presented, the most common option was for the authors to acknowledge that these parameters exist within the range of (0,1). While it is a piece of information, it is not especially useful seeing as this information is well known among common practitioners of reinforcement/machine learning. State space and action space discretization. Reducing state space (for the states such as queue size, ow rate, and density), action space, and even reward space is one of the ways that can be used to reduce computational cost to make the methods more applicable in real-time process in the real world. To this end, the continuous states are grouped, for example, in 3 levels: low, medium, and high. Another way is using the comparison between the states with the previous step. This way the space is divided into two groups: better or worse. Reducing the space using grouping the data may come at the expense of the lower accuracy of the results, thus lowering the eciency or optimality. Nonetheless, 59 (37%) of the papers used discretization while 91 (57%) did not discretize the state or action spaces. 3 (2%) papers used discretization only for the discrete methods, but not for the continuous methods that were evaluated. Tabular vs approximation-based methods. Another point that impacts the eciency of the performance of the meth- ods in real-world is using tabular RL methods where a look-up table is used to map the spaces. Approximation methods are used to provide a good approximation of the states that were not experienced in training. Interestingly, the number of the papers that used either of these two methods are eerily close: 74 (47%) papers used a tabular method while 72 (45%) used approximation methods. The approximation methods that are used in our pool of research studies are shown in Table 7. The applied neural networks, as one of the approximation methods, in the papers are also presented in the table. 38 papers used di erent neural network approximation methods, and the second-most used group is the Tile Coding, observed in 7 papers. One of these applied its own proposed method (Prashanth and Bhatnagar, 2010). •Neural Networks : NNs are loosely designed after the brain and are constructed for tasks such as pattern recognition, labelling, and processing of data. They are commonly used for clustering, classi cation, and predictions. In the case of trac control, prediction is the prominent use, though other use does occur. NNs consist of 3 main sections: input section, hidden section, and output section. Usually, there is only one of each input and output layers, but the hidden section may contain more. Each layer contains nodes. In the output layer, the number of nodes often corresponds with the number of possible outputs. In the input layer, 23 Table 7: First column) approximation methods used in RL-NTS, and second column) the types of NNs used in RL-NTSC. Approximation method Neural Networks types NN denoising auto encoder (DAE) tile coding Deep NN linear model tree feed forward NN (FFNN) simultaneous perturbation stochastic approximation (SPSA) recurrent NN (RNN) smoothed functional (SF) convolutional NN (CNN) phase gate deep convolutional network (Deep CNN) decomposable Q-function (DQF) perceptron radial basis functions (RBF) graph attentional network (GAN) triangular-shaped functions (TF) DNN linear function approximation (Linear FA) graph convolutional network (GCN) average cost algorithm incremental Gaussian mixture network (IGMN) iterative approximation back propagation NN (BPNN) KNN fuzzy NN (FNN) linear combination (LC) neural tted Q-iteration (NFQI) cerebellar model articulated controller (CMAC) wavelet NN (WNN) connectionist Q-learning framework (CQF) gradient-descent linear function approximation double deep Q-learning fuzzy granulation linear regression (LR) sigmoid regression (SR) long short-term memory (LSTM) linear Q-function the number of nodes often corresponds with the di erent types/sources of input data. The number of nodes within the hidden layers depends on what each layer is designed to do, as well as changes with the purpose of the network. There are many types of NNs in deep RL. {Arti cial Neural Networks (ANN): Known as a feed forward network, all incoming information is only processed and pushed in the forward direction. {Recurrent Neural Networks (RNN): Unlike ANN, this neural network pushed processed data back to previous layers and nodes. It shares parameters across di erent time steps and results in fewer overall parameters. The fewer parameters allows for a smaller network. {Convolutional Neural Networks (CNN): These networks use an extra step called convolution (for which it was named), which involves applying di erent " lters" to reduce the complexity of the input data. As the information lters through the network, these lters can be applied to highlight speci c features of the data. This is one of the most common neural networks and is used in many disciplines. With neural networks, deep RL becomes highly suited for complex environments presented by intersections and the dynamic changes that occur within a day of trac. •Tile Coding : Tile coding is another well-known function approximator. Unlike the continuous methods such as radial basis functions (RBF), tile coding is a discretization method which is used in RL. It is a piece-wise constant approximation method that approximates the action-value functions by partitioning the state space into small regions with a constant reward value. Tilings design considers three main components: width of tiles, the resolution, and the number of required tilings based on the hyper-volume of the whole state space. For more information about Tile coding method, the reader can refer to (Abdoos et al., 2014). Deep reinforcement learning based methods. Deep RL takes a di erent approach when dealing with the complex in ux of data associated with trac. It incorporates neural networks into RL algorithms and combines the advances 24 in training layered neural networks into abstract high-level representations of the raw input data, giving non-linear methods (El-Tantawy et al., 2015), (C. Li et al., 2018). These \layers" allow the agent to look at smaller, more reduced versions of data to extract information without an overload. The neural networks are what allow an algorithm to go \deep" and work with large or complex data input more eciently. Here, we compare Q-learning and Deep Q-learning. Q-learning, is the process of iteratively updating Q-Values for each state-action pair using the Bellman Equation until the Q-function eventually converges to Q. Instead of estimating the Q-value of each state-action pair separately in Q-learning, deep reinforcement learning algorithms (Mnih et al., 2013) use deep neural networks as function approximators to map from states to Q-values. This makes possible the use of a larger and/or continuous state space through parameterization (Lillicrap et al., 2015). The integration of arti cial Neural Nets (NNs) into the Q-learning process is referred to as Deep Q-learning, and a network that uses NNs to approximate Q-functions is called a Deep Q-Network (or DQN). In other words, DQN is a Q-learning, which is parameterized with deep NN with parameters , i.e.Q(s;a;). The neural network input is the state, the number of output neurons is the number of the possible actions, and the targets are the Q-values of each of the actions. Unlike Q-learning, whose convergence in the limit (in nity) is guaranteed, we do not have such guarantees for DQN. This is because (i) the data set is not i.i.d. and (ii) as the agent learns, the targets move (Van der Pol and Oliehoek, 2016). To ameliorate this issue, di erent methods can be used, such as dueling architecture (Ziyu Wang et al., 2016) to improve stability and target network(Mnih et al., 2015) to solve the overoptimistic problem. Dropout (Srivastava et al., 2014) can also be used to make the controller more robust and prevent the neural network from over tting. Learning rate needs to be optimized in training deep neural networks. To this purpose, di erent optimization methods can be used, such as stochastic gradient descent, Adagrad, and Rmsprop. In this line, Adam optimizer (Kingma and Ba, 2014) is an adaptive learning rate optimization algorithm, which is computationally ecient, has little memory requirements, and is generally fairly robust to the choice of hyper parameters (Goodfellow et al., 2016). In addition, experience replay is also used to help with stability and convergence behaviour of the algorithm when using a non-linear function approximator. Deep learning was used in 27 (17%) papers. This is certainly a reasonable portion of the study, and the point to be noted is that using deep learning in RL in the network-scale began in 2016 with 2 papers. After a year with no cases, deep learning was used in 5 studies in 2018, followed by a sharp increase in 2019 with 18 papers. It is expected that this trend will continue into the future, as in the rst season of 2020, 3 out of 6 research papers used deep learning. The papers that used deep RL methods are identi ed in Table 5. SUMO has the frequency of 16 (out of 28) in conducting simulations for the deep learning related methods. The trac simulation software tools used for deep learning are depicted in Figure 9. 3.3. Environment attributes and trac simulation In this section, we discuss the environment attributes, including network type, vehicle class, data source, and data communication processing. This classi cation presented in Figure 8. 3.3.1. Simulation and simulated networks 16 trac simulator software or platforms are identi ed in these studies to simulate the trac. These include SUMO3, VISSIM4, PARAMICS5, GLD6, AIMSUN7, ITSUMO (Silva et al., 2004), CityFlow8, TRANSYT9/TRANSYT- 3https://www.eclipse.org/sumo/ 4https://www.ptvgroup.com/en/solutions/products/ptv-vissim/ 5https://www.paramics.co.uk/en/ 6http://www.sf.net/projects/stoplicht 7https://www.aimsun.com/ 8https://city ow-project.github.io/ 9https://trlsoftware.com/products/junction-signal-design/transyt/ 25 Regular network controlPerimeter controlCar bunching controlRL Method proposal in NTSCTheoretical contribution in RL with feasibility assessment in NTSCRL alone, RL + ML/GT/DPRL + other methodsMETHOD ATTRIBUTES DATA SOURCENETWORKVEHICLE CLASSENVIRONMENT ATTRIBUTESCOMMUNICATION PROCESSNetwork of intersectionsArterial networkSignalized roundaboutPrivate vehicles (cars)Public transitEmergency vehiclesGeneral detection devicesLoop detectorsCAVs, V2V, V2I, I2ICamera/imageNo process on imageImage representationImage from simulationEdge, Fog, Cloud computingCONTROL METHOD SCHEMEMETHOD CONTRIBUTIONRL METHOD APPLICATIONNTSC METHODCentralizedFully centralizedHierarchicalDecentralizedDistributed(a) (b)Methods from other fields in RL RL in methods from other fields or in a specific design for NTSCRL in optimization, optimization in RLStreetcar bunching control/Distributed Figure 8: Environment attributes in RL-NTSC. 7F10, MATLAB11, AIM (Dresner and Stone, 2008), TSIS12, MATISSE (Torabi et al., 2018b), USTCMTS (used in (Shi and F. Chen, 2018)), SMPL (used in (Su and Tham, 2007)), and SeSAm (used in (Bazzan et al., 2007)). See the distribution of frequency of the trac simulators in included papers in Figure 9. Since the source of the last three simulation software tools is not found, we referred the reader to the papers in which they are used, for further information. Of interest is that SUMO was used for the rst time in 2015 and has since become the most frequently used software in this area, with 17 out of 27 occasions in 2019 alone. The second most frequently used software, (i.e. VISSIM) started being used in 2010, and PARAMICS, the rst well-known trac simulator has been in use since 2003. Prior to this the tools were either custom-built or not clearly outlined in the research. In 18 studies, the authors designed or used a custom-built environment for simulations. What is surprising is that 27 studies (16.7%) did not state the simulation tools that they used for the test, not including the those that did not use any simulations. With regard to the type of maps used to test the proposed RL methods by the included studies, of those that used maps, 100 (62%) papers used a synthetic map, 45 (28%) papers used a real-world map, and 9 (6%) papers used both types, see Figure 10. Moreover, 49 real maps are used from 14 countries for simulation purposes, with China topping the list at 10 maps, followed by the USA, Ireland and Canada, see Figure 10. These maps are mostly large-scale networks of intersections. Figure 11 shows the distribution of the number of intersections studied. Though many studies are still currently being held with fewer than a dozen intersections, there is also work on 25+ intersections. And while 44% of the papers run simulations in small-scale networks with 8 intersections or lower, we observed that the recent trend is to test the proposed methods in medium and large-scale networks. The size of the network is also important as it may impact the exponential growth of state-space and action-space and generally the complexity and computational e ort required to reach a solution through an RL method, whether for training or testing stages. We categorized the type of the testbed network into three main groups, including: (1) the network of intersections, (2) arterial network, and (3) the signalized roundabout networks. As already mentioned, the isolated intersection case is excluded from our study. The arterial network is an open network, as compared to a closed network. Arterial 10https://mctrans.ce.u .edu/mct/index.php/hcs/transyt-7f/ 11https://www.mathworks.com/products/matlab.html 12https://mctrans.ce.u .edu/featured/tsis/Version6/index.htm 26 SUMOVISSIMCityFlowAIMSUNPARAMICSUSTCMTSCustom-built toolNA/No traffic simulationNumber of publications0481216 3 1 1 1 1 2 3 16SUMOVISSIMPARAMICSGLDAIMSUNITSUMOCityFlowTRANSYT/TRANSYT-7FMATLABAIMTSISMATISSEUSTCMTSSMPLSeSAmNumber of publications010203040 1 1 1 1 2 2 2 2 3 4 6 13 15 19 37Traffic simulatorTraffic simulator (a)(b)Figure 9: Trac simulators: a) distribution of frequency of the trac simulators in RL-NTSC, and b) distribution of frequency of the trac simulators in deep RL-based methods in RL-NTSC. network control is applied on a sequence of intersections to provide preference to progressive trac ow along the arterial. Unlike the isolated intersections, the intersections in the arterial network operate as a system and the system coordinates timing of adjacent intersections. On the other hand, the network of intersections is considered as a closed loop. Being a closed loop, the network demands at least four intersections. The networks of two and three intersections may have the characteristics and behaviour of both groups (the network of intersections and arterial network); therefore, we considered them separately as two sub-groups under the rst group, i.e. the network of intersections. Finally, the signalized roundabout networks deals with both approaching and circulatory lanes, which are fundamentally di erent from the rst two groups. The signalized roundabout networks involve more than one set of trac signals in a node. (Rizzo et al., 2019b,a), both published in 2019 are only two papers in the literature that studied the signalized roundabouts, which indicates a potentially open research problem on the RL area. The dynamics of signalized roundabouts is complex because it deals with both approaching and circulatory lanes. The con icts between approaching and circulatory ows cannot be solved by metering only the approaching lanes and reacting consequently because the circulatory lanes may be occupied. (Rizzo et al., 2019b) proposed a deep RL method for signalized roundabouts in congested networks to maximize trac ow while being able to avoid trac jams in connected junctions. In another research conducted in this domain, (Rizzo et al., 2019a) studied the possibility of deriving explanations from a neural network agent (trained using Policy Gradient) for TSC in a signalized roundabout. How the agent learns to react di erently based on each speci c lane's trac by implicitly predicting the route of the trac and thus its future circulatory occupancy is explored. This is done by analyzing the relation between the agent phase p