## 4 Design of the DRL Algorithm for the Traffic Signal Control
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Paper** & **Year** & \begin{tabular}{c} **DRL** \\ **Method** \\ \end{tabular} & \begin{tabular}{c} **Number of** \\ **Intersections** \\ \end{tabular} & \
\begin{tabular}{c} **Compared** \\ **against** \\ \end{tabular} &
\begin{tabular}{c} **Control** \\ **Strategy** \\ \end{tabular} & **Improvements** \\ \hline
[17] & 2019 & \begin{tabular}{c} Regional A3C \\ + PER 2 \\ \end{tabular} & 42 & \
\begin{tabular}{c} Hierarchical \\ MARL, \\ Rainbow DQL, \\ Decentralized \\ multi-agents 1 \\ \end{tabular} & 43 &
\begin{tabular}{c} Acyclic TSP \\ selection and \\ TSP duration \\ \end{tabular} & 8.78\% lower \\ \hline
[9] & 2019 & Stabilised IA2C 3 & 30 & \
\begin{tabular}{c} IA2C, \\ IQL-LR, \\ IQL-DNN \\ \end{tabular} &
\begin{tabular}{c} Acyclic TSPs \\ with fixed \\ duration \\ \end{tabular} & 63.7\% lower \\ \hline
[34] & 2019 & \begin{tabular}{c} Hierarchical \\ regional A2C 6 \\ \end{tabular} & 24 & \
\begin{tabular}{c} Regional \\ DRL \\ \end{tabular} &
\begin{tabular}{c} Acyclic TSPs with \\ fixed duration \\ \end{tabular} & 44.8\% lower \\ \hline
[46] & 2019 & 3 DQN + PER & 1 & \
\begin{tabular}{c} Actuated and \\ fixed time \\ controller \\ \end{tabular} &
\begin{tabular}{c} Acyclic TSPs with \\ fixed duration \\ \end{tabular} & 10.1\% lower \\ \hline
[18] & 2018 & \begin{tabular}{c} ResNet \\ based \\ A2C \\ \end{tabular} & 9 & \
\begin{tabular}{c} Actuated \\ controller \\ \end{tabular} &
\begin{tabular}{c} Cyclic fixed \\ TSP switch \\ \end{tabular} & 16\% lower \\ \hline
[20] & 2017 & DDPG & 43 & \
\begin{tabular}{c} Q-learning \\ algorithm \\ \end{tabular} &
\begin{tabular}{c} Cyclic TSPs \\ with computed \\ duration \\ \end{tabular} & No data \\ \hline
[21] & 2017 & DDQN + ER & 1 & \
\begin{tabular}{c} LQF, \\ fixed time \\ controller \\ \end{tabular} &
\begin{tabular}{c} Cyclic TSPs \\ with fixed \\ duration 4 \\ \end{tabular} & 47\% lower \\ \hline
[16] & 2016 & DDQN + ER & 1 & STSCA 5 &
\begin{tabular}{c} Acyclic TSPs \\ with \\ intermediate TSPs \\ \end{tabular} & 82\% lower \\ \hline
[14] & 2016 & DeepSAE + RL & 1 & \
\begin{tabular}{c} Q-learning \\ algorithm \\ \end{tabular} &
\begin{tabular}{c} Two TSPs with \\ dynamic \\ duration \\ \end{tabular} & 14\% lower \\ \hline \hline \end{tabular}  
* The decentralized multi-agent approach is based on distributed constraint optimization. 2 Multi-step return and Off-policy A3C. 3 It is stabilized by fingerprinting technique and spatial discount factor. 4 The intermediate TSPs are dependent on the current phase and the chosen actions are used due to safety reasons. 5 It is compared to Shallow Traffic Signal Control Agent (STSCA) based on shallow ANN. 6 Each Agent uses _Wolertinger_ Actor–Critic MLP architecture and DDPG algorithm.  
\end{table} 
Table 1: The most representative DRL frameworks for ATSC design. be constructed based on three distinctive traffic models: microscopic, mesoscopic, and macroscopic. Macroscopic models compute only aggregated macroscopic traffic parameters at a high level [47]. Since those models cannot provide raw traffic data (e.g., position of each vehicle in network) they are not suitable for DRL. Simulators based on microscopic traffic models such as Simulation of Urban Mobility (SUMO), PARAMICS [14], Aimsun, and PTV VISSIM simulate each vehicle behavior independently. These simulators can be used with DRL based algorithms since they provide raw data for each vehicle during the whole simulation run. The SUMO is the most used microscopic simulators applied for ATSC algorithms based on DRL [17; 21; 48; 49]. Its DFROUTER tool can heuristically compute a list of routes and a list of vehicles associated with each route according to the network-wide real-world traffic data [47]. The mentioned tool enables resembling the real traffic distribution which can increase the simulation accuracy and consequently provide better learning convergence of the DRL frameworks applied for ATSC.