## 4 RL Method Selection  
Previous studies often specified their RL method without detailed motivations. Although it is impractical to experimentally investigate all RL methods, here we qualitatively analyze different kinds of methods and pick out a well-suited one for the single-ring RISC problem. Figure 1: Common options for the single-ring phasing. \(\Phi\) represents vehicle phase number, solid one-way arrows represent protected vehicle movements, and Solid two-way arrows represent pedestrian movements. Subfigure (**a**) shows the leading left-turn phasing, and (**b**) the lagging left-turn phasing. Single-agent methods versus multi-agent ones. The nature of RISC problems is to cyclically determine the green time for each vehicle phase at an intersection. As required by the single-ring phasing, different vehicle phases are prohibited from displaying green simultaneously. It means that there are no concurrent decision-making processes. Hence, a single agent deployed for the intersection is capable enough of dealing with the single-ring RISC problem. 2. Model-free methods versus model-based ones. A road user's behavior and other road users' reaction to his behavior are both stochastic. All the road users at an intersection have a highly stochastic impact on the state transitions and the resulting rewards. As a consequence, the intersection's dynamics are too complicated to be thoroughly modeled by human beings. Therefore, model-free RL methods are more sensible options for the single-ring RISC problem. In this way, the agent relies not on prior models but on massive trial-and-error experiences to predict the states and rewards. 3. Value-based methods versus policy-based ones. Regardless of the actions defined, the action space is obviously discrete. In contrast to other action options, choosing the green time for current vehicle phase often leads to a larger action space. Even so, there are only dozens of actions in it. In any case, the agent faces a discrete and small action space, which can be effectively handled by value-based RL methods. 4. Off-policy methods versus on-policy ones. We believe the policy being learned to solve the single-ring RISC problem is greedy. That means a state is mapped onto a deterministic action rather than a probability distribution over actions. However, the policy of behavior during training should be stochastic to ensure sufficient exploration of state-action space. This justifies the choice of off-policy RL methods, in which the learned policy naturally differs from the behavior policy. 5. Deep RL methods versus Tabular ones. The high-resolution vehicle data enable us to build a large continuous state space for the single-ring RISC problem, which can be handled only by approximate RL methods. DNNs are seen as universal function approximators [43]. The deep RL methods, using DNNs to approximate value function, minimize the effort of handcrafting state features and create an opportunity to apply RL in an end-to-end manner. DQN is an RL method that can learn to play many distinct Atari video games at human-level performance [4]. Among the prevailing RL methods, only DQN and its variants simultaneously satisfy the aforementioned requirements on solving the single-ring RISC problem, i.e., being a model-free, value-based, off-policy, and deep RL method. Most variants of DQN address the cases that exist only in particular RL tasks. However, they are usually not our case. For example, Dueling DQN is specific to the cases where actions do not significantly affect the environment in some states [44]. However, in our case, the agent decides the green time for phases. An improper action can cause obvious green starvation or overflow queue. Prioritized DQN is specific to the cases with sparse rewards [45]. However, in our case, the reward is the number of vehicle departures and is fairly dense. It is reported that these variants unexpectedly under-performed the original DQN in some tested tasks.