# Traffic Signal Optimization for Multiple Intersections Based on Reinforcement Learning
## 2 Literature Review  
Reinforcement learning is the most recently used algorithm in the field of signal control research. However, most studies have not considered the constraints applied to a real-world intersection or tested in a local area such as a single intersection. [PERSON] et al. (2017) analyzed the possibility of using the Q-Learning algorithm for adaptive traffic signal control [12]. The Q-Learning algorithm was helpful in resolving traffic congestioncompared to the fixed-time model, but different results were obtained depending on the definition of reward and various traffic volumes. [PERSON] et al. (2019) proposed a DQN-based signal control model ([PERSON] et al., 2019). The state was defined as a grid-type location and the speed of vehicles around the intersection. The reward was the difference in accumulated waiting time between learning cycles, and the action was to select one of phases. [PERSON] et al. (2019) proposed a model based on the assumption that data are collected by a loop detector ([PERSON] et al., 2019). Since the state used as input to the model is not a data format that can be obtained, a method for converting the data acquired through the detector into data useful for the model was presented. [PERSON] et al. (2019) proposed a cooperative learning method in which signal controllers at adjacent intersections share a state they can observe with each other ([PERSON] et al., 2019). Using this, a traffic signal optimization model for multiple intersections was proposed.  
[PERSON] et al. (2019) pointed out the limitations of the centralized reinforcement learning model and suggested a way to optimize a large-scale road network by placing the model at each intersection ([PERSON] et al., 2019). As the algorithm of the model, A2C (Advantage Actor Critic) was proposed, and the state of different scales was delivered to the model with city-unit traffic volume data and actual observable traffic flow data. [PERSON] et al. (2020) pointed out the limitations of the realtime response of the existing adaptive signal control method and suggested a reinforcement learning-based signal control model as an alternative ([PERSON] et al., 2020). Efficiency was evaluated through various indices for the compensation that is judged to have the greatest impact on the model's performance. [PERSON] et al. (2020) introduced a multiagent-based reinforcement learning algorithm ([PERSON] et al., 2020). The model was designed to solve high-level problems such as dynamic traffic volume through cooperation between agents. As a result of the simulation, it was shown that the travel time was reduced through the proposed model.  
In general, traffic signal control has constraints such as the sequence of phases is fixed, and the minimum green time is given. However, in related studies, the constraints are ignored in consideration of only mathematical optimization such as delay minimization ([PERSON] et al., 2020; [PERSON] et al., 2019; [PERSON] et al., 2019). If the phase sequence is random or does not give a minimum green time, this can cause over-waiting for vehicles and confuse drivers ([PERSON] et al., 2019; [PERSON] et al., 2019). In this study, we propose a model that can compensate for the problems that arise when the reinforcement learning-based signal control models proposed in related studies are applied to actual road networks. The proposed model maintains the same signal order as the fixed-time model and gives the minimum green time. Therefore, it can be applied to the actual signal controller. The performance of the proposed model was evaluated in a simulation environment depicting real roads connected with multiple intersections. In order to evaluate the effect of constraints, a comparative simulation between the proposed model and the reinforcement learning model that did not consider the constraints was performed.