# Intelligent Traffic Signal Phase Distribution System Using Deep Q-Network
## 2 Related Works  
Recently, reinforcement learning method is widely applied to traffic signal control and optimization to model and adapt to the dynamic traffic environment. Traffic conditions are unpredictable. Therefore, the traffic signal control model should be able to improve itself through learning. Reinforcement learning proceeds with learning based on past experiences and selects the optimal behavior to achieve its goal. Applying reinforcement learning to traffic signal control research enables flexible traffic flow management in unpredictable and dynamic environments. There are two main ways through which a traffic signaling system controls the flow of vehicles at an intersection: determining the direction in which the vehicles will proceed [7, 8, 9, 10] or determining the duration of the green light [11, 12, 13, 14, 15]. Traditional traffic signals have a predetermined phase for each intersection. Here, the phase indicates the direction in which the green signal is assigned. In the method of determining the preceding signal, the phase applied to receive the green signal is flexibly determined. Depending on the situation, the vehicle in the most necessary phase direction passes through the intersection. Meanwhile, another method determines the optimal duration of a green light phase for receiving the next green signal in a predetermined phase order. In [12], there are only two phases, and thus it is determined whether to continue or change the signal. This enables an efficient signal time distribution. The model proposed in [7], which determines the phase order, minimizes the aggregation of the total time spent in all vehicle queues at a single intersection using a transfer RL. In [8], a vehicle is controlled at multiple intersections with the aim of minimizing the queue length. A Q-value transfer strategy using the DQN was applied to reflect on the status of the surrounding intersections. The Q-value of the surrounding intersection is reflected in the Q-value update of the target intersection to recognize the status of the surrounding intersection. The studies in both [9,10] proposed traffic signal control methods to minimize the wait time at multiple intersections. In [9], the road network features are extracted using GCNN (graph convolutional neural network), and the model is optimized by using a neural fitted Q-iteration. The total travel time is considered as a reward parameter. In [10], the proposed DQN model can be extended from a single agent to multiple agents through transfer learning. Various factors such as travel time, waiting time, delay, and emergency stop are considered reward parameters. Therefore, it is difficult to apply in a real environment. In [11], to determine the duration of a green signal, the traffic signal control system is proposed by using a deep recurrent Q-network technique that combines a recurrent neural network (RNN) and a DQN. In [11], the action of the model determines whether to extend the duration of the current green phase or not. The total number of the halting vehicle is considered as a reward parameter. The aim is to minimize the total number of waited vehicles by establishing a Q-network with long short-term memory, which is a type of RNN. However, because it is based on a single intersection environment, it is necessary to expand to a multi-intersection environment for optimized signal control. In [12], in order to accelerate the learning, a simple RL model was proposed by using the average length of the queue as a reward parameter. In [13], the authors controlled the signal to minimize the average wait time of the vehicles using RL. It provides the Q-value of the target agent to the surrounding agents. There are two traffic signals consisting of a green signal and a red signal. When the lane is allocated a green signal, the opposite lane is allocated a red signal. Accordingly, the traffic signal determines the duration of the green and red signals. In [14; 15], the authors proposed a traffic signal that determines the duration of a green signal.