# Traffic Signal Time Optimization Based on Deep Q-Network
## 2 Related Works  
Many TSC approaches have been proposed in the field of traffic research, and there have been many attempts to implement a signal policy that depends on the traffic conditions. The characteristics of the intersections should be identified for the appropriate signal control. First, intersections are structures that affect each other. Vehicles pass through neighboring intersections and affect them. Second, the intersection is an unpredictable dynamic environment that varies over time or the day of the week. Intersections always face new environments. Therefore, it is not efficient to control traffic flow at the intersection with fixed traffic signal control. Moreover, traffic light control cannot be learned through supervised or unsupervised learning. This is because it is impossible to predict and set the answers to all situations pertaining to what actions should be taken under all situations. Because of the characteristics of an intersection, it is difficult to model the environment; therefore, most studies use RL [7; 8]. TSC studies using RL are largely categorized according to whether the situation in the surrounding intersections is considered. A study of TSC at an isolated intersection that does not consider the circumstances in the surrounding intersections has the following advantage: it can determine more accurately and in greater detail how to implement a signal policy at a single intersection [9; 10; 11; 12]. Therefore, it allows the implementation of the best policy to determine the traffic conditions at an isolated intersection. However, it is impossible to cope with events that occur in the surrounding intersections because the situation there cannot be determined. A TSC study that considers the situation in the surrounding intersections has the advantage of being able to implement policies for various situations, including those in the surrounding intersections [13; 14; 15; 16; 17; 18]. However, its disadvantage is that the calculation complexity is high because there are many situations to consider, not just those in the target intersection but also in the surrounding intersections. Reinforcement learning implements a policy by obtaining the information about the current traffic conditions in an intersection as a state and maximizing the rewards through interactions with the environment. The traffic signal performance depends on those parameters, which are considered as rewards. RL studies provide a variety of results depending on the design method, including the state, action, reward, and algorithms. In this paper, related studies fall into two groups: those that used the queue length and those that used the waiting time as the reward parameter. The queue length indicates the number of vehicles waiting at the intersection. This represents the state of the intersection in the spatial domain. The waiting time indicates the average amount of time the vehicles in the intersection have waited to exit the intersection. This represents the time domain of the vehicle. In general, the queue length is considered as a reward if the aim is to minimize the number of vehicles at the intersection [9; 10; 13; 18]. In contrast, the waiting time is considered as a reward if the aim is to minimize the waiting time of the vehicles at the intersection [11; 12; 14; 15; 16; 17]. In [9], the authors proposed a deep recurrent Q-network (DQRN) technique that combines a recurrent neural network (RNN) with a deep Q-network (DQN) to learn various traffic environments. The DQRN technique builds a Q-network with long short-term memory (LSTM), which is an RNN. The objective of the DQRN is to minimize the total number of waiting vehicles before the stop line at the time unit. It has the advantage of considering not only the current state information but also a series of previous states to better distinguish the environment. In [10], the authors proposed a simple RL model that is less complex to speed up the learning.