# Traffic Signal Time Optimization Based on Deep Q-Network
## 2 Related Works
Their proposed model defines the queue length parameter in the same sense as the travel time of the vehicles. Therefore, to minimize the travel time of the vehicles, they defined the reward as the average of the queue length. In [13], the authors used a Q-value transfer strategy using a DQN to reflect the situation in the surrounding intersections. In other words, the Q-value of the surrounding intersections is reflected in the Q-value update to balance the traffic with the surrounding intersections. Therefore, the impact on the traffic conditions around the intersections can be considered. However, the approach in [13] can continuously distribute green signals to the same lane in the signal distribution. There is a concern regarding a selfish signal distribution, which assigns green signals to only one lane where there are many vehicles. In [18], the authors proposed a traffic signal timing plan that allocates the length of time. The goal is to minimize the queue length at the intersection and optimize the traffic flow. In addition, allocation of excessive green time is penalized. The experiment in [18] was conducted at multi-intersections. Because information on surrounding intersections is shared only through global optimization, the impact on learning is a concern. In [11], the authors proposed an RL model with the goal of minimizing the total travel time. Using a transfer network, the RL model speeds up the learning through the transfer of the accumulated knowledge and it can also be learned with less data. However, only a straight green light is possible at the intersection. In other words, various green signals, such as the left direction or right direction, are not considered. In addition, only a single-intersection environment is considered. In [12], the authors minimized the aggregation of the total time spent in the queue of all vehicles using transfer RL. In [12], the Markov decision process (MDP) considers an isolated four-way intersection; therefore, if the intersection structure changes, the MDP must be modified accordingly. In addition, in [14], the authors proposed an RL model using neural fitted Q-iteration (NFQI). The aim of this NFQI-based RL model is to minimize the waiting time. The traffic environment changes dynamically over time. However, it does not reflect the traffic conditions on the time-domain flow. In [15], the authors proposed signal control in a multi-intersection environment to reduce the travel times. It selects which lanes get a green light. It can be extended from a single- to a multi-agent DQN through transfer learning. In [16], the authors proposed a method for controlling the flow of vehicles by changing the timing of the green and red traffic lights using RL. The goal is to minimize the average waiting time of the vehicles. It considers the surrounding intersections by giving their Q-value to the intersection agent next to the intersection agent directly sending the vehicles to their intersection. In this way, the intersection identifies the surrounding situation and controls the traffic flow. A limitation of the method in [16] is that the signal is simple: only green and red lights can be controlled. In [17], the authors proposed a TSC technique that allocates green signals to the lane that needs them to reduce the total travel time. In addition, because the length of the green time can be adjusted as necessary, customized signal control according to the situation is possible. However, it can be very complex depending on the number of signal orders. Accordingly, high complexity is a concern. Other studies have described the states of intersections using 2N-dimensional feature vectors [11; 12; 14]. An intersection is represented as a 2N-dimensional feature vector; if a vector element has a vehicle in that position, its value is set to 1; otherwise, it is set to 0. In [11; 12], studies that consider a single intersection are discussed; however, there are differences in what the 2N-dimensional vector represents.