# Traffic Signal Time Optimization Based on Deep Q-Network
## 3 Problem Formulation
### Proposed DQN-Based Traffic Signal Control  
The traffic signal problem of an intersection should be addressed by considering the characteristics of the intersection, which are dynamic and unpredictable. Traffic conditions change continuously depending on the time, day of the week, or weather. Intersections constantly face new environments. Moreover, the intersection has a continuous structure. The environment is computationally complex because it is affected by the circumstances in the surrounding intersections. In RL, the environment is modeled as an MDP. An MDP is defined as a state, action, or reward. To optimize the signal control, we need to accurately recognize the current state in the MDP. In addition, it is necessary to recognize the situation in the surrounding intersections for cooperation. The state includes six pieces of information: the first and second are the information about the total traffic load (\(i_{1}\)) and the standard deviation of the traffic load (\(i_{2}\)) at the intersection, respectively. The combination of these two parameters provides a detailed representation of the situation at the intersection. For example, if the traffic load and its standard deviation are both small, there are not many vehicles at the intersection in general. However, a small traffic load with a large standard deviation indicates that, despite few vehicles being at the intersection, there are blocked lanes. Moreover, a large traffic load means that the intersection suffers from traffic congestion regardless of the standard deviation of the load. The third (\(i_{3}\)) and fourth (\(i_{4}\)) pieces of information denote the traffic load of the two directions that will receive the next green light. The fifth (\(i_{5}\)) and sixth (\(i_{6}\)) pieces of information are regarding the traffic load in the directions where the vehicle will exit from the two directions that will receive the next green light. The proposed model provides green signals in two directions according to a traffic signal order. Figure 2 shows the movement information when the vehicle receives a green signal. The intersection in Figure 2 is divided into two directions per road. One is the left-turn direction, and the other is the straight direction where right turns are possible. For example, if the next green signal is scheduled to be assigned the northbound straight or the right-turn direction (\(l_{1}\)) and left-turn direction (\(l_{2}\)), the vehicles waiting in \(l_{1}\) will flow in the direction \(l_{3}\) and the vehicles waiting in the \(l_{2}\) direction will flow in the direction \(l_{4}\). In this case, traffic load information in directions \(l_{1}\) and \(l_{2}\) is \(i_{3}\) and \(i_{4}\), respectively, and that in directions \(l_{3}\) and \(l_{4}\) is \(i_{5}\) and \(i_{6}\), respectively. The information regarding the directions where the vehicle flows in when the green signal is on is needed because the number of vehicles that want to enter from a certain direction depends on the traffic load in the outgoing direction. This is because no matter how many green signals are allocated to the direction receiving the green signal, the number of vehicles that can enter is limited if there is a high traffic load in the directions that they are trying to enter. Therefore, if the time for a green signal is to be efficiently allocated, the traffic conditions in the surrounding intersections where the vehicle is about to enter should also be recognized. Therefore, the state of the MDP recognizes the current situation at the intersection with the following six pieces of information: \(s_{t}=[i_{1},i_{2},i_{3},i_{4},i_{5},i_{6}]\). The proposed optimal TSC system adjusts the time of the traffic signal with the action of the MDP to efficiently use the green signal. Therefore, it is important to distribute the green signal to the required lanes at the appropriate time. An efficient green time allocation can maximize the overall capacity of the intersections.