# Traffic Signal Time Optimization Based on Deep Q-Network
## 3 Problem Formulation
### Proposed DQN-Based Traffic Signal Control
The proposed model controls the time of the green signals; moreover, it is assumed that the lane order in which the green signal is assigned is fixed, as shown in Figure 3. Therefore, the action is defined as the duration of the green signal time. We define the time unit as \(\sigma\); the action of the proposed model is defined as follows: \(a_{t}=[\sigma_{1},\sigma_{2},\sigma_{3},\sigma_{4},\sigma_{5},\sigma_{6},\sigma _{7}]\). The number of lanes varies depending on the structure of the intersection; the number of lanes assigned with green lights also varies. Figure 4 shows the structure of three- to six-way intersections. As can be seen in Figure 5, there can be three possible signal directions at a three-way intersection and eight at a four-way intersection. The five- and  
Figure 3: Order of the green signal assignment. Figure 2: An example of movement information when the vehicle receives a green signal. six-way intersections have more signal directions than those of the four-way intersection. As such, the number of signal directions varies for each intersection structure. Therefore, the number of signal directions is affected by the structure of the intersection. Time is an essential factor when allocating signals to lanes at traffic lights. Each signal direction has a certain length of time during which the vehicles exit the intersection. This length of time does not have to change even if the structure of the intersection changes. Therefore, the time duration of the signal is not affected by the structure of the intersection. Therefore, the signal control model for determining the order of the signal direction has to be changed according to the structure of the intersection. However, the model that determines how long the green light lasts in each direction does not have to change depending on the structure of the intersection. In other words, the proposed MDP does not determine the order of the progress of the vehicles according to the action; rather, it determines it according to the duration of the green signals. Thus, it is possible to apply an MDP to intersections of various structures, such as three-, four-, and five-way intersections. The goal of this study is to maximize the capacity. In addition, for a fair and reasonable distribution of signals, all lanes have to be allocated green signals in order and the appropriate green signal time must be allocated according to traffic conditions. Therefore, the deviation in the queue length and waiting time between all lanes at the intersection can be reduced. In this study, the waiting time, not the queue length, was considered as a reward parameter. This is because assigning waiting time as the reward parameter addresses the TSC problem by allocating an appropriate green signal time, which requires time information regarding how long vehicles must wait at an intersection. However, the queue length does not include sufficient time information on the number of vehicles waiting at  
Figure 4: Structures of n-way intersections. Figure 5: Examples of the possible signals at three- and four-way intersections. the intersection. In [10], there is an expression related to waiting time and queue length as follows:  
\[T=\frac{\tau*q}{N}+l/\mu\]  
Here, \(T\) denotes the average travel time; \(q\), the average queue length; \(\tau\), the time interval; \(N\), the number of vehicles; \(l\), the length of the road; and \(\mu\), the speed of the vehicle. In this equation, time and queue have a positive correlation. However, the concept of \(\tau\) and \(\mu\) related to time is applied separately from the queue length. That is, the time information includes information related to time besides information related to the queue length. Furthermore, the standard deviation of the waiting time rather than the waiting time itself represents the information regarding the deviation between the lane where a vehicle waits for a long time and the lane where a vehicle waits for a short time at an intersection. It is therefore effective to compare the situations in each lane.