# Double Deep Q-Network with a Dual-Agent for Traffic Signal Control
## 5 Results and Discussion
### Experimental Results  
As illustrated in [25], there are mainly three standard action definitions, phase switch [16], phase duration [17], and phase itself [11]. In this paper, we focus on the fixed phasing sequence in traffic signal control. Therefore, the performances of a phase control method that ignores the fixed phasing sequence are not discussed in our work. The control algorithm of phase duration (i.e., the traffic light cycle control algorithm) has difficulties to track the variety of the traffic flow in real-time because there is at least one traffic light cycle time between two time steps. The big gap between two time steps often makes the traffic signal control a more Partially Observable Markov Decision Process (POMDP) [26], leading to large variance in training results. Thus, the method with phase duration is not compared with the proposed algorithm as well. To verify the performance of the proposed DDQN with a dual-agent algorithm, we make comparisons of training results among the proposed algorithm (referred as DDQN with dual-agent), the phase switch (referred to as action with binary number), and DQN algorithm [6](referred as DQN with single agent). The setting of DQN algorithm is the same as [6] because it established the control architecture in the same intersection scenario. The control algorithm with the binary action has similar settings as our algorithm except that it has one agent setting for the sake of comparison. We use one \begin{table}
\begin{tabular}{c c} \hline \hline
**Traffic Stream** & **Production Rate** \\ \hline Straight traffic flow of east-west & 1/6 \\ Left traffic flow of east-west & 2/15 \\ Straight traffic flow of north-south & 1/10 \\ Left traffic flow of north-south & 1/10 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The production rate on different directions. \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Replay memory size of \(MP\) or \(MA\) & 10,000 \\ Minibatch size \(b\) & 32 \\ Starting \(\epsilon\) & 1 \\ Ending \(\epsilon\) & 0.05 \\ Reduction per time of \(\epsilon\) & 0.0005 \\ Target network update rate \(\beta\) & 0.001 \\ Learning rate \(\tau_{l}\) & 0.0002 \\ Discount factor \(\gamma\) & 0.9 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Parameters in Double Deep Q-Network with dual-agent. To validate the expansibility and stability of the DDQN with dual-agent algorithm, we make a comparison among our control algorithm, the fixed timing method [7], the DQN algorithm that does not consider left-turns [6], and control algorithm with binary action [16]. Following the same traffic production rate in Table 2, we change the traffic production by a ratio \(r\) (to multiply the production rate). In order to simulate the low, medium, and high traffic volume, the ratio \(r\) values are set to be 0.5, 0.7, 0.9, 1.0, 1.1, and 1.2. Correspondingly, the traffic production is 1800, 2520, 3240, 3600, 3960 and 4320 veh/h respectively. For each value in traffic production, we conduct 50 evaluating experiments (3600 s per experiment, the same production rate as shown in Table 2) with different random seeds for each control algorithm. Note that these DRL control methods are trained in the traffic system with 3600 veh/h traffic production. We calculate average values from these experiments as our results, which are illustrated from Figure 9. It is noted that the DDQN with dual-agent algorithm has an effective performance on three aspects, average waiting time, vehicle stop times, and the number of arrived vehicles. Due to the adequate capacity in the four-road intersection at low vehicle production rate, there are no significant differences in the number of arrived vehicles among these algorithms in Figure 9c. However, when the available capacity for vehicles is degrading along with the increasement of traffic productions, the DDQN with dual-agent algorithm provides the better performance than other control algorithms in terms of the number of arrived vehicles.