# Double Deep Q-Network with a Dual-Agent for Traffic Signal Control
## 3 Deep Reinforcement Learning Model with Dual-Agent
### Dual-Agent Architecture  
In the predetermined intersection scenario, one phase corresponds to a green signal state. Thus, considering yellow signal states, a traffic signal has eight control states as shown in Table 1. The road 0 and road 2 are not illustrated because of the symmetry in the intersection. The symbols G, r, y, and g represent the green light, red light, yellow light, and left-turn waiting light, respectively. For example, the state 'GGGg' of road 1 represents that lane states from \(L0\) to \(L2\) are 'GGG' and the lane \(L3\) state is 'g'. It means that the straight traffic flow of road 1 can pass through the intersection, and the left-turn traffic flow of road 1 needs to let the straight traffic flow of road 0 go first and then seek a chance to turn left. \begin{table}
\begin{tabular}{c c} \hline \hline
**Road 1** & **Road 3** \\ \hline GGGg & rrrr \\ yyyg & rrrr \\ rrrG & rrrr \\ rrrr & gGGg \\ rrrr & yyyg \\ rrrr & rrrG \\ rrrr & rrrG \\ rrrr & gGGg \\ rrrr & yyyg \\ rrrr & rrrG \\ rrrr & gGGg \\ rrrr & yyyg \\ rrrr & rrrG \\ rrrr & gGGg \\ rrrr & gGGg \\ rrrr & gGGg \\ r  
Previous action settings [16] ensure q fixed the traffic signal phase sequence, but easily result in non-stationary action-states distributions that make it difficult to train an agent. Aiming at accelerating agent learning and improving agent performance, we present the dual-agent architecture, where the agent \(P\) controls the straight traffic flow (i.e., traffic flow in \(L0\), \(L1\), and \(L2\) of each incoming road) and the agent \(A\) controls the left-turn traffic flow (i.e., traffic flow in \(L3\) of each incoming road). Specifically, the agent \(P\) has two actions (0 and 1): turning on green lights for traffic streams of north-south (NS, i.e., \(a=0\)) and traffic streams of east-west (EW, i.e., \(a=1\)). Correspondingly, action 0 and 1 of the agent \(A\) indicates to give way to left-turn traffic streams of north-south (referred as NSL, i.e., \(a=0\)) and east-west (EWL, i.e., \(a=1\)), respectively. Note that four phases of traffic signals represent green signal states of NS, NSL, EW, and EWL, which change in a specific sequence. Figure 3 shows the control strategy of two agents implemented in road 2. At the time step \(t_{p}\), the busy agent (e.g., the agent \(P\) denoted by the busy state, the agent \(A\) denoted by the waiting state) perceives the traffic state and selects an action 0. Then, the traffic signal state of road 2 switches to 'GGGg' and keeps \(\tau_{p}\) time. There is \(\tau_{p}\) time between two time steps if the transition is not taken into account. Note that road 0 is in the same signal state with road 1, and road 2 is in the same signal state with road 3. At the time step \(t_{p}+1\), the agent \(P\) controls the traffic signal in the same manner. The traffic signal state remains the same until the action of the agent \(P\) changes. At time step \(t_{p}+2\), the busy agent selects the other action 1. The traffic signal state then changes to the transition state 'yyyg' and lasts for \(\tau_{y}\) time. Following the same control method as agent \(P\), agent \(A\) exchanges the state with agent \(P\) and obtains the access to control the green signal after the transition finishes. At the time step \(t_{a}+2\), the action of agent \(A\) changes and the traffic signal state shifts to a transition again. After the second transition is over, agent \(P\) denoted by the busy state immediately executes the action 1 and does not receive the traffic state. Therefore, the state directly changes to 'rrrr' and hold for \(\tau_{p}\) time before \(t_{p}+3\). Significantly, two agents regard the action change as a sign to shift the access of the traffic signal control in the dual-agent architecture. The more detailed operation that ensures fixed phasing sequence is illustrated in Figure 4. Agent \(P\) and agent \(A\) are denoted by two different states (busy and waiting), respectively. It means that only one agent can control the traffic green signal at each time.