# Double Deep Q-Network with a Dual-Agent for Traffic Signal Control
## 2 Model and Problem Presentation
### Deep Reinforcement Learning Traffic Signal Controller  
In this paper, we consider a complex intersection scenario, as shown in Figure 1. For each incoming road, the innermost lane named L3 is for left turning of vehicles and the other lanes (referred as \(L2\), \(L1\), and \(L0\)) are for straight going or right turning of vehicles. As shown in Figure 2, the traffic signal control problem is addressed by a reinforcement learning model, in which the traffic light is regarded as a DRL agent that changes their actions (traffic signal phase) by interacting with the traffic environment. Specifically, the DRL model defines the traffic signal control problem as a Markov decision process (MDP), which mainly contains five elements \(<\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma>\). As usual, \(\mathcal{S}\) and \(\mathcal{A}\) are spaces of states and actions separately, \(\mathcal{P}\) is transition function, \(\mathcal{R}\) is known as reward function and \(\gamma\in[0,1]\) is the discount factor. At each time step \(t\), the agent obtains the state \(s_{t}\in\mathcal{S}\) and chooses an action \(a_{t}\in\mathcal{A}\) based on the observation. In return, the agent perceives the next state \(s_{t+1}\sim\mathcal{P}(s_{t},a_{t})\) and acquires a reward \(r_{t}\sim\mathcal{R}(s_{t},a_{t})\). This process continues until the agent reaches a terminal state, which is the last observation state in our traffic simulation.  
The goal of an agent is to find an optimal policy \(\pi:\mathcal{S}\times\mathcal{A}\rightarrow[0,1]\) which maximizes the expected return from each state \(s_{t}\). The expected return \(R_{t}\) at time \(t\) is defined as follows:  
\[R_{t}=E[\sum_{k=0}^{\infty}\gamma^{k}r_{t+k}] \tag{1}\]  
where \(\gamma\) is the discount factor to acquire a tradeoff between the current and future reward. In the Q-learning algorithm, after an action is selected by \(\epsilon\)-greedy policy, the expected return becomes  
\[Q^{\pi}(s,a)=E[R_{t}|s_{t}=s,a] \tag{2}\]  
where \(Q^{\pi}(s,a)\) is the action-value. To obtain the maximal action-value for a state \(s\) and an action \(a\) achievable by any policy, the optimal value function is defined as  
\[Q^{*}(s,a)=\max_{\pi}Q^{\pi}(s,a) \tag{3}\]  
Generally, the action-value function in value-based model-free reinforcement learning methods is described by a function approximator, such as a deep convolutional network. Specifically, \(Q(s,a;\theta)\) is represented as approximator with parameters \(\theta\), and needs to approach to the optimal action value, i.e.,  
\[Q^{*}(s,a){\approx}Q(s,a;\theta) \tag{4}\]  
where parameters \(\theta\) are learned by iteratively minimizing the loss function  
\[L(\theta)=E(r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime};\theta)-Q(s,a; \theta))^{2} \tag{5}\]  
where \(r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime};\theta)\) is the target value, \(s^{\prime}\) and \(a^{\prime}\) is the state and action at the next time step respectively. In the DQN algorithm, a target network with weights \(\theta^{-}\) is adopted to address the instability problem of the policy. Thus, the optimal target value is replaced by the approximate target value  
\[y=r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime};\theta_{i}^{-}) \tag{6}\]  
where \(\theta_{i}^{-}\) are parameters of the target network in the iteration \(i\). This lead to a sequence of loss function \(L_{i}(\theta_{i})\) that changes at each iteration \(i\),  
\[L_{i}(\theta_{i})\approx(r+\max_{a^{\prime}}Q(s^{\prime},a^{\prime};\theta_{i} ^{-})-Q(s,a;\theta_{i}))^{2} \tag{7}\]  
Note that the agent based on the DQN algorithm samples random minibatch of transitions \((s,a,r,s^{\prime})\) from the experience replay to keep networks converging steadily.  
Figure 1: A four-road intersection. The red-dashed region represents incoming road.  
Figure 2: The traffic signal controller based on DRL. There is the traffic signal control system referred as agent in the red-dashed region.