# Model-Based Deep Reinforcement Learning with Traffic Inference for Traffic Signal Control
## 4 Traffic Inference Model  
### Model Overview  
As shown in Figure 1, the task of the traffic inference model is to infer changes in the traffic state based on the current traffic state and other necessary information, aiming to predict and understand the dynamic changes occurring in the traffic situation. Our proposed traffic inference model is based on neural networks, which have powerful learning and approximation capabilities. By harnessing the power of neural networks, the model is able to continuously learn and adapt to new information, enabling it to predict traffic patterns. Through continuous training, the model is able to reason about changes in traffic states that are distributed in the same way as the training data. With a traffic inference model, reinforcement learning algorithms can interact with the model, which in turn significantly reduces the number of actual interactions with the environment and improves the efficiency of data utilization.  
In the Formula (7), the data of the agent's interaction with the environment is organized in the form \(<s,a,r,s^{\prime}>\), where \(s\) is the current state, \(a\) is the selected action, \(r\) is the reward obtained, and \(s^{\prime}\) is the state at the next time step after the action is executed. According  
Figure 1: Traffic inference with future information. The green line means the light is green for the moment.  
to the Formula (7), the q-table converges to the expectation under a given state transfer distribution after continuous iteration to convergence, \(r,s^{\prime}\), for a given \(s,a\). Specifically,  
\[s^{\prime}\to\mathbb{E}[S_{t+1}|S_{t}=s,A_{t}=a], \tag{8}\] \[r\to\mathbb{E}[R_{t}|S_{t}=s,A_{t}=a], \tag{9}\]  
it can be deduced that:  
\[Q(s,a)\to\mathbb{E}[R_{t}|S_{t}=s,A_{t}=a]+\underset{a^{\prime}}{max}\gamma Q( \mathbb{E}[S_{t+1}|S_{t}=s,A_{t}=a],a^{\prime}). \tag{10}\]  
For the traffic signal control environment, the traffic flow is constantly changing and the change of traffic flow also changes the dynamics of the environment. Therefore, the traffic inference model trained based only on \(s,a\) is insensitive to changes in the traffic flow and can only correspond to the given traffic flow. In order to make the trained traffic inference model, i.e., the combined model of the state inference model and the reward inference model, able to cope with different traffic flows and complete inference under different traffic flows, we added the future traffic flow information based on upstream intersections into the model. Specifically,  
\[s^{\prime}\to\mathbb{E}[S_{t+1}|S_{t}=s,A_{t}=a,F_{t}=f], \tag{11}\] \[r\to\mathbb{E}[R_{t}|S_{t}=s,A_{t}=a,F_{t}=f], \tag{12}\]  
it can be deduced that:  
\[Q(s,a)\to\mathbb{E}[R_{t}|S_{t}=s,A_{t}=a,F_{t}=f]+\underset{a^{\prime}}{max} \gamma Q(\mathbb{E}[S_{t+1}|S_{t}=s,A_{t}=a,F_{t}=f],a^{\prime}), \tag{13}\]  
where \(F_{t}\) represents future information, which can influence the environment dynamics. As shown in Figure 1, the inference of traffic state evolution will become easier when future traffic flow information is provided, which can be obtained from upstream intersections.  
In addition, inspired by FRAP, our model will also use inference based on same-direction-phase lane groups to further improve the efficiency of data utilization by making the inference model invariant to data with symmetry. Overall, we design a novel neural network model to reason about the traffic state at the next time step based on the current traffic state, the actions taken, and the future information as well as the rewards. Specifically,  
\[s^{(i)},a^{(i)},f^{(i)}\to s^{\prime(i)},r^{(i)}\, \tag{14}\]  
where \(i\) is the index of same-direction-phase lane group.