# Model-Based Deep Reinforcement Learning with Traffic Inference for Traffic Signal Control
## 2 Related Work
These novel approaches involve either improving training methods [19; 20; 21], improving models [20; 21], or introducing methods from other domains [32; 33], such as imitation learning, meta-reinforcement learning, etc., to allow agents to more rapidly converge. At the same time, experience replay methods have been introduced, such as the introduction of prioritized experience replay for 3 DQN [20] and Ape-X for FRAP [21]. The experience replay method is a training method general to the DRL algorithm and is a crucial improvement for DQN [12]. However, to the best of our knowledge, the previous studies still have limitations. To improve the efficiency of agents in policy improvement, algorithms such as Ape-X use multi-environment training methods, which significantly increase the frequency of interaction between the agent and the environment, thus increasing the diversity of the acquired data and helping to improve the performance of the agent. However, as mentioned before, the cost of making mistakes in traffic signal control tasks is very expensive. Simply significantly increasing the interaction of agents with the environment can simultaneously increase the number of agent errors and add significant costs. Therefore, for traffic signal control tasks, an important task is to improve the efficiency of data utilization and reduce the interaction between agents and the environment. In order to improve the data utilization efficiency of the algorithm, we propose a model-based DRL algorithm that embeds a traffic inference model. This model is capable of making inferences about the evolution of the ensuing traffic state based on the current traffic state and information about the future traffic flow provided by the upstream intersections. Using this model, the agent can significantly reduce the number of actual interactions with the environment and increase the efficiency of utilization of the data collected from the environment. This proposed model is distinct from those that use information such as weather and time of day to predict traffic flow in order to adapt the agent to a state change closer to reality [38] and the traffic inference systems that use correlation analysis of various predicted big data [39]. We are inspired by multi-agent studies on traffic signal control and directly use the traffic flow information given by upstream intersections to infer the next state. Moreover, unlike the multi-agent methods, we do not provide the traffic flow information directly to the agents but use our proposed inference model to infer. The idea of using models to improve the utilization of data is not entirely new. However, the model design used in previous studies was crude, using only simple fully connected neural networks that neither considered the effect of signal changes on traffic state changes nor integrated the model with the DRL algorithm [18]. In addition, our model not only has a novel neural network architecture but also takes into account the impact that changes in traffic flow at upstream intersections can have on changes in traffic state.