# Model-Based Deep Reinforcement Learning with Traffic Inference for Traffic Signal Control
## 1 Introduction  
Since the 1970s, adaptive traffic signal control (ATSC) systems have been used in urban traffic environments in many regions around the world as an effective method of alleviating traffic congestion by controlling traffic signals in response to real-time traffic flow. With adaptive traffic signal control, the aim is to adjust signal timing according to real-time traffic conditions in order to achieve the goal of being closer to the optimal control policy than when using static timing schemes. Early and successful products, such as SCOOT [1] and SCATS [2], treat signal control as an optimization problem in finding effective coordination and control strategies. Various interdisciplinary techniques have been applied to ATSC since the 1990s, such as fuzzy logic [3] and neural networks [4]. The design of ATSC systems is a long-term challenge. Reinforcement learning is a technique in artificial intelligence based on Markov decision processes (MDPs) with the goal of producing fully autonomous agents [5]. These agents are able to interact with the environment to learn optimal behavioral policies, and through trial and error they can continuously improve, gradually approaching the optimal policy. RL has had some success in the past [6; 7], but previous approaches were limited to low-dimensional problems. RL algorithms suffer from the same complexity problems as other algorithms: as the size of the problem increases, the requirement for resources for computation grows exponentially. In recent years, deep learning has gradually emerged [8; 9]. It relies on the approximation ability of the neural network to the function toreduce large-scale sample data, and the generated model can extract the common patterns and features from the samples, so as to achieve the purpose of classification or regression. The generated model has strong generalization, and excellent prediction results can be obtained for homologous distribution data. The rise of deep learning has provided new tools for reinforcement learning. The combination of deep neural networks and reinforcement learning, or deep reinforcement learning (DRL), has been used in higher dimensional problems, resulting in many exciting achievements in the past few years. Deep Q-learning network (DQN) was the start of a revolution in DRL, learning directly from images how to play a series of Atari 2600 video games, reaching superhuman levels [10]. AlphaGo, based on techniques such as Monte Carlo tree search (MCTS) and DRL, defeated the human world champion in Go [11]. Many new architectures are also emerging, such as rainbow [12], asynchronous methods [13], twin delayed deep deterministic policy gradient [14], and proximal policy optimization [15], among others. In recent years, modern DRL methods have started to be applied to ATSC. Similar to other DRL domains, traffic signal control tasks are converted into MDP-based forms. The agent controls traffic signals in a training environment by trial and error to collect data. The collected data will be saved into the experience replay [16; 17] or used directly [13; 15] for policy improvement of the agent. An intuitive experience is that the greater the amount and diversity of data, the more effective it is in policy improvement [17]. Therefore, enhancing the diversity and amount of data obtained by giving agents more opportunities to explore has become a common trend in high-performance DRL algorithms nowadays. However, in traffic signal control tasks, it is very costly for the agent to directly explore the environment. During the exploration process, the agent needs to maintain a trial-and-error process in the environment to obtain reward data to support policy improvement. However, unlike the Atari game, Go, and other tasks where the agent can perform a trial-and-error process at almost no cost, the agent in the traffic signal control task can directly cause traffic congestion during the trial-and-error process, which is very costly.