# Model-Based Deep Reinforcement Learning with Traffic Inference for Traffic Signal Control
## 3 Background  
Reinforcement learning algorithms use trial and error to learn which actions yield the most reward. The agent begins by taking random actions, and as it gathers more information about the environment, it adjusts its actions based on the rewards it receives. This process of learning through interaction with the environment is known as exploration. As the agent explores the environment, it builds up a model of the relationships between its actions, observations, and rewards. This model, known as the agent's policy, guides the agent's decision-making process as it selects actions to take in different situations. The goal of the agent is to optimize its policy by choosing actions that maximize the expected reward over time. The goal of reinforcement learning is to develop intelligent agents that can act independently and adapt their behavior to maximize reward in dynamic environments. The agent interacts with the environment to form sequences:  
\[H_{t}=A_{1},O_{1},R_{1},...,A_{t},O_{t},R_{t}, \tag{1}\]  
where \(A\) represents the agent's behavior, \(O\) represents the agent's observation of the environment, and \(R\) represents the reward received by the agent. One of the key ideas behind reinforcement learning is the concept of the Markov decision process (MDP). An MDP is a mathematical model used to represent a dynamic, uncertain environment in which an agent can interact with and make decisions. Since history is serial data, it contains too much noisy information to be used directly. In an MDP, the agent's current state, denoted as \(S_{t}=f(H_{t})\), represents all of the relevant information the agent has about its environment at a given time. The state includes everything the agent needs to know in order to make an informed decision. Using the state \(S_{t}\) instead of the full history \(H_{t}\) can be beneficial because it reduces the amount of noise and irrelevant information the agent needs to consider when making a decision. By using a function \(f\) to extract the relevant information from the history, the agent can focus on the most important factors and ignore the rest. This can help the agent make more efficient and accurate decisions. The agent's decision, or action, at each time step is based on its current state and is denoted as \(A_{t}\). One important property of an MDP is the Markov property, which states that the future state of the system is dependent only on the current state and not on the past states. In other words, the state at time \(t+1\) is only influenced by the state at time \(t\) and the action taken at time \(t\). This means that given the current state, the agent does not need to know the full history of the system in order to make an informed decision [5], defined as follows:  
\[\mathbb{P}[S_{t+1}|S_{t}]=\mathbb{P}[S_{t+1}|S_{1},...,S_{t}]. \tag{2}\]  
The Markov decision process based on Markov states can be formulated as \(<\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma>\), where:  
* \(\mathcal{S}\) is the set of all states
* \(\mathcal{A}\) is the set of all actions
* \(\mathcal{P}\) is the state transfer matrix, which describes the probability of transitions between states, specifically, \(\mathcal{P}^{a}_{s^{\prime}}=\mathbb{P}[S_{t+1}=s^{\prime}|S_{t}=s,A_{t}=a]\)
* \(\mathcal{R}\) is the reward function, which describes the expected reward in the state, specifically, \(\mathcal{R}^{a}_{s}=\mathbb{E}[R_{t+1}|S_{t}=s,A_{t}=a]\)
* \(\gamma\) is a discount factor used to discount rewards further away from the current time when calculating cumulative rewards, where \(\gamma\in[0,1]\). Unlike traditional classification or regression tasks, reinforcement learning does not have a supervisor; i.e., there is no sample label, but instead a reward function to evaluate the behavior, which is by nature a trial-and-error process. The agent's behavior affects the environmental changes and the rewards obtained, while the algorithm's evaluation of the behavior is delayed and is gradually updated and changed. The goal of the agent is to maximize the sum of rewards.