# Crowd Evacuation Guidance Based on Combined Action Reinforcement Learning
## 1 Introduction
### Deep Reinforcement Learning  
Reinforcement learning [19] is a crucial component in the field of artificial intelligence. This method learns mapping from the environment state to action by interacting with the environment and through trial and error, and then, finds the optimal behavioral strategy to maximize the accumulated reward. Combined with deep neural networks, the deep reinforcement learning agent can directly use images as input, and internalize feature extraction and value function estimation in the network structure, which significantly expands the agent's perception and decision-making capabilities. The iconic achievements of deep reinforcement learning include the DQN method, which surpassed the abilities of human players in Atari video games [5]; AlphaGo [20], which defeated top human players in Go; and AlphaStar [21], which attained master-level ranking in StarCraft 2 online battles. The reinforcement learning model [19] is based on the Markov decision process (MDP), which is described as a four-tuple \((S,A,P_{a},R_{a})\). \(S\) is the set of all states, the state space, and \(A\) is the action space. The state transition function \(P_{a}(s,s^{\prime})=\Pr(s_{t+1}=s^{\prime}|s_{t}=s,a_{t}=a)\) represents the probability of an agent performing action \(a\) in state \(s\) and the environment entering state \(s^{\prime}\). The reward function \(R_{a}(s,s^{\prime})\) represents the agent's instant reward, when it executes action \(a\) in state \(s\) and the environment enters state \(s^{\prime}\). The agent observes the environment state \(s_{t}\) at each discrete time step \(t\), and selects action \(a_{t}=\pi(s_{t})\) to act on the environment according to strategy \(\pi:S\to A\). The environment feeds back reward \(r_{t}\) to the agent and transfers to the next state, \(s_{t+1}\). The interaction process between the agent and environment is shown in Figure 1.  
Based on the MDP model of reinforcement learning, the state-action value function is defined, which can also be called the action-value function:  
\[Q_{\pi}(s_{t},a_{t})=\mathbb{E}_{\pi}[r_{t}+\gamma r_{t+1}+\gamma^{2}r_{t+2}+ \ldots] \tag{1}\]  
It represents the expected cumulative reward obtained after performing action \(a_{t}\) in state \(s_{t}\) according to strategy \(\pi\), where \(\gamma\) is the reward decay coefficient. With the optimal strategy \(\pi^{*}\), the optimal action-value function \(Q^{*}\) satisfies the Bellman equation:  
\[Q^{*}(s_{t},a_{t})=\mathbb{E}_{\pi^{*}}[r_{t}+\gamma\max_{a}Q^{*}(s_{t+1},a)] \tag{2}\]  
Figure 1: Schematic of reinforcement learning model. The DQN method, which was developed by the Google DeepMind team [5], uses a deep neural network to approximate the action-value function. It is divided into the current Q network with parameter \(\theta\) and the target Q network with parameter \(\theta^{-}\), with the current Q network parameters being copied to the target Q network at regular intervals. DQN uses a greedy strategy, always choosing an action with the maximum Q value in the current state, and adds a certain probability to select random actions as the exploration process during training. DQN also uses an experience pool to store and manage samples. For a sample \(e_{t}=(s_{t},a_{t},r_{t},s_{t+1})\) in a time step, there is an estimate of \(Q_{\pi}(s_{t},a_{t})\) from Formula (2):  
\[r_{t}+\gamma\max_{a}Q(s_{t+1},a;\theta^{-}) \tag{3}\]  
which is called the temporal difference (TD) target. Then, the TD error of this sample is defined as the difference between the TD target and the current value of Q function:  
\[\delta_{t}=r_{t}+\gamma\max_{a}Q(s_{t+1},a;\theta^{-})-Q(s_{t},a_{t};\theta) \tag{4}\]  
For a batch of data \(B=\{e_{1},\dots,e_{t}\}\) sampled from the experience pool, the network loss function is defined as the squared error loss \(L(\theta)=\mathbb{E}_{B}(\delta_{t}^{2})\); then, the error backpropagation algorithm is applied to update the network parameters.