# Crowd Evacuation Guidance Based on Combined Action Reinforcement Learning
## 2 Methods
### Combined Action Space DQN  
In the network structure, DQN uses a multi-layer convolutional neural network to process the image input, and then, connects a multi-layer fully connected neural network with each neuron in the output layer, corresponding to a possible combination of discrete actions. For the components of the action, the total action space is the Cartesian product of each dimension's action space. When the action space has \(n\) dimensions and each dimension has \(m\) discrete actions, the DQN network needs \(m^{n}\) output layer nodes to correspond to value \(Q(s,a)\) of different actions. Therefore, as the number of action dimensions increases, the DQN network structure's complexity increases exponentially, making the algorithm unrealizable. Meanwhile, an excessive number of output layers will also reduce sample utilization and the difficulty in updating network parameters. This phenomenon is called the \"dimensional disaster\" of DQN. When applying crowd evacuation guidance, the agent action is defined as the guide signs' display states. Even if each guide sign has only two states, left and right, for \(n\) guide flags, the total action space capacity will reach as much as \(2^{n}\). To solve this problem, the CA-DQN was proposed. Since the guide signs' information corresponding to the action \(a_{t-1}\) in the previous time step has been included in the scene image \(s_{t}\), and the signs do not change frequently, \(a_{t}\approx a_{t-1}\). Therefore, it can be approximately considered that under the  
Figure 2: Reinforcement learning model for evacuation guidance. condition of observation \(s_{t}\), the strategy of an action dimension is independent of other action dimensions, and an independent neural network structure can be used. As shown in Figure 3, for independent action dimensions, each dimension corresponds to a group of nodes in the output layer of the Q function network, and each group contains all discrete actions in this dimension. This change can be seen as setting the value function \(Q_{d}(s,a^{(d)};\theta)\) for each action dimension \(d\), and sharing a set of network parameters. At this time, the number of nodes in the network's output layer is the sum of the number of discrete actions in each dimension, and the growth rate of the number of output layer nodes decreases from exponential growth to linear growth. For example, the number of output layer nodes required for \(n\) guiding signs is \(2n\). In CA-DQN, the agent's action  
\[\mathbf{a_{t}}=(a_{t}^{(1)},a_{t}^{(2)},\ldots,a_{t}^{(D)}) \tag{5}\]  
is a \(D\)-dimensional combined action vector. For each dimension \(d\), the agent uses a greedy strategy to select the actions. \[a_{t}^{(d)}=\operatorname*{arg\,max}_{a}Q_{d}(s_{t},a;\theta) \tag{6}\]  
Referring to Formula (4), for a sample \(e_{t}=(s_{t},a_{t},r_{t},s_{t+1})\), this paper define the TD error in each dimension:  
\[\delta_{t}^{(d)}=r_{t}+\gamma\max_{a}Q_{d}(s_{t+1},a;\theta^{-})-Q_{d}(s_{t},a _{t}^{(d)};\theta) \tag{7}\]  
Combined with the DDQN algorithm that was proposed in [22] to select actions at time \(t+1\) with the current Q network to avoid overestimation, the TD error is further defined as follows:  
\[\delta_{t}^{(d)}=r_{t}+\gamma Q_{d}(s_{t+1},\operatorname*{arg\,max}_{a}Q_{d}( s_{t+1},a;\theta);\theta^{-})-Q_{d}(s_{t},a_{t}^{(d)};\theta) \tag{8}\]  
Then, for a set of samples \(B=\{e_{1},\ldots,e_{t}\}\) from the experience pool, the loss function of the neural network is defined as the arithmetic mean of the squared error loss. \[L(\theta)=\mathbb{E}_{B}\Bigg{[}\frac{1}{D}\sum_{d}\Bigl{(}\delta_{t}^{(d)} \Bigr{)}^{2}\Bigg{]} \tag{9}\]  
Figure 3: Network structure of combined action-space deep Q network (CA-DQN). According to the loss Formula (9), the neural network is trained with the error back-propagation algorithm.