# Crowd Evacuation Guidance Based on Combined Action Reinforcement Learning
## 2 Methods  
### Reinforcement Learning Model for Crowd Evacuation Guidance  
In the crowd evacuation guidance problem, the evacuation guidance system is regarded as an agent, and the environment, with which the agent interacts, includes the actual building scene and people moving within the building. An image is used to represent the architectural scene. Various sensors, such as cameras, collect the crowd's movement state and draw this data into the scene image. As shown in Figure 2, The scene image now contains the information that is required in the current environment, which is defined as the environment state \(s_{t}\in S\) of the MDP. The evacuation guidance system displays signals through dynamic guidance signs to guide and direct crowd movement. Therefore, the agent action \(\mathbf{a_{t}}\in A\) corresponds to the guide sign signal, \(\mathbf{a_{t}}\) is a discrete vector, each dimension corresponds to a guide sign, and the component is a sign display state (left or right). Owing to the complexity of the environment and crowd movement, the state transition function \(P_{a}(s,s^{\prime})\) is unknown; hence, the agent must learn and adapt during the interaction process. The design of the reward function determines the optimization direction and learning purpose of the agent. In the crowd evacuation problem, the reward function should be designed according to the number of people who successfully evacuated or the time required for evacuation. This paper defines \(r_{t}=-1\), which indicates that a fixed penalty is given at each time step. The agent's learning goal is to minimize the cumulative penalty, that is, the shortest evacuation time for the entire crowd.  
The training process for reinforcement learning agents requires continuous interaction with the environment, as well as learning through exploration and trial and error. The required interaction scale is large, generally over tens of thousands of cycles and millions of time steps. Moreover, the agent's lack of knowledge in the initial training period may cause more potential dangers. Therefore, the reinforcement learning agent used for the intelligent evacuation guidance system must be trained in a simulation environment, and deployed in the actual building after the training is completed.  
The evacuation guidance system agent explores and learns through various interactions with the simulation environment, and finally obtains an approximate optimized strategy \(\pi(s)\). There is no need to manually design the building route topology graph or grid model during the learning process. The agent can autonomously discover and optimize the guidance strategy, and the design of additional intermediate algorithms, such as path planning, is not required. In practical applications, the building's map is inputted into the simulation system, and the reinforcement learning agent is pre-trained through simulation to obtain the optimal strategy in this scene. After being deployed to the actual building, the sensors collect crowd movement information at each moment using additional computer vision algorithms, and draw an image combined with the building's map as the agent's observation state input. The agent calculates the action vector according to the previously optimized strategy, and the dynamic guidance signs display the corresponding signal to realize practical guidance for the crowd's evacuation.