# Crowd Evacuation Guidance Based on Combined Action Reinforcement Learning
## 3 Results  
### Experiment Design and Implementation  
In this study, a crowd dynamics simulation system based on the social force model [3], is used as the agent's interactive environment. A typical multi-room, dual-exit indoor scene is constructed, in which the guidance signs are marked as green arrows, and the exits are marked as green rectangles. The simulation system takes the display states of the evacuation guidance signs as input. In the simulation, the individuals first determine a subjective driving force direction based on the nearby guidance signs and the distance to each exit. When the simulated individuals do not see the evacuation guidance signs, they choose the closest exit and escape according to the statically shortest route. When the individuals notice the evacuation guidance sign, they escape in the direction indicated by the sign, unless they are close to an exit, in which case they may ignore the sign. When congestion occurs, the individual's speed is limited, and the subjective driving force that tends to get rid of congestion is added. Then, the dynamic simulation calculates the speed and displacement of the individual movement, simulating the collision between people and people, and people and walls. Individuals within the exit area are deemed to have successfully escaped. At the end of a simulation step, the crowd's positions and the signs' states are painted in a scene image as output, which is passed to the reinforcement learning agent as input to its neural network. The simulation system is based on C++ and the Qt library. As shown in Figure 4, the simulation scene's size is 29.2 m \(\times\) 19.7 m, and the size of the floor map is \(499\times 337\) pixels. The scene contains two exits and six rooms, where the upper and lower channels connect the rooms and exits. Each channel is set to have five dynamic guidance signs, and the signs can indicate one of two directions. The number of people is 200, and their initial positions are randomly distributed in a circular range; the range of the distribution center and radius is \(x\in(100,140)\), \(y\in(60,280)\), and \(r\in(100,200)\), and the maximum individual movement speed is 5 m/s. Each time step of the simulation system dynamics calculation is 40 ms, and the upper limit of the simulation time is 100 s.  
The CA-DQN method is implemented based on Python, the TensorFlow platform, and OpenAI/baseline library. In each time step of the reinforcement learning agent, the simulation system first performs a five-step calculation, simulating the crowd's movement within 200 ms. The last four images obtained are down-sampled into 1/2-size grayscale images, and combined into a four-channel image of \(249\times 168\) pixels, which is input to the agent's Q network as state \(s_{t}\). The Q network structure is shown in Figure 3, comprising a three-layer convolutional neural network and three-layer fully connected neural network. The 20 neurons in the output layer are divided into ten groups. From each group, the larger output value is selected as the display signal for the dynamic guide sign. By combining them, a ten-dimensional discrete output vector is formed as agent action \(a_{t}\). \(a_{t}\) acts on the simulation system to change the direction displayed by the ten guide signs, thereby directing the crowd's movement. At this time, the interaction between the agent and simulation environment completes a cycle. The agent's reward for each step is fixed at \(-1\), implying that it obtains a reward of \(-5\) per second. The agent's training goal is to reduce the overall evacuation time. For example, the exits' congestion and the imbalanced exit utilization lead to longer evacuation time and lower rewards, giving the agent negative feedback. The agent will learn how to avoid this situation and find the optimal strategy through trial and error.