# ESCAPE: Evacuation Strategy through Clustering and Autonomous Operation in Public Safety Systems
## 3 Evacuation Planning through Reinforcement Learning  
Toward realizing distributed and autonomous evacuation planning, we consider evacuees as stochastic learning automata [6]. At each operation time slot \(t\) of the reinforcement-learning loop, each evacuee \(m\) has a set of actions \(a_{m}(t)=\{1,\ldots,e,\ldots,|E|\}\) in terms of selecting an evacuation route. Each action represents a different choice \(e\) of evacuation route. Toward making their decisions, evacuees consider the status of the disaster area, and specifically output set \(\beta(t)=\{I(t),S(t)\}\), where \(I(t)\) is the set of information characterizing the disaster area and \(S(t)=[S_{1}(t),\ldots\ S_{m}(t),\ldots\ S_{|\,\textsc{M}|}(t)]\) is the final strategy that evacuees decide while competing among each other per each evacuation route that they want to go. The latter action is realized through a non-co-operative minority game, analyzed in detail in Section 4. Set of information \(I(t)\) includes evacuation rate \(\lambda_{e}(t)\) of each evacuation route \(e\) at time slot \(t\), which implicitly represents the normalized speed with which evacuees can move within an evacuation route, the capacity \(C_{e}\) of each evacuation route and its number of humans \(|M|_{e}(t)\) that already aim at evacuating through it. Output \(\beta(t)\) is determined from the second layer of the ESCAPE service, i.e., the minority game among evacuees. Based on the chosen final action \(s(t)\) and the corresponding reaction of the disaster area (i.e., updated \(\lambda_{e}(t)\) and \(|M|_{e}(t)\) values), we are able to determine reward function \(r_{m,e}(t)\) for each evacuee \(m\) per each evacuation route \(e\), which is associated with evacuees' action \(a(t)=[a_{1}(t),\ldots\ a_{m}(t),\ldots\ a_{|\,\textsc{M}|}(t)]\). The reward function is formulated as follows. \[r_{m,e}(t)=\left\{ \begin{aligned} &\lambda_{e}(t)\cdot\frac{C_{e}}{|M|_{e}(t)} \\ &\frac{\sum\limits_{e=1}^{t}\lambda_{e}(\kappa)}{t}\\ &\frac{\lambda_{e}(t)\cdot(C_{e}-|M|_{e}(t))}{d_{m,e}\cdot\frac{ \sum\limits_{e=1}^{t}\lambda_{e}(\kappa)}{t}},\ \ \text{otherwise}.\end{aligned} \right. \tag{1}\]The reward function of each user is normalized as \(\hat{r}_{m,e}(t)=\frac{r_{m,e}(t)}{|E|}\) to reflect the reward \(\sum\limits_{e=1}^{\infty}r_{m,e}(t)\)  
probability \(\hat{r}_{m,e}(t),0\leq\hat{r}_{m,e}(t)\leq 1\) of evacuee \(m\) per evacuation route \(e\). Reward probability \(\hat{r}_{m,e}(t)\) represents the level of satisfaction that evacuee \(m\) will have if they select to evacuate via evacuation route \(e\) at time slot \(t\). Given reward probability \(\hat{r}_{m,e}(t)\), each evacuee is considered to act as a stochastic learning automaton and determines their action probability vector \(Pr_{m}(t)=[Pr_{m,1}(t),\ldots Pr_{m,e}(t),\ldots Pr_{m,1|\mathds{E}}(t)]\), where \(Pr_{m,e}(t)\) expresses the probability that evacuee \(m\) wants to go to evacuation route \(e\) at time slot \(t\). Following the action probability updating rule of stochastic learning automata [6], we have the following model:  
\[Pr_{m,e}(t+1)=Pr_{m,e}(t)+b\cdot\hat{r}_{m,e_{m}^{(1)}}(t)\cdot(1 -Pr_{m,e}(t)), e_{m}^{(t+1)}=e_{m}^{(t)} \tag{2a}\] \[Pr_{m,e}(t+1)=Pr_{m,e}(t)-b\cdot\hat{r}_{m,e_{m}^{(1)}}(t)\cdot Pr _{m,e}(t), e_{m}^{(t+1)}\
eq e_{m}^{(t)} \tag{2b}\]  
where \(b,0<b<1\), is a step-size parameter that controls the convergence time of the stochastic learning automata (i.e., evacuees). The impact of parameter \(b\) on the convergence time of the ESCAPE service is numerically studied in Section 6. Equation (2a) represents the probability that evacuee \(m\) will select the same evacuation route \(e_{m}^{(t+1)}\) in time slot \((t+1)\) compared to the one in time slot \(t\), i.e., \(e_{m}^{(t)}\), while Equation (2b) reflects the probability of evacuee \(m\) selecting a different evacuation route. It is noted that the proposed reinforcement-learning approach that is based on the concept of stochastic learning automata needs set of information \(I(t)\).