# A Reinforcement Learning Model of Multiple UAVs for Transporting Emergency Relief Supplies
## 3 Model Description
### Environment  
To estimate the transportation of supplies in the event of a disaster, we selected the depot and shelters at the Ushioe district in Kochi City, Kochi Prefecture, Japan. The map of the Ushioe district is shown in Figure 1. In the case of the occurrence of the Nankai Trough earthquake, this area is expected to be inundated by tsunamis (Figure 2) and to face land subsidence caused by the earthquake. It is assumed that long-term inundation will continue even after the tsunami recedes. Therefore, relief activities in isolated disaster areas are an important issue. According to the tsunami inundation forecast data of the Ministry of Land, Infrastructure, Transport and Tourism of Japan (MILT) [47], six of the seven designated tsunami shelters in the target area are expected to be inundated by the tsunami and flooded for a long period of time. Since there are no helicports at the shelters, it is likely that vehicles, such as trucks, and aircraft, such as helicopters, cannot transport  
\begin{table}
\begin{tabular}{c c} \hline \hline
\(t\)  
\(A_{t}\)  
\(S_{t}\)  
\(R_{t}\)  
\(Q\)  
\(A_{t}\)  
\(Q\)  
emergency relief supplies during inundation. In this scenario, multiple UAVs can transport the supplies in the area.  
A set of shelters \(\mathcal{S}=\{0,1,2,\ldots,N_{S}\}\) are set up in the area, where \((j=0)\) is the depot and the set of \(\{1,2,\ldots,N_{S}\}\) represents shelters where emergency relief supplies are delivered. In this study, one shelter, which is not inundated by the tsunami, is set as depot \((j=0)\) for the transportation of supplies, and a logistics model is developed for transporting supplies from the depot to the other six shelters (\(N_{S}=6\)).  
In the event of a disaster, there will be shortages of various types of supplies, such as medicine, medical equipment, sanitary materials, food, water, and clothes. However, the priority and amount of each type of supply will differ. Therefore, it is important to consider the order of transportation and the amount to be transported according to the priority of the supplies in the transportation plan for emergency relief supplies. In this study, we focus on the duration of the \"three-day crucial rescue period\", which is the first 72 h after a disaster [48], and plan to transport several kinds of lightweight and urgent emergency relief supplies (e.g., medicine, medical equipment, and sanitary materials).  
As a mathematical formulation, \(\mathbf{I}=\{1,2,\ldots,N_{I}\}\) represents the set of item types, \(D_{jit}=\{d_{1|{\mathbf{i}}t},d_{2|{\mathbf{i}}t},\ldots,d_{i|{\mathbf{i}}t}\}\) represents the set of each shelter's demand at instant \(t\), and \(i\) is the index for item types. We assume that the demand of each item is more than one unit (\(d_{i|(t=0)}\geqslant 1\quad\forall i\in\mathbf{I},\forall j\in\mathcal{S}\)).  
In addition, to consider the priority of the supplies, we set the priority rate as the weight \(p_{ij}\) and the time limit \(b_{ij}\) for the item \(i\) of the shelter \(j\). Here, the \"time limit\" of an item is the \"expiration time\" at which the item is no longer usable (e.g., perishable food). In this study, supplies with high urgency were given a higher priority (higher penalty cost when not delivered) and a shorter transportation time limit.  
Figure 1: Depot (red) and shelters (blue) in Ushioe district in Kochi City, Kochi Prefecture, Japan.