# A Reinforcement Learning Model of Multiple UAVs for Transporting Emergency Relief Supplies
## 3 Model Description
### Agent  
The agent is assumed to be a UAV that transports emergency relief supplies. A UAV loads the supplies at depot \((j=0)\) and delivers them to shelters. A set of UAVs \(\mathbf{U}=\{1,2,\dots,N_{UI}\}\) is used to transport them. \(N_{UI}\) represents the number of UAVs.  
We assume that UAVs are subject to the following conditions:  
* All UAVs are homogeneous; thus, they have the same maximum payload \(C\) and maximum amount of energy \(E\);
* At the start of the transport, all UAVs are assumed to be at the depot;
* The batteries are fully charged when the UAV takes off from the depot;
* Since there are sufficient batteries for the UAV in the depot, the battery charging time is not considered;
* Each UAV has its batteries replaced only at the depot.  
Each UAV consumes a battery depending on its payload and traveling distance. The energy consumption function (Equation (1)) describes the amount of energy used by the UAV to travel from shelter \(j\) to \(k\) with payload \(v\), referring to the study of [PERSON] et al. [18].  
\[R(v) = \delta_{0}+\delta v+h_{jk}(\rho_{0}+\rho v) \tag{1}\] \[v = \sum_{n}^{M_{lm}}\sum_{i=1}^{N_{l}}w_{ijklmn} \tag{2}\]  
Here, \(h_{jk}\) is the distance between shelters \(j\) and \(k\), and \(\delta_{0}\) is the energy for take-off and landing without supplies. \(\delta\) is the additional energy amount needed for take-off and landing with an additional item, \(\rho_{0}\) is the energy to fly one distance unit for an empty UAV, and \(\rho\) is the additional energy amount needed to fly one distance unit with one item. In Equation (2), \(w_{ijklmn}\) denotes the amount of item \(i\) transported from shelter \(j\) to \(k\) as the \(n\)th location in trip \(m\) by UAV \(l\), and \(v\) denotes the total amount of items transported after the \(n\)th location in trip \(m\) by UAV \(l\).  
Figure 2: Estimated tsunami inundation depth for the Nankai Trough earthquake in Ushioe district in Kochi City, Kochi Prefecture, Japan. Data from [47].  
From the above assumptions, the UAV's maximum payload and maximum amount of energy constraints can be represented as follows:  
\[\sum_{n=1}^{M_{lm}}\sum_{i=1}^{N_{lj}}w_{ijklmn} \leq C\quad\forall l,\forall m \tag{3}\] \[\sum_{n=1}^{M_{lm}}R(v) \leq E\quad\forall l,\forall m \tag{4}\]  
The transportation cost \(f_{jk}\), which is equal to the consumption time, from shelter \(j\) to \(k\) is defined as Equation (5), referring to the study of [PERSON] et al. [34].  
\[f_{jk} = t_{take}+t_{land}+t_{serve}+t_{jk} \tag{5}\] \[t_{jk} = \left\{\begin{array}{ll}\dfrac{h_{jk}}{V_{\max}}+\dfrac{V_{ \max}}{a}&\left(h_{jk}>\dfrac{V_{\max}{}^{2}}{a}\right)\\ 2\sqrt{\dfrac{h_{jk}}{a}}&\left(h_{jk}<\dfrac{V_{\max}{}^{2}}{a}\right)\end{array}\right. \tag{6}\]  
where \(t_{take}\) is the take-off time, \(t_{land}\) is the landing time and \(t_{serve}\) is the servicing time (e.g., time to change batteries in the depot or unload supplies in the shelter). \(t_{jk}\) denotes the flight time between shelters \(j\) and \(k\). Since each UAV is presumed to accelerate to its maximum speed with uniform acceleration, \(t_{jk}\) can be calculated as Equation (6), where \(V_{\max}\) is the maximum speed of a UAV, and \(a\) represents the uniform acceleration of a UAV. In addition, one trip \(m\) is defined as the period from the time the UAV leaves the depot to the time it transports supplies to one or more shelters and returns to the depot.