# A Reinforcement Learning Model of Multiple UAVs for Transporting Emergency Relief Supplies
## 3 Model Description
### Cost Functions  
This study aims to plan optimal transportation by multiple UAVs from three perspectives: rapidity, urgency, and equity. Following those objectives, the proposed UAVRP aims to minimize the total cost, including the following three costs:  
* Flight Time Cost (_FC_): The cost based on the total flight time of all UAVs. This corresponds to rapidity;
* Priority Cost (_PC_): The cost based on quick transportation of high-priority supplies. This corresponds to urgency;
* Equity Cost (_EC_): The cost based on equitable transportation to all shelters. This corresponds to equity. First, _FC_ is described as follows: \[Z_{1} = \sum_{l=1}^{N_{ll}}\sum_{m=1}^{N_{l}}\sum_{n=1}^{M_{lm}}\sum_{j=0} ^{N_{S}}\sum_{k=0}^{N_{S}}f_{jk}\cdot x_{jklmn}\] (7) \[x_{jklmn} = \left\{\begin{array}{ll}1&\left(\text{UAV }l\text{ transport from }j\text{ to }k\text{ as }n\text{th location in the trip }m\right)\\ 0&\left(\text{otherwise}\right)\end{array}\right.\] (8)  
The cost function \(Z_{1}\) in Equation (7) is the total flight time for all UAVs and all trips. \(f_{jk}\) is the time taken for transportation from shelter \(j\) to \(k\), which is represented in Equation (5), and \(x_{jklmn}\) is a decision variable, which is represented in Equation (8).  
Second, _PC_ is described as follows:\[Z_{2} = \sum_{l=1}^{N_{lt}}\sum_{m=1}^{N_{l}}\sum_{n=1}^{M_{lm}}\sum_{j=0}^{N _{S}}\sum_{k=0}^{N_{S}}\sum_{l=1}^{N_{l}}P_{jlt}(w_{ijklmn})\cdot x_{jklmn} \tag{9}\] \[P_{jlt}(w_{ijklmn}) = p_{ij}\cdot w_{ijklmn}\cdot\max\{(u_{ijlm}-b_{ij}),0\}\] (10) \[u_{ijlm} = \sum_{m=1}^{m}\sum_{n=1}^{M_{lm}}\sum_{j=0}^{N_{S}}\sum_{k=0}^{N _{S}}f_{jk}\cdot x_{jklmn} \tag{11}\]  
The cost function \(Z_{2}\) in Equation (9) is equal to the total penalty cost of items, which depends on the transportation time and urgency of items. \(w_{ijklmn}\) represents the amount of item \(i\) transported to shelter \(j\) (\(j\
eq 0\)) as the \(n\)th location in trip \(m\), \(p_{ij}\) represents the penalty cost of item \(i\), and \(u_{ijlm}\) represents the time when UAV \(l\) transports items for shelter \(j\) in trip \(m\), which is described in Equation (11). Equation (10) represents the penalty cost when the UAV transports item \(i\) for shelter \(j\) later than the time limit \(b_{ij}\). On the other hand, there is no penalty cost when the UAV transports item \(i\) for shelter \(j\) within the time limit \(b_{ij}\).  
Finally, we define the \((EC)\). In previous studies, various cost functions have been defined to describe equity [44; 49]. In this study, we use the \(EC\) of [PERSON] et al. [29], which is described as follows:  
\[Z_{3} = \sum_{t=1}^{T}\sum_{j=1}^{N_{S}}g(r_{ji}) \tag{12}\] \[r_{jit} = \frac{\sum_{i=1}^{N_{l}}d_{ijt}}{\sum_{i=1}^{N_{l}}d_{ijt(t=0)}} \tag{13}\]  
The cost function \(Z_{3}\) in Equation (12) indicates the \"_disutility-weighted arrival time_\". The disutility function (12) encourages UAVs to not necessarily satisfy a shelter's entire demand but rather to save supplies to serve another shelter. Equation (13) represents the rate of remaining demand of shelter \(j\) at instant \(t\). The following piecewise linear disutility function is used in our calculations:  
\[g(r)=\left\{\begin{array}{ll}\frac{4r}{13}&(r<0.25)\\ \frac{8r-1}{13}&(0.25\leq r<0.5)\\ \frac{16r-5}{13}&(0.5\leq r<0.75)\\ \frac{24r-11}{13}&(0.75\leq r)\\ \end{array}\right. \tag{14}\]  
where \(r\) is the rate of remaining demand. The above function can be represented as in Figure 3.  
Figure 3: The disutility function for unsatisfied demand.  
These three cost functions are converted to a single objective function using the weighted sum method. The objective function of this problem is described as follows:  
\[Minimize\qquad\lambda_{1}Z_{1}+\lambda_{2}Z_{2}+\lambda_{3}Z_{3} \tag{15}\]  
where \(\lambda_{1}\), \(\lambda_{2}\) and \(\lambda_{3}\) are the weights of the three costs.