# A Reinforcement Learning Model of Multiple UAVs for Transporting Emergency Relief Supplies
## 2 Related Works
### UAV Routing Problem for Logistics  
There are two types of UAVRPs: drone-only problems, which use only UAVs, and truck-drone problems [25; 39; 40; 41; 42], which combine UAVs and land transportation. In this section, we review research on drone-only problems that are primarily relevant to this study, considering the disruption of transportation infrastructure in a disaster. As the VRP is an NP-hard problem, exact algorithms are efficient only for small problem instances. Since real-world problems tend to be quite large, heuristics and metaheuristics are often more practical [43]. Hence, in the literature, many UAVRPs have been solved using heuristicsand metaheuristics. [PERSON] et al. [17] proposed two multi-trip VRPs for UAV delivery to minimize costs subject to a delivery time limit or minimize the overall delivery time subject to a budget constraint. They proposed a cost function that considers an energy consumption model and UAV reuse and then applied it in a simulated annealing (SA) heuristic to find suboptimal solutions to practical scenarios. [PERSON] et al. [22] proposed a mixed integer linear programming (MILP) formulation for the derivation of persistent UAV delivery schedules and developed a receding horizon task assignment (RRTA) heuristic with numerical examples for island-area delivery. They set the objective function to maximize the weighted sum of two objectives: the total number of customers covered and the total traveling distance during the delivery service. Similarly, [PERSON] et al. [21] established a model for UAV task assignment in logistics that is solved by an improved particle swarm optimization (PSO) algorithm. Initially, they set a time window at each node, imposed a penalty cost for delayed delivery and a cost based on the total distance traveled by all UAVs, and then minimized the cost using the PSO algorithm. [PERSON] et al. [25] proposed a continuous approximation (CA) model that determines the optimal distribution center locations, their corresponding service regions, and ordering quantities to minimize the overall distribution cost for disaster-relief operations. [PERSON] et al. [23] focused on the issue of UAV logistics in urban environments and developed an automatic delivery system to support the delivery of packages. They optimized two objectives: customer satisfaction and total completion time. A variable neighborhood search (VNS) algorithm framework is used to generate the approximate optimal solution for their problem. [PERSON] et al. [18] considered UAV applications in last-mile distribution in HL and presented an optimization model for the delivery of multiple packages of lightweight relief items. They set the objective to minimize the total traveling distance (or time/cost) of the UAV under payload and energy constraints. [PERSON] et al. [19] proposed a bi-objective mixed integer programming model for the multi-trip drone location routing problem, which allows simultaneous pick-up and delivery, and shorten the time to deliver medical supplies in the right place. Then, a modified NSGA-II (Non-dominated Sorting Genetic Algorithm II) which includes double-layer coding, is designed to solve the model. [PERSON] et al. [20] presented an optimization model to design the logistics for a fleet of drones for timely delivery of medical packages to remote locations. They tackled the problem of limited payload capacity by scheduling and sequencing a set of deliveries. [PERSON] et al. [24] first addressed the problem of locating the platforms as well concurrently determining which platform serves which demand points and in what order, in order to minimize total disutility for product delivery. Then, the two-period problem where the platforms can be relocated, using usable road network, after the first period. UAVRPs have been studied for heterogeneous UAVs. [PERSON] et al. [26] dealt with the path planning problem of UAVs with different abilities in multi-region systems.