# A Reinforcement Learning Model of Multiple UAVs for Transporting Emergency Relief Supplies
## 1 Introduction  
When a disaster such as an earthquake or tsunami occurs, a quick and sufficient distribution of emergency relief supply is required. The logistics activities carried out to rapidly provide adequate aid supplies (relief goods) to people affected by disasters are commonly known as humanitarian logistics (HL) [1]. The importance of HL cannot be overstated, as it can affect the death tolls in regions suffering from disasters [2]. Post-disaster HL involves multiple activities; donation soliciting, material convergence, and last-mile distribution [3]. Donation soliciting refers to the collection of materials at donor sites, whereas material convergence refers to the flow of materials from donor sites to the end sites (distribution centers) inside disaster areas [4]. Last-mile distribution, which is the final stage of the relief supply chain, refers to the delivery of materials from end sites to survivors (to individual demand locations; e.g., shelters), and many disasters have shown the challenges of Last-mile distribution [5, 6]. In the Great East Japan Earthquake of 11 March 2011, many problems occurred in the delivery of relief supplies, and, as a result, hundreds of thousands of survivors were unable to receive relief supplies during the first six days of the disaster [5, 6]. One of the most important problems was the lackof sufficient transportation to meet the demands of all the shelters. Road transportation of emergency relief supplies was difficult due to a lack of fuel oil [7] and disruptions in the transportation network [8]. On the other hand, the number of helicopters that can be used in an emergency is limited, and it would be difficult to transport supplies to all shelters by helicopter alone, since helicopters are used for a variety of operations, such as transporting supplies, assessing damage [9], and rescuing and searching for victims [6]. In fact, in the disaster, the maximum number of _bosai_ helicopters used was only 47 [10] for the total of 1692 shelters (refugee centers) [6] in the three hardest-hit prefectures in Japan (Iwate, Fukushima, and Miyagi). It is predicted that it will be difficult to use many helicopters to transport supplies during a disaster. Thus, there is great interest in unmanned aerial vehicles (UAVs) which can be used regardless of the damage to the roads and in large numbers as a means for transportation. UAVs, also known as drones, are currently applied in several fields, such as aerial photography, surveying, and pesticide spraying. In addition, UAVs are used in the field of logistics due to improvements in UAV performance and related technologies, such as batteries [11]. Recently, there has been a growing interest in the field of logistics using UAVs. For instance, Amazon and Walmart have each been working on new platforms that use UAVs to deliver shipments to customers [12]. Similarly, other companies, such as DHL, Google, and Alibaba, also began developing their own UAVs [13]. UAVs have also been demonstrated to transport goods in emergency and disaster scenarios. [PERSON] et al. [14] conducted a series of drone transport experiments to demonstrate the use of drones for transporting emergency supplies. In Rwanda, Zipline transported blood, medicine, and vaccines to remote areas by drone [15]. On the other hand, most UAVs are smaller than helicopters, and the amount of items that can be transported at one time (maximum payload) is limited. Therefore, when transporting emergency relief supplies using UAVs, it is necessary to make the most effective use of the UAVs, taking into consideration their battery and maximum payload limitations. In recent years, there have been various studies on delivery using UAVs [16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27]. In general, these problems are formulated as UAV routing problems (UAVRPs), which are a special kind of vehicle routing problem (VRP) [16]. A VRP attempts to find the optimal routes for one or more vehicles to deliver commodities to a set of locations [17].