# A Reinforcement Learning Model of Multiple UAVs for Transporting Emergency Relief Supplies
## 4 Proposed Method
### State and Action  
In this study, the agents share a single \(Q\) matrix, which is updated sequentially by each agent after each transportation. The state of the environment in the \(Q\) matrix and the action of the agent corresponding to a UAV were defined as follows:  
* States: A state at instant \(t\), \(S_{t}\), represents the set of the remaining shelter demand at instant \(t\). \(S_{t}\) is denoted by \(S_{t}=Y_{t}=\{D_{1t},D_{2t},\ldots,D_{st}\}\), where \(D_{jt}\) is the set of demands of shelter \(j\) at instant \(t\).  
Figure 4: The agent from a reinforcement learning point of view. Image from [50].  
* Actions: Selection of the following two elements among all possible actions: (i) the shelter/depot to transport/return and (ii) the amount of items that the UAV transports. Note that the types of supplies are selected in order of urgency among the supplies demanded by the destination shelter. An action at instant \(t\), \(A_{t}\), is denoted by \(A_{t}=\{L_{m},w_{ijklmn}\}\).  
Now consider the following example, where \(N_{Ut}=1\). The initial state \(S_{t=0}\) is denoted as \(S_{t=0}=\{\{2,2\},\{1,1\}\}\), which means that there are 2 shelters for transportation and the initial demand for Shelter 1 and Shelter 2 are \(D_{1t(t=0)}=\{2,2\}\) and \(D_{2t(t=0)}=\{1,1\}\), respectively. We assume that the initial location of UAV 1 is depot \((j=0)\). In the case of the action of the UAV, 1 is \(A_{t=0}=\{1,3\}\), which means that UAV 1 transports 3 units of supplies to Shelter 1, the new state at instant \(t=1\), \(S_{t=1}\), represented as \(S_{t=1}=\{\{0,1\},\{1,1\}\}\). Subsequently, in the case of the action of the UAV 1 is \(A_{t=1}=\{2,1\}\), which means that UAV 1 transports 1 unit of supplies to Shelter 2, the new state at instant \(t=2\), \(S_{t=2}\), represented as \(S_{t=2}=\{\{0,1\},\{0,1\}\}\). By repeating this process, the action of the agent continues until the current episode is completed.  
In each state \(S_{t}\), the UAV (agent) determines action \(A_{t}\) according to the flow, as shown in Figure 5. Here, we assume that there are six shelters, as shown in Figure 1, and that the maximum payload of the UAV is 5 (\(C=5\)). First, the agent generates \"_All_action_list_\" for each state. This is the set of combinations of destinations and amounts of transport: When the agent is at the depot, there are 30 combinations (6 destinations \(\times\) 5 amounts), and when the agent is at a certain shelter, there are 26 combinations (5 destinations except the current location \(\times\) 5 amounts + return to the depot). Second, the agent determines an action \(A_{t}\) based on the \(\epsilon\)-greedy policy (for the policy, see the next section). Finally, if the constraints of the maximum payload (Equation (3)) and maximum amount of energy (Equation (4)) are satisfied, action \(A_{t}\) is chosen; otherwise, the current action is deleted from \"_All_action_list_\", and the action is chosen again.  
The termination criteria are the conditions when the demands for all shelters are zero. It is described as follows:  
\[d_{ijt}=0\quad\forall i,\forall j \tag{17}\]  
The termination time \((t=T)\) was defined as the time when the above conditions were met and the UAV returned to the depot.  
Figure 5: The flow of UAV action determination at each action instance of each episode.