# UAV Cluster-Assisted Task Offloading for Emergent Disaster Scenarios
## 4 Deep Reinforcement Learning-Based UAV Cluster-Assisted Task Offloading Solution
### Drl-Ucto Algorithm Implementation  
The DRL-UCTO algorithm process is defined as shown in Algorithm 1, which consists of two phases for each training episode, the exploration phase (steps 4-15) and the training phase (steps 16-19). During the exploration phase, the UAV flight trajectory and the assisted offloading policy will be obtained by either the random or greedy methods defined above. If the current behavior causes the UAV to leave the service area, then the reward \(r(s(t),a(t))\) in the current state is penalized, the current behavior is canceled, and the flight policy of the UAV is set to hover to control the UAV to stay in the current position. The environment will then move to the next state \(s(t+1)\) and receive the corresponding current reward \(r(s(t),a(t))\). Meanwhile, the sample data \((s(t),a(t),r(s(t),a(t)),s(t+1))\) will be stored in the experience buffer \(D\).  
In the training phase, this paper will use a random sampling method to randomly select \(n\) training samples from the experience buffer for updating the main network \(Q\) and the target network \(Q^{\prime}\), thus breaking the correlation between the data. First, the weight parameter \(\varphi^{Q}\) of the main network is updated by minimizing the loss in step 18 through \(y(t)\) generated by the target network. Next, the weight parameter \(\varphi^{Q^{\prime}}\) of the target network is periodically replaced by the weight parameter \(\varphi^{Q}\) of the main network, thus completing the update of the target network, but the target network is updated less frequently than the main network, to avoid the overestimation problem. The algorithm execution is terminated until the end of the assigned training episodes, or the computational tasks of each ground user have been completed.  
```
1: Initialize the replay buffer D;
2: Randomly initialize main network \(Q\) and target network \(Q^{\prime}\) with weights \(\varphi^{Q}\) and \(\varphi^{Q^{\prime}}\), and set \(\varphi^{Q^{\prime}}=\varphi^{Q}\);
3:for episode \(=1,\ldots,E\)do
4: Initialize the environment to state s(1);
5:for\(t=1,\ldots,T\)do
6: Select action \(a(t)\in A(s(t))\) based on policy \(\pi(s(t))\);
7: Execute \(a(t)\) obtain \(r(s(t),a(t))\) and \(s(t+1)\);
8:for\(u=1,\ldots,U\)do
9:if UAV \(u\) flies out of service scope then
10: Apply the penalty to \(r(s(t),a(t))\);
11: Cancel the movement of UAV \(u\) and update \(s(t+1)\);
12:endif
13:endfor
14: Store training sample \((s(t),a(t),r(s(t),a(t)),s(t+1))\) in \(D\);
15:endfor
16: Select a random mini-batch of training samples \((s(i),a(i),r(s(i),a(i)),s(i+1))\) of size \(n\) from \(D\);
17:\(y(i)=r(s(i),a(i))+\gamma Q^{\prime}\left(s(i+1),\arg\max_{a(i+1)}Q\big{(}s(i +1),a(i+1)\mid\varphi^{Q}\big{)}\mid\varphi^{Q^{\prime}}\right)\);
18:\(L\left(\varphi^{Q}\right)=\mathbb{E}\left[y(i)-Q\big{(}s(i),a(i)\mid\varphi^ {Q}\big{)}\right]\);
19: Update weights \(\varphi^{Q^{\prime}}\) of \(Q^{\prime}\): \(\varphi^{Q^{\prime}}=\varphi^{Q}\);
20:endfor
```  
**Algorithm 1** Deep Reinforcement Learning-Based UAV Cluster-Assisted Task-Offloading Algorithm (DRL-UCTO).