# UAV Cluster-Assisted Task Offloading for Emergent Disaster Scenarios
## 4 Deep Reinforcement Learning-Based UAV Cluster-Assisted Task Offloading Solution
### Deep Reinforcement Learning-Based UAV Cluster-Assisted Task-Offloading Algorithm (DRL-UCTO) Idea  
Deep reinforcement learning has a strong dependence on the before and after states of the environment, and often the state at the next moment is not only related to the current state, but even has a close connection with the earlier state or the initial state. Due to many user devices and UAVs as well as multiple decision variables in the UAV cluster-assisted task offload model, such a high-dimensional and highly correlated state sequence will lead to a complex state transfer model of the system environment, which is difficult to model. In addition, multilayer neural networks in deep reinforcement learning indicate the existence of computations with higher order polynomials, which will lead to the gradient vanishing problem [34]. Therefore, this paper simplifies the environmental state transfer model for deep reinforcement learning by transforming the system problem into a Markov decision process so that the environmental state at the next moment is only related to the current state, thus making the modeling feasible. Based on this, the deep reinforcement learning-based UAV cluster-assisted task-offloading algorithm (DRL-UCTO) is proposed. The DRL-UCTO algorithm is mainly based on the DDQN algorithm, whose core idea is to model the UAV cluster flight trajectory and the scheduling offload policy to ground user devices as discrete actions \(a(t)\). To evaluate the action quality, each state-action pair \((s(t),a(t))\) is measured using a \(Q\)-value, where \(Q\)-value is defined as:  
\[Q(s(t),a(t))=\mathbb{E}\Bigg{[}\sum_{t^{\prime}=t}^{T}\gamma r\big{(}s\big{(} t^{\prime}\big{)},a\big{(}t^{\prime}\big{)}\big{)}\ |\ s(t),a(t)\Bigg{]}. \tag{21}\]  
The meaning of \(Q\)-value indicates the cumulative discount reward expected under the current policy. During the learning process of interaction with the environment, the policy of the intelligent agent is continuously adjusted by evaluating the \(Q\)-value, and the adjusted policy is used to find the best action. In general, the policy uses a greedy algorithm to select the action that produces the highest \(Q\)-value in state \(s(t)\), which obtains the optimal UAV trajectory and the optimal offloading policy, with the following expression:  
\[\pi(s(t))=\arg\max_{a(t)}Q(s(t),a(t)). \tag{22}\]DRL-UCTO uses two deep neural networks with weights to replace the Q-table in the original Q-learning algorithm, thus solving the problem that the Q-learning algorithm cannot be used in high-dimensional action spaces, which are, respectively, called the main network and the target network. The main network generates predictions \(Q(s(t),a(t)\mid\varphi^{Q})\) based on the input states and actions, where \(\varphi^{Q}\) denotes the weight parameter of the main network, and the main network also selects the behavioral action for the next state according to (22). In the deep neural network learning process, the weight parameter \(\varphi^{Q}\) of the network is trained using the loss function. \[L\Big{(}\varphi^{Q}\Big{)}=\mathbb{E}\Big{[}y(t)-Q\Big{(}s(t),a(t)\mid\varphi^ {Q}\Big{)}\Big{]}, \tag{23}\]  
where \(y(t)\) is the target value, which is evaluated by the target network \(Q^{\prime}\) while considering the predicted results of the main network \(Q\), the calculation process is expressed as follows:  
\[y(t)=r(s(t),a(t))+\gamma Q^{\prime}\bigg{(}s(t+1),\text{arg}\max_{a(t+1)}Q \Big{(}s(t+1),a(t+1)\mid\varphi^{Q}\Big{)}\mid\varphi^{Q^{\prime}}\bigg{)}, \tag{24}\]  
where \(\varphi^{Q^{\prime}}\) denotes the weight parameter of the target network. Since the traditional DQN uses the greedy policy in (22) to select the action with the largest \(Q\) value, the obtained target value \(Q^{\prime}(s(t),a(t)\mid\varphi^{Q^{\prime}})\) is often larger than the actual value, which leads to the overestimation problem.