# UAV Cluster-Assisted Task Offloading for Emergent Disaster Scenarios
## 4 Deep Reinforcement Learning-Based UAV Cluster-Assisted Task Offloading Solution
### Markov Decision Process  
The environmental states of the UAV cluster-assisted task-offloading model are divided according to time slices, and its state space is continuous and countable, and the state at the next moment is only related to the current state, so the system model satisfies Markov properties. Therefore, this paper transforms the UAV cluster-assisted task-offloading problem into a Markov decision process. Markov decision processes represent sequential decision problems with Markov transfer models and additional rewards in fully observable stochastic environments. The Markov decision process consists of a quaternion \((\mathcal{S},\mathcal{A},\gamma,\mathcal{R})\), where \(\mathcal{S}\) is defined as the set of states, representing the observed UAV and ground user state information at each moment. The set of UAV actions executed on the observed states is defined as \(\mathcal{A}\). After the UAV executes an action based on the current state, the environment will enter a new state, the reward return for the current action is generated, and the set of reward returns is then denoted by \(\mathcal{R}\). \(\gamma\) denotes the discount factor determining the present value of future rewards. These state changes and action choices constitute the policy \(\pi\) of the intelligent agent, and the cumulative reward of the system is maximized by continuously updating the policy \(\pi\) during long-term interaction learning with the environment. In this paper, the UAV is considered an intelligent agent, and the motion trajectory and resource information of the UAV and ground users are considered to be the environment. Next, the state space, action space, and reward functions are defined in detail:  
1. \(\mathcal{S}\) denotes the state space, and each state \(s(t)\in\mathcal{S}\) is a multi-dimensional vector containing multiple parameters that represent the current state of the ground user and the UAV, including their current location and resource information, as well as the amount of task data and the task deadline for the ground user. Overall, the state \(s(t)\) of each time slice is defined as: \[s(t)=\big{(}L_{g}(t),L_{u}(t),M_{g}(t),T_{g}(t)\big{)},g\in\{1,2,\ldots,G\},u \in\{1,2,\ldots,U\}.\] (18)2. \(\mathcal{A}\) denotes the action space, which contains all feasible actions that the UAV can take. The main points are the flight direction and distance of the UAV, the task offloading of the ground user, the UAV task forwarding, and the proportion of data offloaded and forwarded. Therefore, the action \(a(t)\) for each time slice can be defined as follows: \[a(t)=\Big{(}d_{u}(t),\alpha_{g,u}(t),\beta_{g,u,u^{\prime}}(t),\lambda_{g1}(t), \lambda_{g2}(t)\Big{)},g\in\{1,2,\ldots,G\},u,u^{\prime}\in\{1,2,\ldots,U\},\] (19) where \(d_{u}(t)\in\{(x_{d},0),(-x_{d},0),(0,y_{d}),(0,-y_{d}),(0,0)\}\) denotes the UAV horizontal displacement vector, which is used to control the UAV flight direction or hovering. Thus, the UAV position for the next time slice can be expressed as \(L_{u}(t+1)=L_{u}(t)+d_{u}(t)\).
3. \(\gamma\) denotes the effect of the policy adopted by the current UAV on the reward returns at future moments, where \(0\leq\gamma\leq 1\).
4. \(\mathcal{R}\) denotes the reward set, which consists of the reward returns for the actions performed by the UAV in the current environmental state. The reward function at a certain moment mainly indicates the energy efficiency of the system and gives a certain penalty when the UAV flies out of the service area. Thus, the reward function at moment \(t\) is defined as follows: \[r(s(t),a(t))=\sum_{g=1}^{C}M_{g}(t)/\sum_{u=1}^{U}\sum_{u^{\prime}=1}^{U}E_{g, u,u^{\prime}}^{total}(t).\] (20)