## 5 Data Offloading Solution
### Task Offloading Decision  
Once given a certain resource value, the number of task splits is obtained. For example, if the amount of computing resources assigned to task \(t\) is \(f\), this task has \(\lceil f/f_{v}\rceil\) sub-tasks that are executed simultaneously. Currently, DRL, which is an extension of the customary reinforcement learning (RL), has been universally applied to many optimization issues. By exploring and receiving feedback from the environment, and combining with extensive state-action space exploration, DRL is good at finding the optimal offloading strategy. Using DRL for task allocation provides a main advantage. That is, during the learning phase, DRL learns the reward function which can be reasonably defined to capture multiple optimization objectives. Because of the advantages of DRL, DRL is adopted to identify specific task performers in this subsection. The key definitions of DRL are modeled as follows:  
Agent: An agent is able to select the next action based on the current state of the environment and make decisions through continuous interaction with the environment according to the observations. In the proposed UAV task offloading scenario, the cluster head in each UAV's monitoring scope plays the role of the agent. The agent is responsible for gathering information and studying the task placement strategy iteratively, so as to maximize the cumulative reward. State: Discovering a suitable representation of the input state is important to DRL model. Since the task uploading decision in our paper relies on the maximum tolerance delay, the data size of each task, and the locations of the parked cars in the cluster, the system state of the proposed DRL is represented as a vector \(S_{t}=(s_{t},l_{1},l_{2},\cdots,l_{n},T_{t}^{\text{max}})\), in which \(s_{t}\) is task data size, \(T_{t}^{\text{max}}\) indicates the maximum tolerance delay threshold, and \(l_{1}\), \(l_{2}\),..., \(l_{n}\) is the locations of the member nodes in the cluster within the coverage range of the UAV. Action: The cluster head makes the decision on which ground vehicles are allocated to the uploaded tasks. We use vector \(A_{t}=(a_{t1},a_{t2},\cdots,a_{tM})\) to represent the actions taken in state \(S_{t}\), in which \(a_{ij}\in\{0,1\}\) for \(\forall j\in\{1,2,\cdots,M\}\). \(a_{ij}=1\) denotes that vehicle \(n\) is chosen as the task performer. Different from traditional DRL, we have \(\sum\limits_{j=1}^{M}a_{ij}>1\). The reason is that, for each task, multiple candidates are chosen for task computing. This is because one vehicle does not have enough resources to meet the task delay requirement. Reward: At each state, the agent takes an action to receive the immediate reward. Our goal is to obtain the minimal task execution cost, while the goal of the DRL model is to maximize the reward. In order to retain consistency with the proposed minimization target, the reward function is defined as follows:  
\[r_{t}= \begin{cases}\frac{\tau\textit{rate}}{\lambda T_{t}+(1-\lambda)E_{t}}&T_{t }\leq T_{t}^{\text{max}}\\ 0&T_{t}>T_{t}^{\text{max}}\end{cases}  \tag{16}\]  
where _rate_ denotes the current task's successful ratio, and \(\tau\) is a coefficient which is exploited to avoid slow gradient descent speed due to a small reward value. Policy: Compared to traditional RL, neural network which brings various observations into the offloading policy is introduced into DRL. The output of the _softmax_ function of the neural network is taken as the policy in our learning-based model. The policy is presented as follows:  
\[\pi_{\theta}(s_{t},a_{tn})=P(a_{tn}|s_{t},\theta)=\frac{e^{\phi(s_{t},a_{tn}) \theta}}{\sum_{j=1}^{M}e^{\phi(s_{t},a_{tn})\theta}} \tag{17}\]  
This policy is used to determine the probability of using parked vehicle \(n\) to complete task \(t\) in state \(s_{t}\).