# Energy Saving Strategy of UAV in MEC Based on Deep Reinforcement Learning
## 5 The DOTO Algorithm
### Offload Function Iteration  
At the \(t-\)th time block, the wireless channel gain is \(c_{t}\), where \(t=1,2,\ldots\). When \(t=1\), the parameter \(\omega_{t}\) of the DNN is randomly initialized according to the standard normal distribution with zero mean. The DNN outputs the first offloading action according to the formula \(\widetilde{u}_{t}=\Gamma_{\omega_{t}}(c_{t})\), which is expressed by the following formula:  
\[\widetilde{u}_{t}=\{\widetilde{u}_{t,j}|\widetilde{u}_{t,j}\in(0,1),i=1,\ldots,N\,\} \tag{15}\]  
In the hidden layer of the neural network, this paper chooses the ReLU function as the activation function. To relax the offloading action of the output, the Sigmoid function is used as the activation function in the output layer. In this paper, \(\widetilde{u}_{t}\) is quantified into \(K\) groups of binary offloading actions by the KNN method, and each group has \(N\) binary data. Let the quantization function \(\Upsilon_{K}\) be:  
\[\Upsilon_{K}:\widetilde{u}_{t}\rightarrow\{u_{\rm k}|u_{\rm k}\in\{0,1\}_{N},k=1,\ldots,K\,\} \tag{16}\]  
There are a total of \(N\) UAVs, so the value range of \(K\) is \(K\in[1,2^{N}]\). When \(K\) is larger, the number of calculations for the residual energy problem will be greater, and the computational complexity of the DOTO algorithm will be higher, but the quality of the decision-making scheme will be better. Similarly, the smaller the \(K\), the lower the computational complexity and the worse the quality of decision-making scheme. In general, setting a larger \(K\) can obtain more UAV residual energy at the cost of higher complexity. However, if a large number of quantization actions are generated in each time block, it is very unnecessary and the efficiency will be very low. Therefore, in order to balance the quality of the algorithm decision-making scheme and the computational complexity, we propose an adaptive \(K\) value setting method. The details are as follows:  
1. Initially, set the \(K\) value to \(2^{N}\). During training, all alternatives generated by the KNN quantization method are sorted by Euclidean distance.
2. Each time block records the index value \(K\) corresponding to the maximum residual energy offloading action. Every \(\Delta\) time block, compare the largest \(K_{\Delta}\) in the \(\Delta\) time block with the \(K_{\Delta-1}\) in the \(\Delta-1\) time block, and select the largest \(K+1\) value as the next iteration.
3. In order to avoid excessive training loss caused by an overly small \(K\) value, when \(K<N\), we set the \(K\) value as \(N\), i.e., the value range of \(K\) is \(K\in[N\,,\,2^{N}]\).  
Figure 3: The structure of the DOTO algorithm.  
The adaptive \(K\) value setting is expressed by the formula below:  
\[K=\left\{\begin{array}{c}2^{N},t=1\\ \max(\ \max(\ {k_{t-1},\ldots,k_{t-\Delta}})+1,\ N)\,t\ \text{mod}\ \Delta=0\end{array}\right. \tag{17}\]  
After obtaining \(K\) offloading decisions at the \(t-\)th time block, we can calculate the UAV residual energy corresponding to each offloading decision by solving the \(P2\) problem. Therefore, the optimal offloading decision \(u_{t}^{*}\) corresponding to the maximum residual energy is selected, which can be expressed as follows:  
\[u_{t}^{*}=\arg\max_{u_{i}\in\{u_{k}\}}E^{*}(c_{t}\,u_{i}\ ) \tag{18}\]  
Finally, output the optimal offloading decision.