## 3 Joint Optimization Algorithm Design
### MAHPPO Algorithm
#### 3.1.3 Reward Function  
The following cases are possible during the multi-UAV mission performance process:  
1. If the mission is not completed, the UAV with sufficient energy will continue to collect or offload data. Although the mission is not completed, the UAV does not have sufficient energy. Hence, the UAV is forced to stop performing the mission at this time. 3. The mission is successfully completed. Note that the UAVs must be offloading data in the last time slot. Therefore, we designed the reward function as \[r_{k}=\left\{ \begin{array}{ll}\sum_{u=1}^{U}\left(\alpha_{u,0,k}R_{u,0,k}\delta +\rho_{u,k}\right)+K^{*},&\text{Case 3}\\ \sum_{u=1}^{U}\left(\sum_{i=0}^{I}\alpha_{u,i,k}R_{u,i,k}\delta\right.\left.+ \rho_{u,k}\right)+E_{\text{pe}},&\text{Case 1 \&2}\end{array} \right.\] (19) where \(E_{\text{pe}}\) is defined as the energy penalty. \(\rho_{u,k}\) is the position penalty when UAV \(u\) violates constraints C8-C10. Furthermore, the UAV mission completion time is upper-bounded by \(K^{\text{max}}\). If the UAVs complete the mission ahead of time, let \(K^{*}{=}K^{\text{max}}{-}K\) denote the time reward. According to the reward function (18), P1 can be rewritten as  
\[(\text{P2}):\max_{A,\mathcal{P},\mathcal{V},\psi} \sum_{k=1}^{K}\sum_{i=0}^{M}R_{u,i,k} \tag{20}\] \[\text{s.t.C1}-\text{C16}.\]  
Since the action space contains two kinds of actions: a discrete variable \(\chi_{u,k}\) and three continuous variables \(\psi_{u,k},V_{u,k}\), and \(P_{u,k}\), the MAHPPO algorithm [26; 29] that can solve the hybrid discrete and continuous action space is employed. As shown in Figure 2, agents send state information \(\mathcal{S}_{k}\) to actor networks, and the number of actor networks is \(U\). These actor networks provide action policies \(\pi_{\theta_{g}}\left(A_{\text{d},\mathbf{k}_{u}}|S_{k}\right)\) and \(\pi_{\theta_{g}}\left(A_{\text{c},\mathbf{k}_{u}}|S_{k}\right)\) for the corresponding agents, where subscripts \(d\) and \(c\) represent the discrete and continuous parts of the actions, respectively. \(\theta_{u}\) is the parameter of the actor network for agent \(u\). For the convenience of expression, let \(\pi_{\theta_{g}}(A_{\mathbf{k}_{u}}|S_{k})\) denote the discrete or continuous strategies. In an actor network, the output branches of \(\pi_{\theta_{u}}(A_{\mathbf{k}_{u}}|S_{k})\) share multiple neural network layers. Since the MAHPPO algorithm adopts a random strategy, the discrete part of the actor network outputs the probability value of each discrete action. Moreover, the discrete action is sampled randomly based on the Softmax distribution. The continuous part of the actor network follows the parameterized Gaussian distribution and outputs the mean and variance of the continuous actions. The continuous part adopts the rule function in the last layer of the neural network. RL is a process in which an agent constantly interacts with the environment to find the optimal policy to maximize the cumulative discount reward \(R_{k}~{}=~{}\sum_{r=0}^{\infty}\gamma^{\tau}r_{k+1+\tau}\), where \(\gamma\) is a discount factor. The critic network outputs the state value function \(V_{\phi}(S_{t})=\mathbb{E}_{\pi_{\theta}}[R_{k}|S=S_{k}]\), where the expectation of \(R_{k}\) is obtained with the policy \(\pi_{\theta_{u}}(A_{\mathbf{k}_{u}}|S_{k})\). The critic network is updated by minimizing the loss function, shown as  
\[L(\phi)=\mathbb{E}\Big{[}\big{(}R_{k}-V_{\phi}(S_{k})\big{)}^{2} \Big{]}, \tag{21}\]  
where \(\phi\) is the critic network parameter.