## 4 Simulation Results and Discussion
4: Let \(\kappa_{k}=\kappa_{k}^{\mathrm{a}}\)
5:for\(u=1\)to\(U\)do
6:for\(u*=1\)to\(U\)do
7:if\(\kappa_{u,k}^{\mathrm{a}}=\kappa_{u*,k}^{\mathrm{a}}\) and \(\kappa_{u,k}^{\mathrm{a}}\
eq 0\)then
8:if\(g_{u,k}>g_{u*,k}\)then
9: Set \(\kappa_{u,k}=-1\). 10:endif
11:endif
12:if does not satisfy C3, C4, C6, C7, and C15 then
13: Set \(\kappa_{u,k}=-1\). 14:endif
15:endfor
16:endfor
17: Perform Lines 7-31 of Algorithm 1. ```  
**Algorithm 5** MAHPPPO algorithm. As shown in Figure 3, we first compared the accumulative reward of SU-MAHPPPO with the other three algorithms. The curves were smoothed for the convenience of observation. We can see that the proposed SU-MAHPPO algorithm was significantly better than the other algorithms. The SU-MAHPPO scheme could be stable and convergent after 10,000 episodes. Although the SU-MAPPO algorithm can quickly converge, the accumulated reward was lower than the SU-MAHPPO algorithm, and the curve suddenly dropped after 15,000 episodes of training. Moreover, the SU-MAPDQN algorithm cannot be convergent after 18,000 episodes. This is because the continuous action values corresponding to each value of discrete actions are output in MAPDQN, which obviously increases the computational complexity. Furthermore, the performance of the MAHPPO algorithm was far lower than that of our proposed algorithm within 18,000 episodes. This is because the complex action space and state space make it more difficult for the agents to explore. This confirms the advantages of the combination of the SAA algorithm and UAA algorithm to narrow the exploration domain of the agents. Figure 4 shows the effectiveness of the technologies mentioned in Section 3.1. It can be seen that the proposed SU-MAHPPPO algorithm without reward scaling had difficulty converging and was more unstable within 18,000 episodes. The reason is that the reward scaling is beneficial to the fitting of the value function. The curve of the SU-MAHPPPO algorithm without state and action normalization shows an upward trend, but it also cannot converge within 18,000 episodes. This is because the order of magnitude difference between the parameters in the state \(\mathcal{S}_{k}\) and action space \(\mathcal{A}_{k}\) is large, which is not conducive to the training of neural networks. Reward scaling is beneficial to the fitting of the value function. Hence, the above two techniques can effectively enhance the convergence and stability of the proposed algorithm. Figure 5 shows the UAV trajectories' comparison between the SU-MAHPPPO algorithm and SU-MAPPO algorithm. The mission completion time of SU-MAHPP  
Figure 4: Convergence without state and action normalization and reward scaling. Figure 3: Convergence with different algorithms. was 53 s and 67 s, respectively. SU-MAHPPO is better, since the designed reward function is positively related to the data transmission rate in each time slot. Moreover, the continuous action space of the SU-MAHPPO algorithm allows the agents to more accurately control the positions of the UAVs in each time slot. Besides, the continuous action space makes it more accurately adjust the transmit power of the buoys to maximize the accumulative reward. The discretized action space makes it difficult for the UAVs to reach the optimal trajectory for collecting and offloading data. Furthermore, it can be seen that the UAV trajectories of the proposed algorithm were not close to the buoy or OBS. This is because the UAVs can change the mission mode in different time slots. Take the trajectory of the UAV close to the buoy for example. In this case, the time of data collection will be reduced, but the UAV will be far away from the OBS, which makes it difficult to meet the SNR threshold of data offloading. Hence, the UAV needs more time to offload the data, so that the total mission completion time is longer. Figure 6 shows the trajectories of the three groups of UAVs taking off from different initial positions. The initial positions of the first group of UAVs are the same as the default positions, namely Position 1.