## 3 Joint Optimization Algorithm Design
### SMP-Based Buoy-UAV Association Algorithm  
Although the designed MAHPPO algorithm can determine the UAV mission modes, the association relationship between the buoys and UAVs is still unknown. In this subsection, we focus on addressing the association relationship problem. Given the transmit power of the buoys and the flight velocity and flight angle of the UAVs, P1 can be further written as  
\[(\text{P3}):\max_{A}\sum_{k=1}^{K}\sum_{i=0}^{M}R_{u,i,k} \tag{25}\] \[\text{s.t.C3},\text{C4},\text{C6},\text{C7},\text{C11}-\text{C16}.\]  
Since P3 is a mixed-integer non-convex problem, the SMP method is used to obtain the optimal solution in an iterative manner. The SMP was first proposed in [32], which contains two groups of elements, called the male set and female set. Everyone in the set has his/her own preference list for the opposite sex. According to the preference list, the current match was considered stable if there was not one male or female who rated the female or male in the other couples as better than the current partner in any two couples. The Gale-Shapley (GS) algorithm effectively solves the above problems. The main idea of the GS algorithm is to take the male's point of view (either male or female) and let a male select a female according to his/her preference list. The female temporarily accepts the choice until a better man chooses her and then updates her choice. According to the above process, the two sets continuously cycle and update the selection until the match is stable. Figure 2: The MAHPPO algorithm structure of agent \(u\). ```
1: Initialize the actor network parameters \(\theta_{u},\forall u\in\mathcal{U}\) and the critic network parameter \(\phi\). 2: Initialize the experience replay buffer \(\mathcal{D}\) with size \(D\), mini-batch size \(D^{\prime}\), sample reuse time \(N\), and state \(S_{0}\). 3:for episode = 1 to \(Ep_{\text{max}}\)do
4: Initialize the number of total mission completion time slots \(k=1\), the energy penalty \(E_{\text{pe}}\), and the position penalty \(\rho_{k}\). 5:while done is not True do
6: Every agent sample action \(A_{u,k}\) with \(\pi_{\theta_{u}}(A_{k_{k}}|S_{k})\). 7:if the UAV \(u\) flies beyond the target area then
8:\(\rho_{k}=1/U\), cancel the actions of \(V_{u,k}\) and \(\psi_{u,k}\), and update \(r_{k}\), \(S_{k+1}\) based on the current state. 9:endif
10:if\(E_{u}^{\text{total}}>\bar{E}_{u},\forall u\in\mathcal{U}\)then
11: Let done be True, and \(E_{\text{pe}}=-50\)
12:endif
13: Get the reward \(r_{k}\) and next state \(S_{k+1}\). 14: Store experience data \((S_{k},A_{k},r_{k},S_{k+1})\) in \(\mathcal{D}\). 15:\(k=k+1\). 16:if UAVs complete the mission or \(k=K^{\text{max}}\)then
17: Let done be True. 18:endif
19:if\(\mathcal{D}\) is full then
20: Compute advantages \(\hat{A}_{k}\) and the state value \(V_{\phi}(S_{t})\). 21:for epoch = 1 to \(N\times(D/D^{\prime})\)do
22: Sample \(D^{\prime}\) from \(\mathcal{D}\). 23: Compute \(L(\theta_{u})\) of each actor network by (21). 24: Update each actor network by a gradient method \(\theta_{u}\leftarrow\theta_{u}-\beta_{a}\
abla_{\theta_{u}}L(\theta_{u})\). 25: Compute \(L(\phi)\) by (18). 26: Update critic network by \(\phi\leftarrow\phi-\beta_{a}\
abla_{\phi}L(\phi)\). 27:endfor
28: Clear the experience replay buffer \(\mathcal{D}\). 29:endif
30:endwhile
31:endfor
```  
**Algorithm 1** Algorithm. In this paper, the buoy-UAV association relationship problem can be regarded as a preference ordering problem, which is a standard SMP. The UAVs and buoys can be regarded as males and females, respectively. In contrast to the traditional SMP problem, \(U\) and \(M\) may not be equal. At time slot \(k\), the goal of minimizing the males' cost can be regarded as maximizing the total amount of data collected by the UAVs. The amount of data collected is positively correlated with the SNR, so the preference value could be denoted by the SNR between the UAVs and buoys.