## 4 MADDPG-Based Integrated Space-Air-Sea Network System Cost Minimization Algorithm
### System Cost Minimization Algorithm for Space-Air-Sea Integrated Network Based on MADDPG
As shown in Algorithm 1, the main idea of the proposed algorithm is to enable each M-IoT device and UAV to learn, via centralized training and decentralized execution, continuous offloading and resource allocation policies that minimize a weighted sum of the delay and energy consumption in a partially observable environment, which first initializes the parameters of each intelligent agent's Actor network, Critic network, and two target networks. Then, the proposed algorithm undergoes E episodes, where each episode requires resetting the network environment to randomly generate the positions of the UAV and maritime IoT devices. Action exploration noise is added to ensure a normal distribution, which helps avoid suboptimal solutions due to local optimization during learning. At each time step, agents observe their state \(o_{i}(t)\) based on the current strategy, select the optimal action \(a_{i}(t)\) output by the Actor network, and obtain the \(Q\) value for that action from the Critic evaluation network and \(Q^{\prime}\) from the Critic target network. Each agent \(i\) executes the action \(a_{i}(t)\) and receives the corresponding reward \(r_{i}(t)\) and next observation \(o_{i}(t+1)\). Each agent then stores the experience tuple \((s(t),a(t),r(t),s(t+1))\) in the experience replay buffer. Next, a mini-batch of samples is randomly selected to update the Actor network parameters using the policy gradient and the Critic network using the minimum loss function. Finally, a soft update method is used to update the parameters of the target Actor and Critic networks. Then, considering the scalability of the proposed MADDPG-based algorithm, the CTDE architecture of the MADDPG enables scalable training by decoupling the critic's polynomially growing joint state action evaluation from the decentralized, real-time execution using only local observations. The empirical results demonstrate near-linear cost scaling under fixed UAV resources, and practical measures such as parameter sharing, distributed replay buffers, federated updates and observation filtering can keep the training overhead and communication demands manageable while accommodating heterogeneous agent additions. To evaluate the feasibility of the MADDPG-based algorithm in dynamic marine environments, this section analyzes the computational complexity of the proposed algorithm. The computational complexity of the algorithm is mainly composed of time complexity and space complexity. ```
1. Initialize the weight parameters of the Actor and Critic networks;
2. Initialize the weight parameters of the target Actor and target Critic networks;
3.for episode = 1 to E do:
4. Reset the space-air-sea integrated network environment;
5. Initialize the positions of the UAVs and maritime IoT devices;
6. Each M-IoT device receives the initial state \(O_{l}(t)\) and global state \(s(t)\) based on the task and network environment;
7.for each time step \(t=1\) to T do:
8.for\(i=1\) to N do:
9. Each intelligent agent \(i\) executes an action based on the current policy;
10. Executes the action \(a_{t}=\{a_{1}(t),a_{2}(t),\ldots a_{n}(t)\}\) in the environment, receives the reward \(r_{t}=\{r_{1}(t),r_{2}(t),\)\(\ldots r_{n}(t)\}\), and transitions to the next state \(s(t+1)\);
11.if the experience replay buffer is not full Then
12. Store (\(s(t),a(t),r(t),s(t+1)\)) in the experience replay buffer \(\mathcal{D}\);
13.else
14. Replace the oldest tuple in \(\mathcal{D}\) with the newly generated (\(s(t),a(t),r(t),s(t+1)\));
15. Randomly sample a mini-batch of \(B_{b}\) samples from \(\mathcal{D}\);
16. Minimize the loss function to update the Critic network parameters;
17. Maximize the policy objective function to update the Actor network parameters;
18.