## 4 MADDPG-Based Integrated Space-Air-Sea Network System Cost Minimization Algorithm
### System Cost Minimization Algorithm for Space-Air-Sea Integrated Network Based on MADDPG  
The MADDPG algorithm proposed in this section comprises intelligent agents within the space-air-sea integrated network environment. The flowchart of the MADDPG algorithm is shown in Figure 3, with a detailed introduction to the training and testing process of intelligent agent \(i\). At each time step, each intelligent agent obtains its local observation \(o_{i}(t)\) and executes its action \(a_{i}(t)\), ultimately receiving its reward \(r_{i}(t)\). Then, the environment updates the state and transitions to a new state \(s_{i}(t+1)\). Each intelligent agent is equipped with an Actor network \(\mu_{i}(o_{i}(t)\Big{|}\theta_{i}^{\mu})\) and a Critic network \(Q_{i}\big{(}s_{i}(t),a_{i}(t)\big{|}\theta_{i}^{\omega}\big{)}\), with the corresponding target networks \(\mu_{i}^{\prime}\Big{(}o_{i}(t)\Big{|}\theta_{i}^{\mu^{\prime}}\Big{)}\) and \(Q_{i}^{\prime}\Big{(}s_{i}(t),a_{i}(t)\Big{|}\theta_{i}^{\omega^{\prime}} \Big{)}\). The primary network and target network have the same structure but different parameters. During the training phase, the Actor network of intelligent agent \(i\) takes the agent's partial observation \(o_{i}(t)\) as input, selects the action \(a_{i}(t)\) based on the policy, and adds a certain amount of noise to the final output action, which can be defined as:  
\[a_{i}(t)=\mu_{i}\Big{(}o_{i}(t)\Big{|}\theta_{i}^{\mu}\Big{)}+\Psi(t) \tag{19}\]  
Here, \(\Psi(t)\) is the exploration noise that is used to encourage the agent to explore, and \(\mu_{i}\Big{(}o_{i}(t)\Big{|}\theta_{i}^{\mu}\Big{)}\) is the action output by the current Actor network of agent \(n\). In this way, the actions of all the agents can obtain rewards in the given environment, execute actions, and transition to the next state. Then, the combination of each agent's state, action, reward, and next state \((s(t),a(t),r(t),s(t+1))\) is stored in the experience replay buffer \(\mathcal{D}\). When the accumulated count of experiences in the buffer reaches a certain amount, a mini-batch of samples is fed into the Actor and Critic networks of each agent. The sampled mini-batch is then input into the target Critic network to output the target network \(Q\) values as:  
\[y_{i}(j)=r_{i}(j)+\gamma Q_{i}^{\prime}\Big{(}s(j+1),\mu_{i}^{\prime}\Big{(}o_ {i}(j+1)\Big{|}\theta_{i}^{\mu^{\prime}}\Big{)}\Big{|}\theta_{i}^{\omega^{ \prime}}\Big{)} \tag{20}\]  
Here, \(\gamma\) represents the discount factor, which is an important coefficient used to weigh current rewards against future rewards, and \(\gamma\in[0,1]\). Figure 3: MADDPG algorithm flowchart. The Critic network is updated by minimizing the loss function, which is defined as:  
\[L(\theta_{i}^{\omega})=\frac{1}{B_{b}}{\sum_{j}}{\left[Q_{i}(s(j),a(j)\,|\,\theta_ {i}^{\omega})-y_{i}(j)\right]^{2}} \tag{21}\]  
Here, \(B_{b}\) is the batch sample size. For the Actor network, the network parameters are updated by maximizing the policy objective function, which is defined as:  
\[J\left(\theta_{i}^{\omega}\right)=\frac{1}{B_{b}}{\sum_{j}}Q_{i}(s(j),a(j)| \theta_{i}^{\omega}) \tag{22}\]  
Finally, the parameters \(\theta_{i}^{\omega^{\prime}}\) and \(\theta_{i}^{\mu^{\prime}}\) of the target network are updated using a soft update method as follows:  
\[\theta_{i}^{\omega^{\prime}} \leftarrow\xi\theta_{i}^{\omega}+(1-\xi)\theta_{i}^{\omega^{ \prime}} \tag{23}\] \[\theta_{i}^{\omega^{\prime}} \leftarrow\xi\theta_{i}^{\omega}+(1-\xi)\theta_{i}^{\omega^{ \prime}} \tag{24}\]  
Here, \(\xi\) is the soft update parameter, which is a smoothing factor used to control the update rate of the target network parameters, with \(\xi\in[0,1]\). Algorithm 1 lists the steps for implementing the total cost minimization problem based on the MADDPG algorithm.