## 3 MADDPG Approach for Cooperative Multi-USV Network
### Multi-Agent DDPG Approach  
In the collaborative MARL scenario, each agent engages in interactions with the environment and acquires team rewards to facilitate collaboration. Single-agent reinforcement learning algorithms, such as proximal policy optimization (PPO) and the deep Q network (DQN), can be straightly utilized to deal with collaborative scenarios by letting each agent learn its optimum Q function in an independent manner. Nevertheless, the environment can be unstable from the perspective of any agent. To tackle the unstable issue in MARL, we utilized the MADDPG algorithm [38] learning a centralized Q-function for each agent as per the global information. When all the actions of the agents are known, the environment is stable. This is due to  
\[\mathcal{P}\big{(}s^{\prime}\mid s,a_{1},\ldots,a_{L},\pi_{1},\ldots,\pi_{L} \big{)}=\mathcal{P}\big{(}s^{\prime}\mid s,a_{1},\ldots,a_{L}\big{)}=\mathcal{ P}\big{(}s^{\prime}\mid s,a_{1},\ldots,a_{L},\pi_{1}^{\prime},\ldots,\pi_{L}^{ \prime}\big{)} \tag{12}\]  
for any \(\pi_{i}\
eq\pi_{i}^{\prime}\), in which \(\pi_{i}\) is the policy of agent \(i\). The aim of each agent is to select the beneficial policy that maximizes the accumulated reward \(J(\theta)=\mathbb{E}_{s\sim p^{\pi},a\sim\pi_{\theta}}[R]\) prior to the description of the MADDPG arithmetic. Let us start with the policy gradient (PG) algorithm utilized for the continuous control issue. The crucial goal of the PG is to straightly adjust the parameter \(\theta\) of policy \(\pi\) at the orientation of \(\
abla_{\theta}J(\theta)\), which is expressed as  
\[\
abla_{\theta}J(\theta)=\int_{\mathcal{S}}\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!  
\[\
abla_{\theta}J(\theta)=\int_{S}\rho^{\mu}(s)\
abla_{\theta}\mu_{\theta}(s)\
abla_ {a}Q^{\mu}(s,a)\bigg{|}_{a=\mu_{\theta}(s)}ds=\mathbb{E}_{s\sim p^{\mu}}\Big{[} \left.\
abla_{\theta}\mu_{\theta}(s)\
abla_{a}Q^{\mu}(s,a)\right|_{a=\mu_{ \theta}(s)}\Big{]} \tag{14}\]  
It is obvious from (14) that the DPG arithmetic prevents the integral over the entire action space, which can decrease the computational intricacy and ameliorate the training efficiency. In contrast to PG, DPG can tackle the difficult enhancement issue with high-dimensional actions. As the extension of DPG, DDPG adopts the deep neural network (DNN) to approximate the policy \(\mu\) and critic \(Q^{\mu}(s,a)\). Nevertheless, the updated network \(Q^{\mu}(s,a|\theta^{Q})\) cannot be straightly utilized for estimating the target value, which will induce the update \(Q\) divergent. Hence, the DDPG algorithm utilizes the soft target updates rather than straightly copying the weights of the update \(Q\). Specifically, DDPG utilized a copy of the actor and critic networks to calculate the target value, which is denoted as \(Q^{\prime}(s,a)\) and \(\mu^{\prime}_{\theta}(s)\), respectively. For the sake of suiting the environment and ameliorating the exploration efficiency, another trick of DDPG is to add stochastic noise into the actor policy, as presented in Algorithm 1. ```
Parameters: batch size \(\beta\), training episodes. \(M\), training step \(T\), action noise \(\mathcal{N}\) and the number of USVs \(N\), actor networks' weights for each agent \(\theta^{\mu^{\prime}}_{i}\)
1:For episode = 1 to \(M\)do
2: Initialize observations \(\mathcal{O}_{\text{init}}\), \(\mathcal{O}_{\text{new}}\)\(\leftarrow\)\(\mathcal{O}_{\text{init}}\). Initialize: Actor network \(\mu\), critic network Q with weights \(\theta^{\mu},\theta^{Q}\). Initialize: Target actor, critic network: \(\mu^{\prime}\), \(Q^{\prime}\), with weights \(\theta^{\mu^{\prime}}\)\(\leftarrow\)\(\theta^{\mu^{\prime}}\), \(\theta^{Q^{\prime}}\)\(\leftarrow\)\(\theta^{Q}\). Initialize: Replay buffer \(D\) with capacity \(C\), exploration counter.