## 3 MADDPG Approach for Cooperative Multi-USV Network
### Multi-Agent DDPG Approach
\(=0\)
3:for step \(t=1\) to \(T\)do
4:if Counter \(<\) C then
5: each USV \(i\) randomly chooses \(a_{i}\);
6:
7:else
8:\(a_{i}=\mu_{\theta_{i}}\Big{(}o^{i}_{\text{new}}\Big{)}+\mathcal{N}\)
9:endif
10: Execute actions \(a=[a_{1},a_{2},\dots a_{N}]\), and observe reward \(R\), new states \(o_{new}\);
11: Store transition \((o,a,r,o_{\text{new}})\) into experience replay buffer \(D\);
12: Sample a mini-batch of \(\theta\) transitions \((o^{m},a^{m},r^{m},o^{m}_{\text{new}})\) from replay. Buffer \(D\);
13: Set \(y^{m}=R^{m}+\left.\gamma Q^{\mu}_{i}\left(o^{m}_{\text{new}},a^{\prime}_{1}, \dots,a^{\prime}_{N}\right)\right|_{a^{\prime}=\mu^{\prime}_{i}(o_{i})}\)
14: Update critic by minimizing the loss \(L(\theta_{i})=\frac{1}{5}\sum\limits_{m=1}^{5}\Big{(}y^{m}-Q^{\mu}_{i}(o^{m}_{ \text{new}}\), \(a_{1},\dots,a_{N})\Big{)}^{2}\)
15: Update actor using the sampled policy gradient: \(\
abla_{\theta_{i}}J=\frac{1}{5}\sum\limits_{m=1}^{5}\
abla_{\theta_{i}}\mu_{ \theta_{i}}(o^{m}_{i})\
abla_{a_{i}}Q^{\mu}_{i}(o^{m},a_{1},a_{2},\dots,a_{N}) \Big{|}_{a_{i}=\mu_{\theta_{i}}(o^{m}_{i})}\)
16: Updating actor networks  Updating critic networks  Update target networks with updating rate \(\tau\): \(\theta^{\mu^{\prime}}_{i}\)\(\leftarrow\)\(\theta^{\mu}_{i}+(1-\tau)\theta^{\mu^{\prime}}_{i}\) \(\theta^{\Omega^{\prime}}_{i}\)\(\leftarrow\)\(\theta^{\Omega}_{i}+(1-\tau)\theta^{\Omega^{\prime}}_{i}\)
17:endfor
18:endfor
```  
**Algorithm 1**Training algorithm using the MADDPG framework. Now we resort to the MADDPG algorithm that extends DDPG to the multi-agent scenario. Specifically, a multi-agent game with L agents, \(\mu_{\theta}=\{\mu_{\theta_{1}},\dots,\mu_{\theta_{L}}\}\), which is parameterized by \(\theta=\{\theta_{1},\ldots,\theta_{L}\}\), can be regarded as the set of all deterministic policies for all agents, the gradient of the expected return for each agent is expressed as  
\[\
abla_{\theta_{i}}J\big{(}\mu_{\theta_{i}}\big{)}=\mathbb{E}_{G,a\sim\mathcal{ D}}\bigg{[}\
abla_{\theta_{i}}\mu_{\theta_{i}}(a_{i}\mid s_{i})\times\
abla_{a_{i}}Q_{i}^{ \mu}(o,a_{1},a_{2},\ldots,a_{L})\Big{|}_{a_{i}=\mu_{\theta_{i}}(s_{i})}\bigg{]} \tag{15}\]  
where \(Q_{i}^{\mu}(o,a_{1},a_{2},\ldots,a_{L})\) is a centralized function with the inputs of all the agents' action \(a=\{a_{1},\ldots,a_{L}\}\), as well as the observation information \(o\). \(G=\{o_{1},o_{2},\ldots,o_{L}\}\) represents the information that the environment feeds back to the agent, including the distance between agents, the distance between agent \(i\) with surrounding obstacles, and the distance from agent \(i\) to each targets, all of which constitute the observation of all agents that are denoted by \(\big{[}\text{x},\text{y},d_{i}^{U},d_{i}^{O},d_{i}^{T}\big{]}\). In our networks, the inputs of the critic network are the sates of all the USVs, the distance between USVs, as well as distance from the USV to the target area. The output of the critic network, i.e., \(Q_{i}^{\mu}(o,a)\) is returned to the actor network for evaluating the action, which can be seen in Figure 2. The actor \(i\) executes the action based on its own policy \(\mu_{\theta_{i}}\), and the critic uses the \(Q_{i}^{\mu}\) function to evaluate the action of the actor. In order to remove the correlation of the samples generating from the environment, the framework adopts the experience replay mechanism to store the experience of each agent, which is a finite buffer with capacity \(M\) expressed as the tuple \((o,o^{\prime},a_{1},\ldots,a_{L},r_{1},\ldots,r_{L})\). When the replay buffer is full, the oldest samples will be discarded. In order to better adapt to the environment and improve the stability of exploration, we updated the target network in a \"soft update\" way, so that it can slowly track the learning strategy, which can be expressed as \(\theta^{\prime}\gets\tau\theta+(1-\tau)\theta^{\prime}\).