## 3 MADDPG Approach for Cooperative Multi-USV Network
### Markov Game for Multi-USV Cooperation  
To study the best behavior mode in given states, we employ a Markov decision process (MDP) [39] to formulate the AUV motion planning, as the Markov decision process (MDP) offers a mathematical framework for simulating the randomized strategies that can be implemented with Markov states in a given scene. The method is modeled as tuples \((S,A,R,P,\gamma)\), and in this case, the tuples are action set \(A\), state set \(S\), state transition probability \(P\), discount factor \(\gamma\), and reward function \(R\) of the system. The action selection of the agent, a functional mapping from each state \(s_{i}^{t}\in\mathcal{S}\) to action \(a_{i}^{t}\in\mathcal{A}\), is modeled as policy \(\pi\). The value function \(v_{\pi}(s)\) of the process is defined as the expected sum of discount rewards that act continuously from and along with state \(s\). In case a policy \(\pi\) can attain the best from any initial state \(s_{i}^{t}\in\mathcal{S}\), we detect it as an optimal policy \(\pi^{*}\). This is the target of this training of the deep neural network with a deep reinforcement learning module. Agent: Each USV can be considered an agent. USVs maintain a certain distance from each other during navigation to facilitate communication, and each USV obtains a state containing its own information and the information of the surrounding environment in the process of interaction with the environment. The environment in this problem is completely observed; therefore, the observations are equivalent to the state. Each actor has its own actor and critic network which act as the executor and evaluator of the policy, respectively. Each agent observes its own state, takes actions according to its own strategy, and then obtains rewards from the environment to reach the next state, as shown in Figure 2.  
State: The state of any agent is a one-dimensional vector with five components, \(\left[\mathrm{x},\mathrm{y},d_{i}^{U},d_{i}^{O},d_{i}^{T}\right]\). The first two elements represent the current 2D position of the agent, the third entry \(d_{i}^{U}\) represents the distance between agents, and the fourth entry \(d_{i}^{O}\) represents the distance between agent \(i\) and the surrounding obstacles. The fifth element \(d_{i}^{T}\) represents the distance from agent \(i\) to the current matching target which can be changed with the changes in the USV's selection. Action: The action for each agent is the output of its actor network, which is expressed by \(v_{i}=u_{i}(o_{i})\). This is a Gaussian distribution where \(v_{i}\) is composited by the velocities of the \(x\) and \(y\) axes. An action is executed by the formula \(\overrightarrow{a}=\overrightarrow{v}*\Delta t+\mathcal{N}_{t}\), where \(\overrightarrow{a}\) is a displacement vector and \(\mathcal{N}_{t}\) is a random noise. The action is determined by the position of the target area and the risk factors of the surrounding environment. Reward: Each USV obtains its own reward, which depends on the current state, current actions, and the next state \((s,a,s^{\prime})\) at each time slot t. In our proposed multi-USV scenario, the reward consists of four parts: communication distance \(l^{\mathrm{max}}\) limitation penalty, distance penalty, threat area penalty, and collision penalty. We utilized \(R_{l}\) to represent the communication limitation reward, which depended on the next state of the agent. There will be a large minus penalty \(-R_{l}\) for the agent if the agent is out of the communication space in the next state. The space limitation is set as follows  
\[0\leq X_{k,t}\leq l^{\mathrm{max}},\forall k\in K,t\in T \tag{6}\]  
and  
\[0\leq Y_{k,t}\leq l^{\mathrm{max}},\forall k\in K,t\in T \tag{7}\]  
Figure 2: The framework of MADDPG for the cooperative multi-USV network. where \(X_{k,t}\) and \(Y_{k,t}\) represent the horizontal and vertical coordinates of USV, respectively. In the same way, we denote \(R_{k,o}\) as the collision penalty, which can be described as  
\[\left\{