## 4 Proposed Scheme
### Formation Control Algorithm Based on Reinforcement Learning
#### 4.2.1 Formation Generation Policy
During training, each USV in the team has independent networks with different parameters and independent optimization for its formation control policy. The advantage function describes the advantage of selecting an action \(a\) in the state \(s\), compared with other actions under the state \(s\), as denoted by  
\[A(s_{i},a_{i})=Q(s_{i},a_{i})-V(s_{i}) \tag{6}\]  
For the critic part, the loss function is denoted as follows:  
\[L\Big{(}\theta_{i}^{V}\Big{)}=\Big{(}R_{i}-V_{i}\Big{(}s_{i}|\theta_{i}^{V} \Big{)}\Big{)}^{2} \tag{7}\]  
where  
\[R_{i}=r_{0}+\gamma r_{1}+\gamma^{2}r_{2}+\ldots+\gamma^{n-1}r_{n-1}+\gamma^{n }V_{i}^{\prime}\Big{(}s_{n}|\theta_{i}^{V^{\prime}}\Big{)} \tag{8}\]  
The critic is updated by minimizing the loss,  
\[\
abla_{\theta_{i}^{V}}L\Big{(}\theta_{i}^{V}\Big{)}=-\mathrm{E}_{s,\mu,r,s^ {\prime}}\Big{[}\Big{(}R_{i}-V_{i}\Big{(}s_{i}|\theta_{i}^{V}\Big{)}\Big{)} \
abla_{\theta_{i}^{V}}V_{i}\Big{(}s_{i}|\theta_{i}^{V}\Big{)}\Big{]} \tag{9}\]  
The actor is updated by the gradient of Q-value and advantage function:  
\[\
abla_{\theta_{i}^{\mu}}L\Big{(}\theta_{i}^{\mu}\Big{)}=\mathrm{E}_{s}\Big{[} \
abla_{a_{i}}(R_{i}-V(s_{i}))|_{a_{i}=\mu_{i}(s_{i})}\
abla_{\theta_{i}^{ \mu}}\mu_{i}\Big{(}s_{i}|\theta_{i}^{\mu}\Big{)}\Big{]} \tag{10}\]  
The interactions between the USV team and the ocean environment can be divided into separate episodes. An episode starts in a random state of the team and ends at a terminal state or after a specified number of time steps. The number of episodes is set to 20,000, and the maximum episode length is 100--that is, each episode has up to 100 time-steps. At each time step in every episode, every USV interacts with the environment, selects its action \(a_{t}\) according to the current state \(s_{t}\), and dynamically calculates the reward \(r_{t+1}\) generated by the environment in real time according to the reward function \(r_{1}\) and \(r_{2}\). The reward function \(r_{1}\) is defined to be the minimum distance between the USVs and the  
Figure 4: The network structure of each USV’s policy. geometric points of formation. If a collision occurs between members during the formation control process, a collision penalty is given--that is, the negative reward value \(r_{2}\). \[r_{1}=-\sum_{j=1}^{N}\left(\underset{u_{i}}{\min}Dis(p^{\prime}{}_{u_{i}}-p^{ \prime}{}_{\eta_{j}})\right) \tag{11}\]  
Figure 3:0  
where \(p^{\prime}{}_{\eta_{j}}\) is the position of the formation node \(\eta_{j}\) in the coordinate frame \(O-X^{\prime}Y^{\prime}\), which can be calculated by the predefined location of the leader and the parameters in the formation shape matrix \(F_{s}\). \(Dis\left(p^{\prime}{}_{u_{i}};p^{\prime}{}_{\eta_{j}}\right)\) is the Euclidean distance between the USV \(u_{i}\) and the formation node \(\eta_{j}\). \(c_{1}\) is a positive constant. ```
1:Input Reward Function \(r_{1}\),\(r_{2}\) for formation generation scenario
2:Input The predefined formation shape \(F_{s}\)
3:Output formation generation policies \(\mu=\left\{\mu_{\theta_{1}},\mu_{\theta_{2}}\ldots\mu_{\theta_{N}}\right\}\) for the USV team Initial experience replay buffer \(D_{s}\)
4:for episode \(=1:M\)do
5: Initialize a random process \(N_{rp}\) for action exploration
6: Receive environment state
7:for\(t=1:T\)do
8:for\(i=1:N\)do
9: Select action \(a_{t}=\mu_{i}\left(s_{t}|\theta_{t}^{t}\right)+N_{rp}\)
10: Execution actions \(a_{t}\) and observe reward \(r_{t}\) and new state \(s_{t+1}\)
11: Add \(e_{t}^{i}=\left(s_{t}^{i},a_{t}^{i},r_{t}^{i},s_{t+1}^{i}\right)\) into replay buffer \(D_{s}\)
12: Sample a random minibatch of transitions \(e_{j}\) from \(D_{s}\)
13: Calculate reward in real-time \(R_{i}=\left\{ \begin{aligned} 0&\quad\text{for terminal }s_{t}\\ V(s_{t})&\quad\text{for non }-\text{terminal }s_{t}\end{aligned}