## 5 Experiment and Analysis
### Results Analysis  
#### 5.2.1 Formation Generation  
We compare the performance of the proposed scheme with the following other schemes through simulation results and analysis of the results:  
* The deep deterministic policy gradient (DDPG) scheme: In this scheme, USV learns formation generation policy based on the deep deterministic policy gradient.
* The deep Q-learning (DQN) scheme: In this scheme, USV learns formation generation policy based on deep Q-learning.  
We train the decentralized formation maintenance policies based on the control objective in Equation (3). We evaluate the proposed scheme by averaging the episode reward for every 100 episodes. Figure 5 shows the mean episode reward of the USV team with different configurations of formations and different team sizes over 20,000 episodes when using the proposed formation generation policies. Figure 5a-c show the performance of the team with 3 USVs, 5 USVs, and 7 USVs, respectively. The proposed scheme can learn effective formation generation policies for different USV team sizes. As shown in Figure 5, with the policy training going on, the formation generation policies based on deep reinforcement learning is continuously optimized, and the cumulative discounted return increases until the policy converges. Figure 5 shows that the proposed formation control scheme can perform formation generation with different team sizes effectively.  
Next, we study the performance comparison of the proposed scheme and other existing schemes. Figure 6 shows the total length \(len_{fg}=\sum_{i=1}^{N}(s(u_{i})-e(u_{i}))\) and average length \(Mlen_{fg}=\frac{1}{N}\sum_{i=1}^{N}(s(u_{i})-e(u_{i}))\) of the moving path of the team with different team sizes. It illustrates that in the case of changing team size, the proposed scheme can still form an expected formation shape with the shortest path length. Moreover, the proposed scheme can obtain better performance than other schemes. Figure 6a shows that the total length of the USV team's moving path increases as the size of the team increases, and the proposed scheme has the shortest total length during the formation generation. Figure 6b shows that the proposed scheme has the shortest average length of the moving path during the formation generation. The reasons for these are as follows: first, the proposed scheme is used directly to parameterize the whole policy and find the optimal policy, which can get rid of the limitation of discrete action space. Second, the designed reward function is designed to drive all USVs to reach nodes in the predefined formation as soon as possible to maximize the cumulative discounted return of the team. Third, the advantage accelerates the learning of policies. As for the DDPG and DQN, the target locations in the generated formation for each USV maybe not the optimal, and the average moving path is not the shortest. Consequently, the proposed scheme is better compared with other schemes, as shown in Figure 6.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameter** & **Value** \\ \hline Minibatch size & 1024 \\ Replay buffer size & 10\({}^{6}\) \\ Discount factor & 0.95 \\ Learning rate & 0.01 \\ Maximum episode length & 100 \\ Number of episodes & 20,000 \\ Number of units in the multilayer perceptron (MLP) & 64 \\ \hline \hline \end{tabular}
\end{table}
Table 1: The summary of the main hyperparameters and their values.  
Figure 5: The reward of multiple cooperative USVs with formation generation policies for 40,000 episodes with different numbers of USVs: (**a**) mean episode reward for the team of 3 USVs during formation generation policies training over 20,000 episodes; (**b**) mean episode reward for the team of 5 USVs during formation generation policies training over 20,000 episodes; (**c**) mean episode reward for the team of 7 USVs during formation generation policies training over 20,000 episodes.