## 5 Experiment and Analysis
### Results Analysis
#### 5.2.2 Formation Maintenance  
Aiming at the formation maintenance problem of the dynamic moving UAV team, we train the path planning algorithm for the leader and design the formation maintenance algorithm for the followers so that the whole formation team can move toward the target with a relatively stable geometric structure. Thus, we evaluate the performance of the proposed scheme according to the following evaluation criteria:  
* Cumulative discounted reward during training
* The final distance between the leader and the team goal in each episode
* The stability difference of the whole team  
We train the decentralized formation maintenance policies based on the reward function in Equations (12) and (13) and the control objective in Equation (4). Figure 7 shows the reward of the USV team with decentralized formation maintenance policies with different numbers of USVs and over 10,000 episodes. Figure 7a-c show the mean episode reward for the team of 3 USVs, 5 USVs, and 7 USVs, respectively. We can see that the policies quickly converge to stable optimized policies.  
In the formation maintenance scenario, the leader of the formation guides the movement of the USV team. Hence, the path planning algorithm of the leader determines the success of the formation task directly. As shown in Figure 8a-c, we test the performance of the formation maintenance policy of the leader in the team of 3 USVs, 5 USVs, and 7 USVs. The mean episode distance between the leader and the team goal all become near zero from about 2000 episodes. That means the leader in these teams can successfully reach the mission target quickly.  
We adopt a stability error function \(SF(U)=\frac{1}{T\times m}{\sum_{j=0}^{T}\sum_{i=1}^{N}}|H^{\prime}_{i}-H_{i}|\) to measure the formation stability when following the leader. Comparison simulations are conducted amongst the proposed decentralized formation maintenance policies, DDPG, and DQN. The stability error of formation with different team sizes is selected for analysis, as shown in Figure 9. We can see that the average stability error in the proposed scheme is lower than that in other schemes. The results show that the followers can track their respective immediate leaders and maintain the predefined distance and direction from their immediate leaders. The proposed scheme can obtain the best performance for maintaining the formation compared with other schemes.  
Figure 6: The total length and the average length of the moving path with the different number of multiple cooperative USVs and different schemes: (**a**) comparison of total moving length of the USV team with different team sizes; (**b**) comparison of the total average length of the USV team with different team sizes.  
Figure 8: _Cont_.  
Figure 7: The reward of multiple cooperative USVs with formation generation policies for 40,000 episodes with different numbers of USVs: (**a**) mean episode reward for 3 USVs during formation generation policies training over 10,000 episodes; (**b**) mean episode reward for 5 USVs during formation generation policies training over 20,000 episodes; (**c**) mean episode reward for 7 USVs during formation generation policies training over 10,000 episodes.