## 3 System Models
### USV Model  
We consider a group of \(N\) USVs for formation control in the leader-follower structure, described as \(\text{U}=\{u_{1},u_{2},\ldots u_{N}\}\) with geometric shapes, such as V-shape, as shown in Figure 1. Because this paper focuses on how to design and train formation generation policies and formation maintenance policies for the USV team, we simplify the USV into a particle, mainly considering the kinematics model \(\dot{p}_{u_{i}}=v_{u_{i}}\) of the particle, and temporarily ignoring the impact of the kinetics model of the particle. Each USV \(u_{i}\) has coordinates \(p_{u_{i}}=(x_{u_{i}},y_{u_{i}})\) and velocity \(v_{u_{i}}=(v_{u_{i}}^{x},v_{u_{i}}^{y})=\dot{p}_{u_{i}}\) in the \(O-XY\) coordinate frame in the two-dimensional maritime plane. At each time step, the action taken by USV \(u_{i}\) is the change of velocity, \(a_{u_{i}}=(\Delta v_{u_{i}}^{x},\Delta v_{u_{i}}^{y})\). In a leader-follower structure, the followers should follow the leader in a geometrically balanced manner and keep a specific direction and distance from the leader. Thus, we establish a new relative coordinate frame \(O-X^{\prime}Y^{\prime}\) that the origin is the original origin, and the direction of the \(Y^{\prime}\)-axis is the heading angle of the leader. Each USV \(u_{i}\) has new coordinates \(p^{\prime}_{u_{i}}=(x^{\prime}u_{i^{\prime}}y^{\prime}_{u_{i}})\) in the relative coordinate frame, as calculated by the coordinate transformation matrix in Equation (1), where \(\theta\) is the leader's heading angle. The formation control of a multi-robot system transforms the formation control problem into the problem of followers tracking the leader's position and  
Figure 1: The standard V-shape formations for multiple unmanned surface vehicles (USVs). Every USV should keep the required distance from its immediate leader in the \(x\)-axis and \(y\) -axis according to the new coordinate frame \(O-X^{\prime}Y^{\prime}\).  
direction. The classical leader-follower mode is that all the followers track a single leader with different distance offsets individually, while a chain leader-follower formation is used in this paper. The chain leader-follower is inspired by three flocking rules of [PERSON] (1964) that agents in a group should stay close to their neighbor agents, avoid collisions with their neighbor agents, and match speed with its neighbor agents instead of the leader of the group. In short, each agent aligns with its neighbors. In a cooperative USV team, it is more conducive to improve the team efficiency to use communication to share information than to collect information of members in the team with sensors. USVs are equipped with industrial control computers, GPS, and other sensors, so the followers obtain the coordinates of their leaders through wireless communication. For instance, the USV \(u_{2}\) tracks the USV \(u_{1}\) while the USV \(u_{4}\) tracks the USV \(u_{2}\) according to the required chain formation. USV \(u_{2}\) is regarded as the immediate leader of its follower USV \(u_{4}\). In addition to the leader USV, \(H_{i}=(d_{i}^{X^{\prime}},d_{i}^{Y^{\prime}})\) is the predefined offset vector, i.e., the relative positional relationship for USV \(u_{i}\) concerning its immediate leader in the \(O-X^{\prime}Y^{\prime}\) coordinate frame.  
Each USV takes action following its formation generation policy \(\mu_{\theta_{i}}\) to make the USV team generate the expected formation shape (see Section 4.2.1). Each USV takes action following its formation maintenance policy \(\mu_{\theta_{i}}^{fm}\) to keep the stability of the formation structure with the predefined teammate spacing (see Section 4.2.2).  
\[\left[\begin{array}{c}x^{\prime}u_{i}\\ y^{\prime}u_{i}\end{array}\right]=\left[\begin{array}{cc}\cos(\theta-\frac{ \pi}{2})&\sin(\theta-\frac{\pi}{2})\\ -\sin(\theta-\frac{\pi}{2})&\cos(\theta-\frac{\pi}{2})\end{array}\right] \left[\begin{array}{c}x_{u_{i}}\\ y_{u_{i}}\end{array}\right] \tag{1}\]