# Research on Multi-Robot Formation Control Based on MATD3 Algorithm
## 4 Multi-Robot Formation Strategy Design Based on Improved MATD3
### Reward Function Design  
The reward function of the robot is designed according to the formation control problem of multiple robots. To avoid the problem of low training efficiency due to sparse rewards, we used a combination of guided rewards and sparse rewards. In the process of robot-environment interaction, if reaching the target point, the collision between robots, the collision between robots and obstacles, or reaching the preset spacing between robots occurs, then the sparse reward is used; if none of them occurs, then the guided reward is used.  
Different types of robots have different roles, and for the leader robot, the design of the guided reward function:  
\[R^{leader}_{G}=-\left\|p_{lea}-p_{g}\right\|_{2} \tag{26}\]  
and sparse reward:  
\[R^{leader}_{5}=\left\{\begin{array}{ll}R_{1},\;\text{if}\;\left\|p_{lea}-p_ {g}\right\|_{2}\leq\rho_{g}\\ R_{4},\;\text{if}\;\left\|p_{lea}-p_{oj}\right\|_{2}\leq\rho_{o}\\ R_{5},\;\text{if}\;\left\|p_{lea}-p_{i}\right\|_{2}\leq\rho_{r}\end{array}\right. \tag{27}\]  
where \(i=1,\ldots,N-1\), \(R_{1}\) is the leader robot's reward for reaching the target point, \(R_{4}\) is the penalty for the collision of the leader robot with the obstacle, \(R_{5}\) denotes the penalty for the collision of the leader robot with the other robots, \(\rho_{g}\) denotes the target point radius, \(\rho_{o}\) denotes the obstacle safety distance, and \(\rho_{r}\) denotes the collision safety distance.  
When the leader robot enters the obstacle area, the formation will change to a more passable one. Specifically, the formation strategy will be changed when the leader robot satisfies \(p^{x}_{o,\text{min}}<p^{x}_{lea}<p^{x}_{o,\text{max}}\) and \(p^{y}_{o,\text{min}}<p^{y}_{lea}<p^{y}_{o,\text{max}}\), where \(p^{x}_{o,\text{min}}\), \(p^{x}_{o,\text{max}}\), \(p^{y}_{o,\text{min}}\), \(p^{y}_{o,\text{max}}\) is the range of the obstacle area.  
We set up a hybrid reward mechanism for the following robot. When there is no obstacle area, the robot teams form a V-shaped formation to perform the formation task, and each following robot needs to maintain a preset distance from all its neighboring robots, as shown in Figure 2; when entering the obstacle area, each following robot only needs to maintain a preset distance from a specific neighboring robot in order to form a chain formation structure, as shown in Figure 5. Set the follower guide reward function:  
\[R^{follower}_{G}=\left\{\begin{array}{ll}-\mid&d^{j}_{i}-d^{j^{*}}_{i}\mid \text{, if beyond the obstacle area}\\ -\mid&d^{m}_{n}-d^{m^{*}}_{n}\mid\text{, others}\end{array}\right. \tag{28}\]  
and sparse reward:  
\[R^{follower}_{S}=\left\{\begin{array}{ll}R_{2},\;\text{if beyond the obstacle area and}\;d^{j}_{i}=d^{j^{*}}_{i}\\ R_{3},\;\text{if not beyond the obstacle area and}\;d^{m}_{n}=d^{m^{*}}_{n}\\ R_{4},\;\text{if}\;\left\|p_{i}-p_{oj}\right\|_{2}\leq\rho_{o}\\ R_{5},\;\text{if}\;\left\|p_{i}-p_{j}\right\|_{2}\leq\rho_{r}\end{array}\right. \tag{29}\]where \(d_{i}^{j}\) and \(d_{i}^{j^{*}}\) denote the actual relative distance and the desired relative distance of any two robots in the obstacle-free area, respectively. \(d_{n}^{m}\) and \(d_{n}^{m^{*}}\) denote the actual relative distance and desired relative distance between the left-side \(mth\) following robot and the right-side \(nth\) following robot when in the obstacle area, respectively. \(R_{2}\) and \(R_{3}\) denote the reward for maintaining the formation when the formation is in the non-obstacle and obstacle areas, respectively. \(R_{4}\) and \(R_{5}\) denote the penalty for collision between any robot and any obstacle and the penalty for collision between any two robots, respectively.