# Research on Multi-Robot Formation Control Based on MATD3 Algorithm
## 5 Simulation Experiments
### Experimental Results and Analysis  
The goal of reinforcement learning is to maximize the cumulative discounted reward, so the cumulative episode average reward is commonly used to measure the goodness of the training results, and we used the average reward of the last 1000 episodes as the final episode average reward value. Training was performed using the three algorithms described above, respectively. The episode average rewards are shown in Figure 6. It can be seen that the three algorithms can learn better policies around 20,000 episodes and gradually reach convergence. In comparison, the improved MATD3 algorithm had the highest average reward after convergence and had better stability compared to the other two algorithms.  
Figure 5: Changing formation strategy.  
On the other hand, to measure the ability of the robot formations to avoid collisions, we set two metrics, including the collision rate between robots (CBRS) and the collision rate between robots and obstacles (CBRO). They were obtained based on whether the distance between the robots was less than \(\rho_{r}\) and whether the distance between the robots and obstacles was less than \(\rho_{o}\), as shown in Equation (30).  
\[\left\{\begin{array}{l}CBRS=\frac{\text{Number of collisions between robots}}{\text{Number of time steps per episode}}\\ CBRO=\frac{\text{Number of collisions between the robot and the obstacle}}{\text{Number of time steps per episode}}\end{array}\right. \tag{30}\]  
The experimental results are shown in Figure 7. Obviously, the improved MATD3 algorithm had better performance in formation collision avoidance.  
The path-planning ability of the leader robot directly determined the success of the formation navigation task, so we used the final distance between the leader robot and the target point (UDBLT) as one of the algorithm metrics. The experimental results are shown in Figure 8. All three algorithms can converge to near 0, indicating that the leader robot can successfully plan the path to the target point.  
Figure 6: Episode average rewards.  
Figure 7: Collision rate during robot formation movement: (**a**) collision rate between robots (CBRS); (**b**) collision rate between robots and obstacles (CBRO).  
Formation stability error (RFSE) is an important indicator of robot formation navigation, which we define according to the following equation.  
\[e(\mathbf{r})=\frac{1}{T}\sum_{t=0}^{T}\sum_{i,j}^{N}\Bigl{|}d_{i}^{j}-d_{i}^{j^{*}} \Bigr{|},i\
eq j \tag{31}\]  
where \(d_{i}^{j}\) and \(d_{i}^{j^{*}}\) denote the actual relative distance and the desired relative distance between robot \(\mathbf{r}_{i}\) and robot \(\mathbf{r}_{j}\), respectively. The experimental results are shown in Figure 9. It can be seen that the improved MATD3 algorithm had a lower formation stability error compared to the other algorithms, indicating that the multiple robots can maintain the preset formation well.  
Table 1 shows the comparative data of the above metrics in more detail. According to Table 1, the improved MATD3 algorithm performed better in terms of collision avoidance and formation stability errors in the obstacle environment.  
The trained multi-robot formation path is shown in Figure 10, where the robot team formed a preset formation at Step 10 and was moving toward the target area. At Step 35, the robot formation completed the formation change and entered the obstacle area. At Step 95, the robot formation passed the obstacle area, resumed the preset formation, and continued to move towards the target area. At the end, the robot formation successfully navigated to the target area. As can be seen, due to the hybrid reward mechanism, the robot formation can reasonably change its formation when facing the obstacle area and, thus, achieve obstacle avoidance more easily.  
Figure 8: Final distance of the leader robot from the target point (UDBLT).  
Figure 9: Formation stability error (RFSE).