## 5 Simulation and Analysis
### Results Analysis
It is demonstrated from Figure 13c that the efficiency of the algorithm could help the formation configuration reconstruct in a very short time, and this algorithm leads the configuration start to form an adaptable one after reaching the collision conditions. Then, the formation would pass through the obstacles with the new unrigid configuration  
Figure 12: An illustration of the average reward per formation learning at agents \(n=4\). (**a**) obstacles 4 (**b**) obstacles 6 (**c**) obstacles 8. It is noted that higher rewards are obtained by I-MAPPO in red. shape. Figure 13b shows the low speed of online learning of the I-MAPPO algorithm. Figure 13a-c show that although the formation may encounter the obstacle or reach the collision condition, the flight system can still change the configuration to adapt to the current environment. In addition, the I-MAPPO converges faster than the MADDPG. It is shown in Figure 14c that the oscillations increase rapidly along with the growth of the numbers of agents and obstacles. In addition, even though the MADDPG would achieve the high average reward, this algorithm fails to converge in our final step. Instead, the I-MAPPO could lead the formation to the convergence stage. Figure 14b shows a relatively rapid rate of convergence. In the experiment from Figure 15, the experimental results show that the two algorithms are affected by the numbers of agents and the obstacles, and the oscillations stay with the flight all along. Thus, this shows that the dense environment (other agents are also considered as the obstacles) would limit the capability of fully autonomous formation flight. From Figure 15a, the effectiveness of two algorithms almost shares the same result, and the result from Figure 15b oscillates the most in this experimental stage. In addition, the reward value is not strongly relevant to the numbers of the agents and obstacles. The reason for I-MAPPO outweighing MADDPG is that the on-policy execution strategy (Equation (15)) enables all agents to choose more optimal continuous actions from a local perspective and the guidance under the reference baseline (Equation (11)), resulting in greater reward feedback. Figure 14: An illustration of the average reward per formation learning at agents \(n=6\). (**a**) obstacle 4 (**b**) obstacle 6 (**c**) obstacle 8. It is noted that higher rewards are obtained by I-MAPPO in red. Figure 13: An illustration of the average reward per formation learning at agents \(n=5\). (**a**) obstacle 4 (**b**) obstacle 6 (**c**) obstacle 8. It is noted that higher rewards are obtained by I-MAPPO in red. It can be seen from Figure 16 that the average collision numbers show that three algorithms predict the obstacle collision in all learning steps based on the experimental condition above (agents 4-7, obstacles 4, 6, 8). The average trends of each algorithm demonstrate the same collision situation during the learning and training process. In previous work [15], the reason for collision numbers increasing is that the traditional algorithm can perform well in the static environment with the predefined trajectory. In addition, the MADDPG algorithm produces undesirable results due to the offline training based on multiple parameters and the historical experience in memory buffer. The proposed I-MAPPO can achieve better results based on the current adjustable policy. Then, we compared the traditional baseline (TB) and the proposed baseline-SEM during the learning steps for the two MARL algorithms. As shown in Table 3, the I-MAPPO-SEM used the gradient of state error, which can greatly enhance the ability of the adjusting the local trajectory of each agent. The advantage of this technique (Equation (11)) can be concluded as follows. Using gradient information fits the state error to guide the direction rapidly; meanwhile, the current change in the state error can replace the historical one from the traditional baseline. I-MAPPO-SEM showed the better of the three measurements than I-MAPPO-TB.