## 5 Simulation and Analysis
### Results Analysis  
In this section, we proceeded to conduct numerical simulations with the setups to illustrate the theoretical results and validate the effectiveness of the novel configuration change control algorithm. In addition, we compared I-MAPPO with MADDPG algorithms on the multi-agent environment. Based on the online learning policy, the formation is steered by the control protocol and MARL to pass through the two-dimensional environment with obstacles when adjusting the formation configuration. Firstly, we evaluated the performance of the DMPC-based method in reference generation using two metrics, the average tracking error and the average acceleration error of six agents, shown in Figure 11. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Parameter** & **Symbol** & **Value** \\ \hline Circular buffer & \(D\) & 50,000 \\ Max step & \(T\) & 4500 \\ Learning rate & \(l_{\text{a}}\) & 0.006 \\ Discount rate & \(\gamma\) & 0.87 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The parameter settings of MARL. Figure 11: Illustration for the average tracking and acceleration error for the formation trajectory navigation. (**a**) average tracking error (**b**) average acceleration. The numbers of 1–7 represent each agent. Figure 10: Illustration of the proposed algorithm framework. It can be seen from Figure 11 that the DMPC method effectively tracks the affine trajectory planning based on the local environment. The algorithm tracks each agent's predefined trajectory, transforming disordered sequences of formation into an ordered one while minimizing error during configuration changes. Figure 11a illustrates that a closer alignment with the predefined trajectory results in smaller errors. The DMPC algorithm practically integrates local environment data to provide a more accurate reference, which would replace the traditional baseline in MARL process. Thus, the I-MAPPO algorithm is tested in the following along with the compared MADDPG. Then, we evaluate the performance of the proposed algorithms using two key measurements--the average reward and the average number of collisions (note that it contains the numbers reaching the obstacles' radius)--during the learning process under the proposed reference baseline using Equation (11). The two distinct DRL strategies, MADDPG and I-MAPPO, which can both reduce the variance of the policy gradient and enhance continuous action selection. Thus, the experiment analyzes the algorithm's performance with varying numbers of agents and obstacles. The average reward is calculated as the mean reward of all agents from 4 to 7 and obstacles 4, 6 and 8. In all the experiments below, the various numbers of agents and obstacles are designed to analyze the proposed algorithm in different topology (4-7) based on Figure 7. All the experimental values are directly obtained from TensorBoard. Thus, it is divided into four parts: firstly, using these two algorithms to test the small formation with agents \(N=4\), obstacles 4, 6 and 8; then, these two algorithms are designed to test in the normal size formation with \(N=4\), \(N=5\) agents, obstacles 4, 6 and 8, respectively; finally, the experiment tests the formation with \(N=7\) agents, obstacles 4, 6 and 8. These two MARL methods share the similar policy-based (MADDPG-off policy/MAPPO-on policy) learning structure. In small formation, compared to MADDPG and I-MAPPO algorithm, both MADDPG and I-MAPPO achieve higher rewards at the final stage of convergence, indicating that the addition of the reference baseline enhances reward acquisition. In addition, we can know from Figure 12a-c, the I-MAPPO can achieve the convergence faster than MADDPG. Moreover, the I-MAPPO has a better ability of trajectory replanning and configuration reconstruction from oscillations above, especially from Figure 12c, and the formation could replan the local trajectory of each agent to form the new unrigid configuration after encountering the collision at the descent point (Learning step \(T=800,000\)).