## 5 Simulation and Analysis
### Results Analysis
Figure 16: Illustration of the average collision numbers (note that they contain the numbers of those reaching the obstacles’ radius). It is noted that I-MAPPO obtained lower numbers in learning process. Figure 15: An illustration of the average reward per formation learning \(n=7\). (**a**) obstacle 4 (**b**) obstacle 6 (**c**) obstacle 8. It is noted that higher rewards are obtained by I-MAPPO in red. Among all the measurements, the smooth rate \(sr\) is denoted as follows:  
\[sr=\frac{\sum_{t=1}^{T}\sum_{i=2}^{n-1}\lvert\theta_{i+1}-\theta_{i}\rvert}{nt}, \tag{34}\]  
where \(\theta_{i}=\arctan(\lVert p_{i+1,y}-p_{i,y}\rVert/\lVert p_{i+1,x}-p_{i,x} \rVert)\). Thus, the traditional reward function (TRF) and proposed reward function-cost function (CF) are also tested as the comparison. It is noted that the TRF basically uses the constant value in each predefined condition. In this case, based on Equations (28)-(30), we divide the TRF into three parts: (1) the reward of obstacle collision \(r_{obs}\); (2) the reward of neighbors of each agent collision \(r_{ij}\); (3) the reward for tracking the reference trajectory \(r_{d}\). More precisely, these rewards are designed as \(R=\lambda_{1}r_{obs}+\lambda_{2}r_{ij}+\lambda_{3}r_{d}\), shown as follows:  
\[\left\{ \begin{array}{l}\lambda_{1}r_{obs}=-50\\ s.t.\,dist(\hat{X}_{i},X_{obs})=\lVert p_{i}-p_{o}\rVert^{2}\leq 2R_{a}\\ \lambda_{2}r_{ij}=-30\\ s.t.\,dist(\hat{X}_{i},\hat{X}_{j})=\lVert p_{i}-p_{j}\rVert\leq R_{s}\\ \lambda_{3}r_{d}=200\\ s.t.\,dist(\hat{X}_{i},X_{d}^{*})=\lVert p_{i}-p_{d}^{*}\rVert\leq E_{\text{ min}}\end{array} \right., \tag{35}\]  
where \(\lambda_{1},\ \lambda_{2},\ \lambda_{3}\) represents the weight of the three kinds of reward values, respectively. Thus, the comparison test result is shown in Table 4. It can be seen that the I-MAPPO-TRF demonstrated that the obtained result cannot reach the high success rate due to the constant reward value, which limits the flexibility to the current changeable state. The total Success rate, Ave-collisions rate and Ave-smooth rate are compared for the different algorithms during learning steps. As shown in Table 5, the success rate of our algorithm was almost 100% within three algorithms, which made great improvement by 47% and 41% compared with the previous work and MADDPG, respectively. Due to the difficulty of convergence of the previous work, the success rate was the most unstable. Secondly, although MADDPG can solve the problems in the continuous action space, it still has disadvantages of the policy selected from the historical experience buffer without completely considering the current state. Our algorithm combines the prediction module on the basis of I-MAPPO, so that the algorithm can achieve a higher success rate in different environments. In addition, based on the combined action selection mechanism, the proposed flight framework would enhance the angular velocity learning ability to guarantee the flight practicability in unknown environments with smoother trajectory. The configuration changes can basically depend on the environment rather than the rigid \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Algorithm** & **Success Rate (\%)** & \
\begin{tabular}{c} **Ave-Collisions** \\ **Rate (\%)** \\ \end{tabular} &
\begin{tabular}{c} **Ave-Smooth Rate** \\ **(rad)** \\ \end{tabular} \\ \hline I-MAPPO-TRF & 36 \(\pm\) 5.8 & 51.0 & 0.5019 \\ I-MAPPO-CF & 86 \(\pm\) 5.8 & 21.2 & 1.0301 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Algorithm comparison in Success rate, Ave-collisions rate and Ave-smooth rate. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Algorithm** & \begin{tabular}{c} **Success Rate** \\ **(\%)** \\ \end{tabular} & \
\begin{tabular}{c} **Ave-Collisions Rate** \\ **(\%)** \\ \end{tabular} &
\begin{tabular}{c} **Ave-Smooth Rate** \\ **(rad)** \\ \end{tabular} \\ \hline I-MAPPO-TB & 33 \(\pm\) 5.8 & 49.6 & 0.4037 \\ I-MAPPO-SEM & 86 \(\pm\) 5.8 & 21.2 & 1.0301 \\ \hline \hline \end{tabular}
\end{table}