## 5 Simulation and Analysis
### Maneuvering Strategy Training and Testing  
#### 5.2.1. Comparative Analysis of Training Process  
In this section, a comparative analysis of the training methods is performed. These three training algorithms are the basic MADDPG algorithm (MADDPG), the MADDPG algorithm with only reward shaping (RS-MADDPG), and the MADDPG algorithm with reward shaping and prediction of the target position (PRED-RS-MADDPG).  
When training the air combat countermeasures, the initial states of the two UCAVs are randomly generated according to the four situations described in Section 5.1.3. At the same time, to prevent the two UCAVs from falling into the \"boring\" Nash equilibrium of \"biting the tail\" during training, the performance parameters of the two UCAVs to be different from one another: one UCAV uses the \"advantage\" performance parameters, and the other UCAV uses the \"disadvantage\" performance parameters.  
Figure 7 depicts the curve of the normalized average reward of MADDPG, RS-MADDPG, and PRED-RS-MADDPG about the episodes, which is one complete play of the UCAVs interacting with the air combat environment. The reason for normalization is that, after reward shaping, there is no guarantee that the reward per step is between \([-1,1]\). As the problem is a two-player ZSMG, the average reward of the two UCAVs will be negative to each other. Therefore, in Figure 7, only the average reward of the UCAV with an advantageous performance is plotted.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Parameter** & **Advantage Value** & **Disadvantage Value** \\ \hline Maximum roll rate (deg/s) & 140 & 110 \\ Maximum thrust (\(N\)) & 100,000 & 80,000 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Unmanned combat air vehicles (UCAV) performance parameters setting.  
The naive MADDPG algorithm does not work, and its average reward value is always kept around 0, which is unreasonable, as can be seen from Figure 7. The reason why the algorithm is invalid is that the algorithm uses a sparse reward function. When the reward is sparse, the effective policy learning information is little, which makes the algorithm invalid. As the performance parameters of the UCAVs of both the sides are different, the average reward value gradually converges to a positive value. By comparing the RS-MADDPG algorithm with the PRED-RS-MADDPG algorithm, it can be determined that the training convergence speed of the PRED-RS-MADDPG algorithm is slightly better than that of the RS-MADDPG algorithm. A possible reason is that the \(Q\) value error on both sides of the \(Q\) network output in the early stage of training is large; the \(Q\) value gradient error is also large, which affects the accuracy of the predicted target position and, hence, slows the training speed of the RS-MADDPG algorithm. However, the average reward of the PRED-RS-MADDPG algorithm is higher than that of the RS-MADDPG algorithm. This shows that, by predicting the target position, the UCAV on the advantage side can quickly lock the target and complete the attack necessary to win the air battle. Regarding the essential reason for converging to a better average reward, we believe that the introduction of the target's position prediction provides more target information for our UCAV, which will enable the air combat strategy to achieve a better equilibrium strategy.