# Mdpj _sciences_
## 4 Experiments and Results
### Ablation Studies  
In this section, five methods are used for experiments: FRE, FRE-U, FRE-S, GAE and DSR. FRE stands for using FRE in the objective function and collecting samples according to FRE, FRE-U stands for using FRE in the objective function and collecting samples randomly, FRE-S stands for using FRE in the objective function without intermediate rewards and collecting samples according to FRE, GAE stands for using GAE in the objective function, DSR stands for using DSR in the objective function. In each iteration, the evaluator records the evaluation results of the method. Figure 3 shows the change in the average return of FRE, FRE-U and FRE-S during the agent training process. The solid line represents the mean of average returns of the five experiments, and one point is drawn on the solid line every five gradient steps. The shaded part represents the standard deviation of average returns. As can be seen from Figure 3, since agents have not yet acquired any effective maneuver policy, average returns in the early training are negative. After several times of training, the average returns of FRE and FRE-U have an obvious upward trend, while the average return of FRE-S is always around 0. The blue line represents the average return of FRE, the orange line represents the average return of FRE-U and the black line represents the average return of FRE-S. Two points about the change of average return need to be declared: 1. In air combat maneuver decision-making, the return value obtained by the agent may be related to whether it can win air combat without causality, that is, the maneuver decision-making ability of air \begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameter** & **Value** \\ \hline azimuth angle & (\(-\pi/4\), \(\pi/4\)) \\ distance & (40,000 m, 100,000 m) \\ velocity & (250 m/s, 400 m/s) \\ batch size & 1024 \\ optimizer & Adam \\ actor learning rate & 0.0002 \\ critic learning rate & 0.001 \\ actor architecture & (256, 256, 4) \\ critic architecture & (256, 256, 1) \\ activate function & tanh \\ epoch & 8 \\ \(\gamma\) & 0.995 \\ \(\lambda\) & 0.98 \\ \(\epsilon\) & 0.2 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Hyperparameters. Figure 3: Average Return. combat agent cannot be compared only according to returns it obtained. A high return value does not mean more probability of a win, and a low return value does not mean less probability of a win. The increase in the return value only means that the agent gradually learns the policy to increase the reward. 2. FRE-S only includes the final reward, while FRE and FRE-U include not only the final reward but also the intermediate reward. Therefore, the possible return value of FRE and FRE-U is more than that of FRE-S. According to the above two points, to accurately evaluate the performance of different methods, we need to use the results of air combat confrontation. Therefore, the results of different methods are compared in the evaluator, where each evaluation contains 108 rounds. The variations of the number of wins, losses, and draws in each method with the number of iterations are shown in Figure 4. The solid line represents the mean of the number of wins, loses or draws of this iteration over the five experiments, and the shaded part represents the standard deviation of the number of wins, loses or draws. Obviously, the more wins, the stronger the decision-making ability of the agent. It should be pointed out that more losses also mean more effective agent training because the agent is playing against its own past versions. Therefore, more losses represent the stronger the decision-making ability of the past versions of the agent, which can indicate that the training of the agent is effective. On the contrary, a large number of draws means that the training of the agent may be invalid. At the beginning of training, there are usually a large number of draws because, at this time, the agent often maneuvers randomly, so its missile is likely to miss the target, which results in a large number of draws.