## 2 Method  
### Aircraft Model and Missile Model  
The aircraft model is listed as follows [26]:  
\[\left\{\begin{array}{l}\dot{x}=v\cos\gamma\cos\psi\\ \dot{y}=v\cos\gamma\sin\psi\\ \dot{z}=v\sin\gamma\\ \dot{v}=g(n_{x}-\sin\gamma)\\ \dot{\gamma}=\frac{\delta}{\sigma}(n_{z}\cos\mu-\cos\gamma)\\ \dot{\psi}=\frac{\delta}{\cos\gamma}n_{z}\sin\mu\\ n_{x}\in[-0.5,1.5]\\ n_{z}\in[-3,9]\\ \mu\in[-\pi,\pi]\\ \end{array}\right. \tag{1}\]  
where \(x\), \(y\), and \(z\) are three-dimensional coordinates of the aircraft. \(\gamma\) and \(\psi\) are pitch angle and yaw angle, respectively. \(v\) represents the aircraft speed. g is the gravitational acceleration. \(\mu\), \(n_{x}\), and \(n_{z}\) are control signals. The missile model is [27]:  
\[\left\{\begin{array}{l}\dot{x}_{m}=v_{m}\cos\gamma_{m}\cos\psi_{m}\\ \dot{y}_{m}=v_{m}\cos\gamma_{m}\sin\psi_{m}\\ \dot{z}_{m}=v_{m}\sin\gamma_{m}\end{array}\right. \tag{2}\]  
\[\left\{\begin{array}{l}\dot{v}_{m}=\frac{(P_{m}-Q_{m})g}{C_{m}}-g\sin\gamma _{m}\\ \dot{\psi}_{m}=\frac{n_{m}\cos\gamma_{m}}{\psi_{m}}-\frac{g\cos\gamma_{m}}{ \psi_{m}}\\ \dot{\gamma}_{m}=\frac{n_{m}\sin\gamma_{m}}{\psi_{m}}-\frac{g\cos\gamma_{m}}{ \psi_{m}}\end{array}\right. \tag{3}\]  
where \(x_{m}\), \(y_{m}\), and \(z_{m}\) are three-dimensional coordinates of the missile. \(v_{m}\) represents the missile speed. \(\gamma_{m}\) and \(\psi_{m}\) are pitch angle and yaw angle, respectively. \(n_{mc}\) and \(n_{m}\) are control signals. \(P_{m}\), \(Q_{m}\), and \(G_{m}\) are thrust, resistance, and mass, respectively:  
\[P_{m}=\left\{\begin{array}{cc}P_{0}&t\leq t_{w}\\ 0&t>t_{w}\end{array}\right. \tag{4}\]  
\[Q_{m}=\frac{1}{2}\rho v_{m}^{2}S_{m}C_{Dm} \tag{5}\]  
\[G_{m}=\left\{\begin{array}{ll}G_{0}-G_{t}t&t\leq t_{w}\\ G_{0}-G_{t}t_{w}&t>t_{w}\end{array}\right. \tag{6}\]  
where \(t_{w}\) = 12.0 s, \(\rho\) = 0.607, \(S_{m}\) = 0.0324, and \(C_{Dm}\) = 0.9. \(P_{0}\) is the average thrust, \(C_{0}\) is the initial mass, and \(G_{t}\) is the rate of flow of fuel. \(K\) is the guidance coefficient of proportional guidance law.  
\[\left\{\begin{array}{l}n_{mc}=K\cdot\frac{P_{m}\cos\gamma_{m}}{g}[\dot{\beta }+\tan\varepsilon\tan(\varepsilon+\beta)\dot{\varepsilon}]\\ n_{mh}=\frac{\gamma_{m}}{g}\frac{\gamma_{m}}{\cos(\varepsilon+\beta)}\dot{ \varepsilon}\end{array}\right. \tag{7}\]  
\[\left\{\begin{array}{l}\beta=\arctan(r_{y}/r_{x})\\ \varepsilon=\arctan(r_{z}/\sqrt{r_{x}^{2}+r_{y}^{2}})\end{array}\right. \tag{8}\]\[\left\{\begin{array}{l}\dot{\beta}=(\dot{r}_{y}r_{x}-r_{y}\dot{r}_{x})/(r_{x}^{2 }+r_{y}^{2})\\ \dot{\varepsilon}=\frac{(\frac{r_{x}^{2}}{r_{y}^{2}})r_{z}-r_{z}(r_{x}r_{x}+ \dot{r}_{y}r_{y})}{R^{2}\sqrt{r_{x}^{2}+r_{y}^{2}}}\end{array}\right. \tag{9}\]  
where \(n_{mc}\) and \(n_{mh}\) are control commands of the missile. \(\beta\) and \(\varepsilon\) are yaw angle and pitch angle of line-of-sight, respectively. The line-of-sight vector is the distance vector \(r\), where \(r_{x}=x_{t}-x_{m}\), \(r_{y}=y_{t}-y_{m}\), \(r_{z}=z_{t}-z_{m}\), and \(R=\|r\|=\sqrt{r_{x}^{2}+r_{y}^{2}+r_{z}^{2}}\). If the missile has not hit the target after 27 s, the target is regarded as missed. If the azimuth angle of the target relative to the aircraft exceeds \(60^{\circ}\) (off-axis angle) when at the launch time, the target is regarded as missed. The conditions for the end of simulations are: 1. One of both sides in air combat is hit by the missile (the missing distance is 30 m); 2. The missiles of the both sides miss their targets; 3. The simulation time has reached 100 s. Meanwhile, we do not use any handcrafted reward functions; namely, the reward obtained by the agent is 1 if it defeats the opponent, \(-1\) if it is defeated by the opponent, and 0 in other cases.