# Mdpj _sciences_
## 3 Method
### Reward Design  
In RL, the agent maximizes its return through trial and error experience [47]. Due to sparse rewards, it is difficult for agents to obtain rewards in an air combat environment and complete the given task. Therefore, in order to avoid invalid training caused by sparse rewards, this paper proposes intermediate rewards and final rewards. The intermediate reward is determined by the situation information of air combat, and the agent can get an intermediate reward at each time step, so the reward is not sparse, and the agent may get the final reward according to the assistance of the intermediate reward. The final reward is determined by the result of air combat, namely win, lose or draw. As the agent can only get the final reward at the end of air combat, the final reward is sparse. The combination of intermediate reward and final reward can effectively avoid bad training performance when only using the final reward. The formula of intermediate reward \(r_{M}\) and the explanation of the meaning behind the formula is shown as follows:  
\[ \begin{array}{l}r_{a}=\omega_{a1}(\dot{\psi}_{\max 1}-\dot{\psi}_{r1})+\omega_{a2}(\gamma_{\max 1}-\gamma_{r1})+\omega_{a3}(\dot{\psi}_{2}-\dot{\psi}_{r1})+\omega_{a4}( \gamma_{r2}-\gamma_{r1})+\min(\beta,1)\\ r_{b}=\omega_{b}\frac{|z_{b}-\mathrm{H}|}{\mathrm{H}}\\ r_{c}=\omega_{c}z_{\min\min\{}}\\ \end{array}  \tag{10}\] \[ \begin{array}{l}r_{d}=\left\{\begin{array}{l}1,\text{ launched}\\ 0,\text{ not launched}\\ -1,\text{ miss the target}\\ r_{M}=r_{a}+r_{b}+r_{c}+r_{d}\\ \end{array}\right.\end{array} \]  
where, \(\gamma_{\max1}\) and \(\psi_{\max1}\) are the maximum pitch and yaw angle of the missile, respectively, \(\gamma_{r1}\) and \(\psi_{r1}\) are, respectively, the pitch and yaw angles of the speed vector of one aircraft relative to the line of sight, \(\gamma_{r2}\) and \(\psi_{r2}\) are, respectively, the pitch and yaw angles of the speed vector of another aircraft relative to the line of sight. \(\omega_{a1}\), \(\omega_{a1}\), \(\omega_{a3}\), and \(\omega_{a4}\) are corresponding weights. As can be seen in (10), smaller \(\gamma_{r1}\) and \(\psi_{r1}\) mean larger \(r_{a}\), which motivates the aircraft to fly to the target (it can be regarded as an attack); larger \(\gamma_{r1}\) and \(\psi_{r1}\) mean smaller \(r_{a}\), which indicates that the target is approaching the aircraft (it can be regarded as being attacked). D is the typical attack distance of the missile, and \(d\) is the distance between both sides, so the item min(D/\(d\), 1) motivates the aircraft to approach the target. Therefore, \(r_{a}\) is set as its opposite during the final guidance stage to make the aircraft fly away from the target since, during the final guidance stage, the missile no longer needs the information transited by the aircraft. \(z_{I}\) is the altitude of the aircraft, H is the typical flight altitude, and \(\omega_{b}\) is the weight. Therefore, \(r_{b}\) encourages the aircraft to stay at its typical flight altitude. \(t_{missile}\) is the flight time after the missile is launched and \(\omega_{c}\) is the weight. This means that earlier launching corresponds to a larger reward. When the missile is launched, the agent gets a reward of 1. When the missile is not launched, the agent gets a reward of 0. When the missile loses its target, the agent gets a reward of \(-1\). This item means launching the missile is encouraged, and missing the target is punished. The sum of the above four rewards is the intermediate reward \(r_{M}\). The formula of the final reward \(r_{F}\) is:  
\[r_{F}=\left\{ \begin{array}{l}30,\;\text{win}\\ -30,\;\text{lose}\\ 0,\;\text{draw}\end{array} \right.