# Mdpj _sciences_
## 3 Method
### FRE-PPO Algorithm  
TRPO repeatedly performs the following steps:  
* Use the single path or vine procedures to collect a set of state-action pairs along with Monte Carlo estimates of their Q-values;
* By averaging over samples, construct the estimated objective and constraint in \[ \begin{array}{l}\underset{\theta}{\text{maximize}}\;\text{E}_{s\sim\rho_{ \theta_{old}}}\;\mu\sim q\left[\frac{\pi_{\theta}(a|s)}{q(a|s)}Q_{\theta_{old}} (s,a)\right]\\ \text{subject to}\;\text{E}_{s\sim\rho_{\theta_{old}}}\left[D_{KL}(\pi_{\theta _{old}}(\cdot|s)\big{|}\big{|}\tau_{\theta}(\cdot|s))\right]\leq\delta.\end{array} \] (12)
* Approximately solve this constrained optimization problem to update the policy's parameter vector \(\theta\) by means of the conjugate gradient algorithm followed by a line search, which is altogether only slightly more expensive than computing the gradient itself. Where, \(\theta_{old}\) is the parameter of the current strategy, \(\rho\) is the probability distribution of state transition, \(q\) is the sampling distribution, and \(D_{KL}\) is the \(KL\) divergence. By establishing the Fisher information matrix, TRPO avoids storing a dense Hessian matrix or policy gradients from a batch of trajectories and improves computational efficiency. Theoretically, the penalty coefficient of larger \(KL\) divergence can lead to a small step size, so the penalty coefficient should be decreased. However, it is difficult to select a penalty coefficient that can ensure the stability of the effect, so TRPO uses a constraint term instead of a penalty term. At the same time, since the maximum \(KL\) divergence is not easy to be optimized and estimated, TRPO uses the average \(KL\) divergence. PPO is an improved version of TRPO. TRPO may cause excessive policy updates when maximizing the objective function, so PPO modifies the objective function to punish the policy changes that make the probability ratio far from 1. The objective function of PPO is:  
\[ \begin{split} L^{CLIP}(\theta)=&\mathrm{E}[\min(r_{t }(\theta)A_{t},\mathrm{clip}(r_{t}(\theta),1-\varepsilon,1+\varepsilon)A_{t})] \\ P_{r}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta, \mathrm{clip}}(a_{t}|s_{t})}.\end{split}  \tag{13}\]  
by clipping the probability ratio \(P_{r}\) and removing the strategy change that makes the probability ratio exceed the interval [1 - \(\varepsilon\), 1 + \(\varepsilon\)]. The formula of the DSR advantage function used in PPO is: \(A_{t}\)= \(\mathrm{r}_{t}+\gamma\mathrm{r}_{t+1}+\cdots\gamma^{T-t+1}\mathrm{r}_{T-1}+ \gamma^{T-t}V(s_{T})-V(s_{t})\). The formula of generalized advantage estimation (GAE) [48] used in PPO is: \(A_{t}=\delta_{t}+(\gamma\lambda)\delta_{t+1}+\cdots+(\gamma\lambda)^{T-t+1} \delta_{T-1}\), where \(\delta_{t}\)= \(\mathrm{r}_{t}+\gamma V(s_{t+1})-V(s_{t})\). Since GAE and DSR are not effective for solving the air combat maneuver decision-making problem, this paper proposes a new function, namely, final reward estimation (FRE). The formula of FRE is defined as follows:  
\[A_{t}=\left\{ \begin{array}{l}\mathrm{r}_{t}+\gamma^{T-t}\mathrm{r}_{T},t<T \\ \mathrm{r}_{T},t=T\end{array} \right. \tag{14}\]  
where \(T\) is the last time step in air combat. This method uses self-play samples obtained from the air combat environment to train the maneuver policy, which is represented by neural networks. The input of the neural networks is air combat situation information, and the output is the three control quantities of aircraft and whether to launch a missile. At the same time, in order to increase the exploration of the policy, Gaussian noise is added to each action. After obtaining training samples, a certain number of samples need to be selected from these samples to train the neural network. A common sampling method is randomly uniform sampling. In order to improve the efficiency and effect of training, the method of sampling according to FRE is proposed in order to collect training samples, that is, the higher FRE of the sample, the greater the probability of being collected.