# Mdpj _sciences_
## 2 Background  
The advantage of the maneuver decision-making method based on RL (e.g., PPO) is that it can directly use the continuous action space without discretization of the action space. At the same time, it does not rely on expert experience and can adapt to changes in the air combat environment. Meanwhile, RL can create agents outperforming human beings [14; 15; 16]. Therefore, RL has some advantages and potential in air combat maneuver decision-making. However, there are still some problems when using RL in air combat maneuver decision-making: it is difficult to train effective agents using RL; RL method usually performs well in Atari [27] and robot control tasks [28], but air combat maneuver decision-making is different from these tasks. Therefore, how to guarantee better and more stable performance in air combat maneuver decision-making using RL needs to be addressed. The purpose of RL is to learn policies to maximize rewards in sequential decision-making [29]. Sequential decision-making is that, at each discrete time step t, an agent starts from a certain state, \(s_{t}\), chooses an action, \(a_{t}\), according to its policy and obtains a reward, \(r_{t}\). Then, it takes the discounted sum of rewards (DSR) as its return with a discount factor. The aim of the agent acting in a stochastic environment by sequentially choosing actions over a sequence of time steps is to obtain more rewards or win the game. Air combat maneuver decision-making can be regarded as a sequential decision-making problem in continuous action space. Therefore, the maneuver decision-making problem is modeled as the Markov decision process (MDP), a sequential decision-making process, and the agent interacts with the air combat environment by its policies and then obtains a tuple (\(s_{t},a_{t},r_{t}\)). RL can be divided into value-function-based RL (VBRL) and policy-based RL (PBRL). VBRL uses the temporal difference (TD) error to optimize the value function of the action and selects the action corresponding to the value function. Its disadvantage is that if the action space is continuous, the action space must be discretized at first, which results in the exponential growth of actions. Value-based algorithms include Q-learning [11], deep Q-network (DQN) [10] and its variants: (1) prioritized experience replay [30] weights sampling of data based on TD error to improve learning efficiency; (2) dueling DQN [31] improves the network structure by decomposing the action value function Q into the state value function and the advantage function to improve the function approximation ability; (3) double DQN [32] uses two independent action value functions to solve the problem of over estimation; (4) noisy DQN [33] adds noise to network parameters to increase the exploration ability of agents; (5) distributed DQN [34] refines the state action value estimation into the estimation of the state action value distribution; (6) rainbow [13] combines the characteristics of the above DQN based methods; (7) R2D2 [35] studies the effect of adding recurrent neural networks to RL; (8) NGU [36] uses short-term novel incentives in an episode and long-term novel incentives across multiple episodes to increase the exploration ability of agents; (9) agent57 [37], based on NGU, solves the problem of long-term credit allocation by improving the training stability and dynamically adjusting discount factors. PBRL methods directly parameterize policies to maximize DSR by updating the policy using a policy gradient. Its advantage is its fast convergence speed and can be directly used in the continuous and high-dimensional action space without discretization. Policy-based algorithms include policy gradient algorithm [38], natural policy gradient algorithm [39], deterministic policy gradient algorithm [40], deep deterministic policy gradient algorithm [41], dual delay deep deterministic policy gradient algorithm [42], trust region policy optimization (TRPO) [43] and proximal policy optimization (PPO) [44]. TRPO and PPO limit the update of the policy.