## 1 Introduction  
Autonomous maneuver decision-making is a sequential decision-making problem. In air combat, the goal of agents is to maneuver according to different states and launch missiles to defeat opponents. Usually, decisions are made by a human pilot. However, with the development of artificial intelligence, there have been many studies regarding programs or algorithms [1, 2] as virtual pilots to make decisions in simulation environments, which aims to realize autonomous decision-making for unmanned combat aerial vehicles in the future. For example, [PERSON] [3] proposed an improved deep Q-network [4] for air combat maneuver decision-making, verifying the feasibility of deep reinforcement learning (RL) for air combat maneuver decision-making. [PERSON] et al. [5] applied game theory to the confrontation process of air combat and proposed a differential game method combined with the missile attack area [6] in order to attack the static high-value targets. [PERSON] et al. [7] compared different methods of evaluating the most effective moment for launching missiles during air combat. They found that supervised learning on the basis of simulated data can promote the flight quality in beyond-visual-range air combat and increase the likelihood of hitting the desired targets. [PERSON] et al. [8] used asynchronous advantage actor critic algorithm [9] to address the problem of air combat maneuver decision-making. They proposed a two-layer reward mechanism, including internal rewards and sparse rewards. The simulation results indicated that the method could reduce the correlation between samples through asynchronous training. [PERSON] et al. [10] adopted deep deterministic policy gradient for beyond-visual-range air combat and validated the effectiveness of the method in simulations. [PERSON] et al. [11] applied Bayesian inference and moving horizon optimization to air combat maneuver decision-making. The method adjusted the weights of maneuver decision factors by Bayesian inference theory, and then computed the control quantities by moving horizon optimization. [PERSON] et al. [12] proposed a hierarchical maximum-entropy reinforcement learning with reward shaping on the basis of expert knowledge. This approach achieved the second place among eight competitors in the final DARPA's AlphaDogfight Trials event. Although these studies are valuable and lay foundation for replacing human pilots with computers in real environments, there are still some shortcomings. First, in order to simplify the complexity of decision-making, many studies use discrete action spaces, whereas real action spaces are continuous. Second, in these studies, hitting the target by a missile is usually simplified as aiming at the target. However, in reality, even if the target has been locked, it may not be hit. Finally, these studies use handcrafted reward functions; namely, the agent obtains a non-zero reward at each time step. The disadvantage of handcrafted reward functions is that they may not be necessary, and the design of these functions may cost lots of effort and time. On the other hand, [PERSON] et al. [13] introduced a method of automatically selecting curricula based on the increase rate of prediction accuracy and network complexity in order to improve the learning efficiency. [PERSON] et al. [14] introduced teacher-student curriculum learning, in which the former selects sub-tasks for the latter and the latter attempts to solve these sub-tasks. Automatic curriculum learning methods also produced great performance in addition of decimal numbers [15] and Minecraft [16]. A goal proposal module was introduced in [17]. This method prefers goals that can decrease the certainty of the Q-function. The authors investigated the certainty through thirteen robotic tasks and five navigation tasks and illustrated great performance of the method. [PERSON] et al. [18] created a new method which automatically prioritizes samples with a little loss to efficiently fulfill the core mechanism of curriculum learning without changing the training procedure.