# Deep Reinforcement Learning for UAV Intelligent Mission Planning

Authors: Longfei Yue; Rennong Yang; Ying Zhang; Lixin Yu; Zhuangzhuang Wang
Journal: Complexity
 (2022)

---

## Abstract

Rapid and precise air operation mission planning is a key technology in unmanned aerial vehicles (UAVs) autonomous combat in battles. In this paper, an end-to-end UAV intelligent mission planning method based on deep reinforcement learning (DRL) is proposed to solve the shortcomings of the traditional intelligent optimization algorithm, such as relying on simple, static, low-dimensional scenarios, and poor scalability. Specifically, the suppression of enemy air defense (SEAD) mission planning is described as a sequential decision-making problem and formalized as a Markov decision process (MDP). Then, the SEAD intelligent planning model based on the proximal policy optimization (PPO) algorithm is established and a general intelligent planning architecture is proposed. Furthermore, three policy training tricks, i.e., domain randomization, maximizing policy entropy, and underlying network parameter sharing, are introduced to improve the learning performance and generalizability of PPO. Experiments results show that the model in this work is efficient and stable, and can be adapted to the unknown continuous high-dimensional environment. It can be concluded that the UAV intelligent mission planning model based on DRL has powerful intelligent planning performance, and provides a new idea for researching UAV autonomy.


---

## Body

## 1. Introduction

Mission planning is the process of making an operational plan, including a route plan, weapon plan, and avionics plan [1, 2]. Intelligent planning capability is an important symbol of unmanned aerial vehicle (UAV) autonomy. With the development of UAV technology, UAVs can fly independently and complete simple missions, such as reconnaissance and strike, which greatly improves efficiency and reduces the labor cost. However, for complex cooperative missions, UAV intelligent planning is still a key research issue.Mission planning is a decision optimization problem that solves the optimal solution of mission objective function under certain constraints, such as the shortest route [3], minimum threat [4], and maximum efficiency [5, 6]. Because the mission planning problem considers many mutually coupled factors, large decision space, and nonlinear constraints, traditional mission planning is mostly solved by intelligent optimization algorithms. Xin et al. [3] modeled the route planning problem by solving the shortest-path problem from the starting point to the target point, established an optimization model based on an ant colony optimization (ACO) algorithm, and searched for the shortest route. Zhang et al. [4] studied the tactical maneuver planning problem. Taking the minimum threat of a fighter and the maximum damage effectiveness of ground targets as the optimization objectives, and considering the capability constraints of weapons and equipment, the optimal flight route and weapon delivery time were solved using the multiobjective evolutionary algorithm based on decomposition (MOEA/D) [5]. Aiming at the task allocation problem of UAV identification, attack, and evaluation targets, a genetic algorithm (GA) was used to solve the optimal allocation result [6]. Zhang et al. [7] studied electronic warfare mission planning. By taking the route safety width and electronic jamming effect of a jammer as the objective function, the multiobjective particle swarm optimization (PSO) algorithm was used to solve it, and the optimal jamming array model was obtained. A search-based intelligent optimization algorithm can find the global optimal solution or suboptimal solution of the complex objective function through the parallel optimization mechanism, but its essence is still random search. Each solution can only be researched in the static and known environment (explicit objective function and constraints) and cannot be generalized to the dynamic and unknown environment. The computational complexity increases exponentially with the growth in problem scale. Therefore, its application has certain limitations for rapid planning and dynamic scenarios of future large-scale operations.In recent years, due to the expansion of computing power, the emergence of big data [8], and the development of artificial intelligence (AI) algorithms, learning-based methods such as neural networks [9, 10] and reinforcement learning (RL) [11] have promoted the second wave of AI. From AlphaGo [12] to AlphaGo Zero [13], AlphaZero [14], and AlphaStar [15], DRL has achieved a series of breakthroughs in a range of challenging domains. Among such methods, deep learning (DL) has been used to solve high-dimensional mapping problems, RL has been used to solve sequential decision-making problems, and DRL has been successfully applied to a series of robotics [16, 17], autonomous driving [18], real-time strategy (RTS) games [19], and optimization and scheduling [20] problems. A learning-based method, also known as the data-driven method, refers to feeding data to improve the prediction or decision-making performance of a model. Such method uses a neural network to learn or fit the complex and high-dimensional nonlinear relationship between input and output, so as to achieve the minimal mean square error or optimal prediction and decision results, and saves the mapping network parameters to realize offline training and online inferencing. It also has certain robustness and generalization for new input data, which is suitable for fast and dynamic mission planning. The comparison of the two methods is shown in Table 1.Table 1
Traditional and intelligent planning.
CategoryTraditional planningIntelligent planningAlgorithmIntelligent optimization algorithmDeep reinforcement learningMethodSearch-basedLearning-basedAdvantageParallel optimizationOffline training and online inferencing, high-dimension, scalableLimitationsNot scalable, staticComputational powerTherefore, by taking a high-risk typical suppression of enemy air defense (SEAD) mission planning [21, 22] as an example, we propose an end-to-end UAV intelligent mission planning method based on DRL. First, the mission planning problem of SEAD operation is formalized, then the basic principles of a DRL algorithm are introduced, and a DRL-based intelligent planning model is established. Finally, the superiority and potential value of this method are analyzed and verified by simulation experiments.


## 2. SEAD Mission Planning Problem Formulation

A SEAD mission is an operational style of offensive counter-air (OCA) combat carried out by an air force. Its mission goal is to break through the enemy’s surface-to-air missile (SAM) threat and strike the enemy’s radar or target through a cooperative combat between attacking UAV, named fighter, and jamming UAV, named jammer. A schematic of a SEAD mission is presented in Figure1.Figure 1
Schematic of a SEAD mission.In Figure1, the mission is to use a fighter to safely destroy an enemy SAM. However, since the detection range of the enemy SAM is longer than the attack range of the fighter, the fighter faces the threat of being detected and attacked by the SAM. Therefore, a jammer is required to jam the SAM to reduce its detection range. Only then the fighter can take the opportunity to attack and destroy the SAM. Therefore, the jammer must jam at the right position and time, and the fighter must attack at the right position and time simultaneously. The two cooperate to complete the mission.To summarize, mission planning is essentially a sequential decision-making problem. Under different space-time sequence states, a combat unit adopts the optimal decision-making sequence to transfer from the initial situation to the termination situation to achieve the mission objectives. Therefore, SEAD mission planning can be modeled as an end-to-end sequence optimization problem from the state (position and situation) to the decision (maneuver, attack, and jamming). The optimization goal is to solve an optimal state decision sequence to meet the needs of fighters and jammer to destroy an enemy SAM and ensure their own safety through tactical cooperation, as shown in Figure2.Figure 2
Intelligent mission planning SEAD model.


## 3. Deep Reinforcement Learning

### 3.1. Principles of Reinforcement Learning

RL is a machine learning approach for teaching agents how to solve tasks by trial and error. The main characters of RL are the agent [23] and the environment. The agent perceives an initial state of the environment and then decides on an action to take. The environment changes when the agent acts on it and gives the agent an instant reward signal, while the agent transfers to the next state and continues to choose new actions until reaching a termination state. The goal of the agent is to maximize the sum of rewards in the entire decision-making process, i.e., to find the optimal policy. Therefore, RL is a decision optimization method. The reinforcement learning framework is shown in Figure 3.Figure 3
Schematic of reinforcement learning framework.RL is usually modeled as a finite Markov decision process, which is represented by five tuplesS,A,R,P,γ, in which S is a finite state set, A is a finite action set, R is a reward function, P is a state transition function, and γ is a discount factor, which is used to calculate the long-term discount rewards. Assuming that the state s∈S of the agent at time t is to take action a∈A according to policy π:S⟶A, the environment will feed back an instant reward r∈R to the agent, and the agent will transfer to a new state s′∈S. Tabular reinforcement learning evaluates the state-action value through a discrete Q table, but for continuous and high-dimensional problems, it encounters the “curse of dimensionality” problem, which spurred the development of DRL [23–26].


### 3.2. Deep Reinforcement Learning

DRL refers to the combination of RL with DL. DRL uses a neural network to approximate policy and value functions to solve the high-dimensional mapping problem. The goal of the agent is to find the parameterized policyπθ with the maximum expected return Jπθ=Ετ∼πθRτ, which refers to the discounted cumulative rewards Rτ=∑t=0∞γtrt on the trajectory τ=s0,a0,s1,a1,…. θ is a policy parameter. The optimal policy is as follows:(1)πθ∗=argmaxπθEτ∼πθ∑t=0∞γtrt.A DRL algorithm is divided into three learning paradigms: value-based, policy gradient, and actor-critic. The actor-critic reinforcement learning algorithm integrates the value function and policy gradient, and uses the value function error to guide the policy gradient update to accelerate the learning speed. The policy is updated through the gradient of expected return, which can be written as follows:(2)∇θJπθ=Ετ∼πθ∑t=0T∇θlogπθa|sRτ,where πθa|s is an actor and Rτ a critic, and can also take other forms, such as a state-action value function Qπst,at, advantage function Aπst,at, or temporal difference (TD) residual Gt=rt+Vπst+1−Vπst. When the critic takes the TD residual and the value function is approximated by the neural network with parameter ω, the derivative of equation (2) is obtained, the critic is updated according to equations (3) and (4), and the actor is updated according to equation (5).(3)δ←Gt−v^St,ω,(4)ω←ω+βδ∇wv^st,ω,(5)θ←θ+αδ∇θlogπθat|st.


### 3.3. Proximal Policy Optimization Algorithm

Because proximal policy optimization (PPO) [27] algorithm is a simple, stable, and easy-to-implement actor-critic algorithm. Both Dota2 AI (OpenAI Five) [28] and Honor of Kings AI Juewu (Tencent) [29] are implemented by PPO. The PPO algorithm addresses the computationally expensive problem in the trust region policy optimization (TRPO) [30] algorithm, which needs enormous calculations to ensure monotonous policy performance improvement. Through the first-order approximation, the surrogate loss function is optimized. The new policy is calculated in each iteration, and it is close to the old policy. The policy is optimized in the direction of minimizing the loss function (maximizing the expected return). The PPO algorithm achieves a balance between sampling efficiency, algorithm performance, and engineering implementation complexity.In a PPO algorithm, a critic uses the advantage function to measure the quality of the action, and (2) becomes the following:(6)∇θJπθ=Ea∼πθ∇θlogπθa|sAπs,a.Because PPO is an on-policy method, importance sampling is introduced to improve sample utilization, and the oldπθ′ is used the sample to obtain the following:(7)∇θJπθ=Ea∼πθ′πθπθ′Aπθ′s,a∇θlogπθ.Becauseπθ∇θlogπθ=∇πθ, Eq. (7) becomes the following:(8)∇θJπθ=Ea∼πθ′∇πθπθ′Aπθ′s,a.The optimization objective function corresponding to the gradient is as follows:(9)Jπθ=Ea∼πθ′πθ/πθ′Aπθ′s,a.In practical application, based on the sampling estimation expectation, the optimization objective of PPO, i.e., the surrogate loss function, is simplified as shown in equation (10). The policy update amplitude is limited by the truncation operation, i.e., CLIP, to ensure the training stability.(10)JCLIPθ=∑s,aminrθA^,cliprθ,1−ε,1+εA^,where rθ=πθa|s/πθ′a|s is the ratio of new and old policies, and ε is a hyperparameter. The generalized advantage estimation [31] is used to calculate the advantage and keep the variance and deviation estimated by the value function small, as shown in the following:.(11)A^tGAEγ,λ=∑l=1∞γλlrt+γVst+l+1−Vst+l.The PPO algorithm is executed as Table2:Table 2
Algorithm: PPO (CLIP).
Proximal policy optimization algorithm (PPO)1. Fori = 1 to N do2. Run policyπθ′ for T timesteps, collecting s,a,r,s′,d;3. Estimate returnRτ and advantage A^tGAEγ,λ;4. Fork = 1 to K do5. Sample minibatch from the trajectory, calculate policy loss and value loss;6. Optimize surrogate loss functionJθ;7. Update policy parameterθ′←θ;8. End for9. End for


## 3.1. Principles of Reinforcement Learning

RL is a machine learning approach for teaching agents how to solve tasks by trial and error. The main characters of RL are the agent [23] and the environment. The agent perceives an initial state of the environment and then decides on an action to take. The environment changes when the agent acts on it and gives the agent an instant reward signal, while the agent transfers to the next state and continues to choose new actions until reaching a termination state. The goal of the agent is to maximize the sum of rewards in the entire decision-making process, i.e., to find the optimal policy. Therefore, RL is a decision optimization method. The reinforcement learning framework is shown in Figure 3.Figure 3
Schematic of reinforcement learning framework.RL is usually modeled as a finite Markov decision process, which is represented by five tuplesS,A,R,P,γ, in which S is a finite state set, A is a finite action set, R is a reward function, P is a state transition function, and γ is a discount factor, which is used to calculate the long-term discount rewards. Assuming that the state s∈S of the agent at time t is to take action a∈A according to policy π:S⟶A, the environment will feed back an instant reward r∈R to the agent, and the agent will transfer to a new state s′∈S. Tabular reinforcement learning evaluates the state-action value through a discrete Q table, but for continuous and high-dimensional problems, it encounters the “curse of dimensionality” problem, which spurred the development of DRL [23–26].


## 3.2. Deep Reinforcement Learning

DRL refers to the combination of RL with DL. DRL uses a neural network to approximate policy and value functions to solve the high-dimensional mapping problem. The goal of the agent is to find the parameterized policyπθ with the maximum expected return Jπθ=Ετ∼πθRτ, which refers to the discounted cumulative rewards Rτ=∑t=0∞γtrt on the trajectory τ=s0,a0,s1,a1,…. θ is a policy parameter. The optimal policy is as follows:(1)πθ∗=argmaxπθEτ∼πθ∑t=0∞γtrt.A DRL algorithm is divided into three learning paradigms: value-based, policy gradient, and actor-critic. The actor-critic reinforcement learning algorithm integrates the value function and policy gradient, and uses the value function error to guide the policy gradient update to accelerate the learning speed. The policy is updated through the gradient of expected return, which can be written as follows:(2)∇θJπθ=Ετ∼πθ∑t=0T∇θlogπθa|sRτ,where πθa|s is an actor and Rτ a critic, and can also take other forms, such as a state-action value function Qπst,at, advantage function Aπst,at, or temporal difference (TD) residual Gt=rt+Vπst+1−Vπst. When the critic takes the TD residual and the value function is approximated by the neural network with parameter ω, the derivative of equation (2) is obtained, the critic is updated according to equations (3) and (4), and the actor is updated according to equation (5).(3)δ←Gt−v^St,ω,(4)ω←ω+βδ∇wv^st,ω,(5)θ←θ+αδ∇θlogπθat|st.


## 3.3. Proximal Policy Optimization Algorithm

Because proximal policy optimization (PPO) [27] algorithm is a simple, stable, and easy-to-implement actor-critic algorithm. Both Dota2 AI (OpenAI Five) [28] and Honor of Kings AI Juewu (Tencent) [29] are implemented by PPO. The PPO algorithm addresses the computationally expensive problem in the trust region policy optimization (TRPO) [30] algorithm, which needs enormous calculations to ensure monotonous policy performance improvement. Through the first-order approximation, the surrogate loss function is optimized. The new policy is calculated in each iteration, and it is close to the old policy. The policy is optimized in the direction of minimizing the loss function (maximizing the expected return). The PPO algorithm achieves a balance between sampling efficiency, algorithm performance, and engineering implementation complexity.In a PPO algorithm, a critic uses the advantage function to measure the quality of the action, and (2) becomes the following:(6)∇θJπθ=Ea∼πθ∇θlogπθa|sAπs,a.Because PPO is an on-policy method, importance sampling is introduced to improve sample utilization, and the oldπθ′ is used the sample to obtain the following:(7)∇θJπθ=Ea∼πθ′πθπθ′Aπθ′s,a∇θlogπθ.Becauseπθ∇θlogπθ=∇πθ, Eq. (7) becomes the following:(8)∇θJπθ=Ea∼πθ′∇πθπθ′Aπθ′s,a.The optimization objective function corresponding to the gradient is as follows:(9)Jπθ=Ea∼πθ′πθ/πθ′Aπθ′s,a.In practical application, based on the sampling estimation expectation, the optimization objective of PPO, i.e., the surrogate loss function, is simplified as shown in equation (10). The policy update amplitude is limited by the truncation operation, i.e., CLIP, to ensure the training stability.(10)JCLIPθ=∑s,aminrθA^,cliprθ,1−ε,1+εA^,where rθ=πθa|s/πθ′a|s is the ratio of new and old policies, and ε is a hyperparameter. The generalized advantage estimation [31] is used to calculate the advantage and keep the variance and deviation estimated by the value function small, as shown in the following:.(11)A^tGAEγ,λ=∑l=1∞γλlrt+γVst+l+1−Vst+l.The PPO algorithm is executed as Table2:Table 2
Algorithm: PPO (CLIP).
Proximal policy optimization algorithm (PPO)1. Fori = 1 to N do2. Run policyπθ′ for T timesteps, collecting s,a,r,s′,d;3. Estimate returnRτ and advantage A^tGAEγ,λ;4. Fork = 1 to K do5. Sample minibatch from the trajectory, calculate policy loss and value loss;6. Optimize surrogate loss functionJθ;7. Update policy parameterθ′←θ;8. End for9. End for


## 4. Modeling of SEAD Intelligent Planning Based on the PPO Algorithm

### 4.1. UAV Kinematics Equation

In this paper, we aim to study the feasibility and future value of an end-to-end DRL method in intelligent mission planning. Therefore, we construct a simple two-dimensional (2-D) environment, where the fighter and jammer adopt a three-degree-of-freedom (3-DOF) model, and its kinematic equation is as follows:(12)x˙f=vfcosφf,y˙f=vfsinφf,x˙j=vjcosφj,y˙j=vjsinφj,where x˙f, y˙f, vf, and φf represent the differentiation of coordinate X and Y, speed, and heading of the fighter, respectively, and x˙j, y˙j, and vj, and φj represent the differentiation of coordinate X and Y, speed, and heading of the jammer, respectively. The heading is any continuous value in −π,π and the speed is a continuous value in 0,1.


### 4.2. MDP Modeling

#### 4.2.1. State Space

The state of the fighter, jammer, and SAM is defined as the coordinate in 2-D space, which are continuous values and represented byxf,yf, xj,yj, and xs,ys, as shown in Table 3.Table 3
State-space description.
State spacePosition coordinateDimension6DescriptionCoordinatesX, Y of fighter, jammer, and SAM


#### 4.2.2. Action Space

The actions of the fighter and jammer are their respective heading and speed, as shown in Table4. The UAV can change the position in 2-D space by controlling the heading, and reaching the attack position by controlling the speed. When the fighter fires the missile and the jammer turns on the jamming, it is automatically completed by default by setting the distance conditions. In an actual mission, it is necessary to calculate the fire time, position, and parameters in detail.Table 4
Action-space description.
Action spacePosition coordinateDimension4DescriptionHeading and velocity of fighter and jammer


#### 4.2.3. Reward Function

The design of the reward function follows the principle of Occam’s razor, i.e., it should be simple and effective. If the jammer suppresses the SAM without entering its missile range, and the fighter destroys the enemy’s SAM radar without entering its missile range, it will get a reward +1. If the jammer or fighter enters the range of SAM or flies out of the environmental boundary, it will get a reward −1. For other circumstances, reward shaping is adopted according to experience knowledge, and a continuous reward of the relative distance is given to guide the agent to learn. The reward function formula is expressed as follows:(13)r=1,ds<dfs≤dfandds<djs≤dj,−1,dfs≤dsordjs≤dsor,xf,j,yf,j≤0orxf,j,yf,j≥xmax,xf+yf+xj+yj,−xs−ys−0.1,other,where dfs and djs represent the distance between fighter and SAM and between jammer and SAM, respectively, ds represents the attack range of the SAM, df represents the attack range of the fighter, and dj represents the jamming range of the jammer.


#### 4.2.4. Environment Class Development

The environment class mainly includes two parts: the step() and reset() functions. The step() function realizes the deterministic state transition according to the UAV kinematics equation, and returns the judgment results of the new state, reward, and termination. The reset() function is designed to reset the fighter and jammer to the initial position.


### 4.3. Policy Training Tricks

#### 4.3.1. Domain Randomization

To improve the robustness of the agent policy and adapt to diversified inputs, add disturbances to the inputs in the training stage [32], as shown in Eq. (14); that is, train the agent in different environments with parameter disturbances on each random seed, so that the agent can abstract higher-level policy features, avoid overfitting the one environment and policy, and that the final learned policy is the more robust and better generalization to new environments.(14)x˜=x+rand,y˜=y+rand.


#### 4.3.2. Maximization Policy Entropy

Entropy is used to measure the randomness of random variables. The greater the entropy, the more random it is. Therefore, while maximizing the cumulative rewards, the entropy of the policy is maximized and the policy is made as random as possible. The agent can fully explore the state space to avoid the policy falling into local minima, and can explore multiple feasible schemes to complete the mission, which improves the exploration performance, robustness, and generalization performance of the policy. The calculation formula is shown as follows:(15)Hπ·|st=−∑tπ·|stlogπ·|st.


#### 4.3.3. Parameter Sharing

Actor and critic networks share network parameters. The loss function is the sum of policy loss, value function loss, and policy entropy, and the loss function gradient is backpropagated. The training difficulty is reduced by sharing the underlying network characteristics through parameter sharing. The loss function is expressed as follows:(16)Total_loss=loss_a+0.5×loss_v+0.01×loss_e.


#### 4.3.4. Construction of Intelligent Planning Model

To summarize, an end-to-end intelligent mission planning model for SEAD mission is established. The input is the position of fighter, jammer, and SAM, which is normalized by zero mean (Z-score) according to equation (17), and then put into the two-layer fully connected neural network. Finally, the heading, speed of the fighter, and jammer are output. The optimization goal is to maximize the cumulative rewards. The solution is the optimal or suboptimal policy, and the policy network architecture of the intelligent planning model is shown in Figure 4.(17)x∗=x−μσ.Figure 4
Illustration of observation vector and policy network architecture used in the present work.Therefore, this paper adopts the idea of offline training and online inferencing. First, the agent is trained in the environment. After the training is completed, the inference module is carried out to test the planning performance of the agent. The research framework is shown in Figure5.Figure 5
Research framework used in the present work.A general intelligent planning architecture is proposed, including environment, planner (agent), and controller. First, the planner inputs the initial situation and interacts with the environment, that is, offline training. The trained planner can be directly applied and input the new initial situation for online inferencing. The inference decision sequence, i.e., planning results, is put into the controller for execution. The architecture realizes the entire process of intelligent planning, as shown in Figure6.Figure 6
General architecture of intelligent planning.


## 4.1. UAV Kinematics Equation

In this paper, we aim to study the feasibility and future value of an end-to-end DRL method in intelligent mission planning. Therefore, we construct a simple two-dimensional (2-D) environment, where the fighter and jammer adopt a three-degree-of-freedom (3-DOF) model, and its kinematic equation is as follows:(12)x˙f=vfcosφf,y˙f=vfsinφf,x˙j=vjcosφj,y˙j=vjsinφj,where x˙f, y˙f, vf, and φf represent the differentiation of coordinate X and Y, speed, and heading of the fighter, respectively, and x˙j, y˙j, and vj, and φj represent the differentiation of coordinate X and Y, speed, and heading of the jammer, respectively. The heading is any continuous value in −π,π and the speed is a continuous value in 0,1.


## 4.2. MDP Modeling

### 4.2.1. State Space

The state of the fighter, jammer, and SAM is defined as the coordinate in 2-D space, which are continuous values and represented byxf,yf, xj,yj, and xs,ys, as shown in Table 3.Table 3
State-space description.
State spacePosition coordinateDimension6DescriptionCoordinatesX, Y of fighter, jammer, and SAM


### 4.2.2. Action Space

The actions of the fighter and jammer are their respective heading and speed, as shown in Table4. The UAV can change the position in 2-D space by controlling the heading, and reaching the attack position by controlling the speed. When the fighter fires the missile and the jammer turns on the jamming, it is automatically completed by default by setting the distance conditions. In an actual mission, it is necessary to calculate the fire time, position, and parameters in detail.Table 4
Action-space description.
Action spacePosition coordinateDimension4DescriptionHeading and velocity of fighter and jammer


### 4.2.3. Reward Function

The design of the reward function follows the principle of Occam’s razor, i.e., it should be simple and effective. If the jammer suppresses the SAM without entering its missile range, and the fighter destroys the enemy’s SAM radar without entering its missile range, it will get a reward +1. If the jammer or fighter enters the range of SAM or flies out of the environmental boundary, it will get a reward −1. For other circumstances, reward shaping is adopted according to experience knowledge, and a continuous reward of the relative distance is given to guide the agent to learn. The reward function formula is expressed as follows:(13)r=1,ds<dfs≤dfandds<djs≤dj,−1,dfs≤dsordjs≤dsor,xf,j,yf,j≤0orxf,j,yf,j≥xmax,xf+yf+xj+yj,−xs−ys−0.1,other,where dfs and djs represent the distance between fighter and SAM and between jammer and SAM, respectively, ds represents the attack range of the SAM, df represents the attack range of the fighter, and dj represents the jamming range of the jammer.


### 4.2.4. Environment Class Development

The environment class mainly includes two parts: the step() and reset() functions. The step() function realizes the deterministic state transition according to the UAV kinematics equation, and returns the judgment results of the new state, reward, and termination. The reset() function is designed to reset the fighter and jammer to the initial position.


## 4.2.1. State Space

The state of the fighter, jammer, and SAM is defined as the coordinate in 2-D space, which are continuous values and represented byxf,yf, xj,yj, and xs,ys, as shown in Table 3.Table 3
State-space description.
State spacePosition coordinateDimension6DescriptionCoordinatesX, Y of fighter, jammer, and SAM


## 4.2.2. Action Space

The actions of the fighter and jammer are their respective heading and speed, as shown in Table4. The UAV can change the position in 2-D space by controlling the heading, and reaching the attack position by controlling the speed. When the fighter fires the missile and the jammer turns on the jamming, it is automatically completed by default by setting the distance conditions. In an actual mission, it is necessary to calculate the fire time, position, and parameters in detail.Table 4
Action-space description.
Action spacePosition coordinateDimension4DescriptionHeading and velocity of fighter and jammer


## 4.2.3. Reward Function

The design of the reward function follows the principle of Occam’s razor, i.e., it should be simple and effective. If the jammer suppresses the SAM without entering its missile range, and the fighter destroys the enemy’s SAM radar without entering its missile range, it will get a reward +1. If the jammer or fighter enters the range of SAM or flies out of the environmental boundary, it will get a reward −1. For other circumstances, reward shaping is adopted according to experience knowledge, and a continuous reward of the relative distance is given to guide the agent to learn. The reward function formula is expressed as follows:(13)r=1,ds<dfs≤dfandds<djs≤dj,−1,dfs≤dsordjs≤dsor,xf,j,yf,j≤0orxf,j,yf,j≥xmax,xf+yf+xj+yj,−xs−ys−0.1,other,where dfs and djs represent the distance between fighter and SAM and between jammer and SAM, respectively, ds represents the attack range of the SAM, df represents the attack range of the fighter, and dj represents the jamming range of the jammer.


## 4.2.4. Environment Class Development

The environment class mainly includes two parts: the step() and reset() functions. The step() function realizes the deterministic state transition according to the UAV kinematics equation, and returns the judgment results of the new state, reward, and termination. The reset() function is designed to reset the fighter and jammer to the initial position.


## 4.3. Policy Training Tricks

### 4.3.1. Domain Randomization

To improve the robustness of the agent policy and adapt to diversified inputs, add disturbances to the inputs in the training stage [32], as shown in Eq. (14); that is, train the agent in different environments with parameter disturbances on each random seed, so that the agent can abstract higher-level policy features, avoid overfitting the one environment and policy, and that the final learned policy is the more robust and better generalization to new environments.(14)x˜=x+rand,y˜=y+rand.


### 4.3.2. Maximization Policy Entropy

Entropy is used to measure the randomness of random variables. The greater the entropy, the more random it is. Therefore, while maximizing the cumulative rewards, the entropy of the policy is maximized and the policy is made as random as possible. The agent can fully explore the state space to avoid the policy falling into local minima, and can explore multiple feasible schemes to complete the mission, which improves the exploration performance, robustness, and generalization performance of the policy. The calculation formula is shown as follows:(15)Hπ·|st=−∑tπ·|stlogπ·|st.


### 4.3.3. Parameter Sharing

Actor and critic networks share network parameters. The loss function is the sum of policy loss, value function loss, and policy entropy, and the loss function gradient is backpropagated. The training difficulty is reduced by sharing the underlying network characteristics through parameter sharing. The loss function is expressed as follows:(16)Total_loss=loss_a+0.5×loss_v+0.01×loss_e.


### 4.3.4. Construction of Intelligent Planning Model

To summarize, an end-to-end intelligent mission planning model for SEAD mission is established. The input is the position of fighter, jammer, and SAM, which is normalized by zero mean (Z-score) according to equation (17), and then put into the two-layer fully connected neural network. Finally, the heading, speed of the fighter, and jammer are output. The optimization goal is to maximize the cumulative rewards. The solution is the optimal or suboptimal policy, and the policy network architecture of the intelligent planning model is shown in Figure 4.(17)x∗=x−μσ.Figure 4
Illustration of observation vector and policy network architecture used in the present work.Therefore, this paper adopts the idea of offline training and online inferencing. First, the agent is trained in the environment. After the training is completed, the inference module is carried out to test the planning performance of the agent. The research framework is shown in Figure5.Figure 5
Research framework used in the present work.A general intelligent planning architecture is proposed, including environment, planner (agent), and controller. First, the planner inputs the initial situation and interacts with the environment, that is, offline training. The trained planner can be directly applied and input the new initial situation for online inferencing. The inference decision sequence, i.e., planning results, is put into the controller for execution. The architecture realizes the entire process of intelligent planning, as shown in Figure6.Figure 6
General architecture of intelligent planning.


## 4.3.1. Domain Randomization

To improve the robustness of the agent policy and adapt to diversified inputs, add disturbances to the inputs in the training stage [32], as shown in Eq. (14); that is, train the agent in different environments with parameter disturbances on each random seed, so that the agent can abstract higher-level policy features, avoid overfitting the one environment and policy, and that the final learned policy is the more robust and better generalization to new environments.(14)x˜=x+rand,y˜=y+rand.


## 4.3.2. Maximization Policy Entropy

Entropy is used to measure the randomness of random variables. The greater the entropy, the more random it is. Therefore, while maximizing the cumulative rewards, the entropy of the policy is maximized and the policy is made as random as possible. The agent can fully explore the state space to avoid the policy falling into local minima, and can explore multiple feasible schemes to complete the mission, which improves the exploration performance, robustness, and generalization performance of the policy. The calculation formula is shown as follows:(15)Hπ·|st=−∑tπ·|stlogπ·|st.


## 4.3.3. Parameter Sharing

Actor and critic networks share network parameters. The loss function is the sum of policy loss, value function loss, and policy entropy, and the loss function gradient is backpropagated. The training difficulty is reduced by sharing the underlying network characteristics through parameter sharing. The loss function is expressed as follows:(16)Total_loss=loss_a+0.5×loss_v+0.01×loss_e.


## 4.3.4. Construction of Intelligent Planning Model

To summarize, an end-to-end intelligent mission planning model for SEAD mission is established. The input is the position of fighter, jammer, and SAM, which is normalized by zero mean (Z-score) according to equation (17), and then put into the two-layer fully connected neural network. Finally, the heading, speed of the fighter, and jammer are output. The optimization goal is to maximize the cumulative rewards. The solution is the optimal or suboptimal policy, and the policy network architecture of the intelligent planning model is shown in Figure 4.(17)x∗=x−μσ.Figure 4
Illustration of observation vector and policy network architecture used in the present work.Therefore, this paper adopts the idea of offline training and online inferencing. First, the agent is trained in the environment. After the training is completed, the inference module is carried out to test the planning performance of the agent. The research framework is shown in Figure5.Figure 5
Research framework used in the present work.A general intelligent planning architecture is proposed, including environment, planner (agent), and controller. First, the planner inputs the initial situation and interacts with the environment, that is, offline training. The trained planner can be directly applied and input the new initial situation for online inferencing. The inference decision sequence, i.e., planning results, is put into the controller for execution. The architecture realizes the entire process of intelligent planning, as shown in Figure6.Figure 6
General architecture of intelligent planning.


## 5. Evaluation in Simulation Experiments

### 5.1. Experimental Setup and Training Tricks

In this paper, the SEAD environment is a square area of 100 × 100 km2. The fighter position is (40, 40), the attack range is 30 km, the jammer position is (30, 30), the attack range is 50 km, the SAM position is (80, 80), and the attack range is 40 km. After jamming, the SAM attack range is reduced to 25 km, as shown in Figure 7. In the experiment, the above range is reduced 100 times for normalization, which is easy for neural network training and can prevent gradient vanishing. The simulation environment uses Python 3.6 and PyCharm. The intelligent planning experiments of three different scenarios are completed separately. Compared with other classical DRL algorithms, robustness and ablation studies are carried out to verify the algorithm performance.Figure 7
SEAD environment in present work.The hyperparameter settings in this work are shown in Table5. The neural network adopts orthogonal initialization, the optimizer is Adam [33], and other training parameters are described in Section 5.1.1.Table 5
Hyperparameter settings.
ParameterValuemax_episode300learning_rate0.0003Clip0.2Gamma0.995Lambda0.95max_step_per_round1000coeff_loss_policy1coeff_loss_value0.5coeff_loss_entropy0.01minibatch_size64num_seed3


#### 5.1.1. State Space

The status of the fighter, jammer, and SAM is defined as the coordinate position in 2-D space, which are continuous values and represented byxf,yf, xj,yj, and xs,ys.(1) Advantage function normalization. The advantage function value is normalized to improve the training stability and policy learning skills. The formula is as follows:(18)A∗=A−μAσA.(2) Value function normalization. Similarly, the value function loss is also normalized. The policy training skills are studied and compared in the subsequent simulation. The formula is as follows:(19)Vloss∗=∑iNVi−Ri26TσRi.


#### 5.1.2. Adaptive Adjustment Parameters

(1) Adaptive learning rate. In the early training stage, a larger learning rate is adopted to accelerate convergence, and in the later training stage, a smaller learning rate is adopted to find the optimal value. The formula is as follows:(20)lr=LR∗1−i_EPMAX_EP.(2) Adaptive clip value. The clip adaptive change is consistent with the change of learning rate. A larger clip value is allowed to accelerate the policy update in the early training stage, and a smaller clip value is used in the later stage to ensure the policy update is stable. The formula is as follows:(21)clip=CLIP×1−i_EPMAX_EP.


### 5.2. Intelligent Planning for Fighter-Jammer Scenario

In Experiment 1, we set up a fighter-jammer scenario, as depicted in Section 5.1. Therefore, the fighter has to be able to safely destroy the SAM under jamming to complete the mission.Therefore, in this work, the intelligent planning model is used to simulate the 300 episodes to train the model, and use the trained model for online inferencing to test the intelligent planning performance and cooperative attack performance.


#### 5.2.1. Offline Training

Three different random seeds are used to train the PPO policy and value function network. We record the average timesteps reward in the training process, and compare it with the classical advantage actor-critic (A2C) and TRPO algorithms to obtain the cumulative rewards learning curve, as shown in Figure8.Figure 8
Learning curves for the fighter-jammer scenario. The shaded region represents the standard deviation of average evaluation over three trails.As it can be seen from the above figure, the A2C model has a large variance and the TRPO model has poor convergence performance. However, the PPO model used in this work has higher episode rewards, more stable training, smaller episode rewards variance, and good robustness, so its performance is better.


#### 5.2.2. Online Planning

The trained model is used to input the initial situation information of the environment, test the online inferencing performance of the model, and obtain the cooperative attack process, as shown in Figure9.Figure 9
(a) Fighter and jammer flight cooperatively; (b) jammer jamming the Sam to reduce the weapon range, and the fighter takes the opportunity to attack the Sam. When the Sam is destroyed, it turns black.
(a)(b)It can be seen from Figure9(a) that the fighter and jammer cleverly complete a cooperative attack. Before the jammer successfully jams, the fighter does not enter the detection range of the SAM or start an attack for its own safety. As can be seen from Figure 9(b), when the jammer jams the SAM and degrades the detection range of the SAM, the fighter attack quickly and successfully destroys the SAM, which reflects strong intelligent cooperative planning performance.


### 5.3. Intelligent Planning for Fighter-Decoy Scenario

To further test the intelligent planning performance of the model proposed in this paper, a cooperative attack for fighter-decoy scenario is designed in which the SAM attack range is 20 km and that of the fighter is also 20 km, which could not directly and safely destroy the SAM and target. The decoy is a low-cost expendable UAV, which could be sacrificed. Therefore, the fighter has to attack with the decoy cooperatively, sacrificing the decoy first and utilizing the interval between the SAM attacks on the decoy to destroy the SAM and target to complete the mission. Therefore, the reward function is modified as follows:(22)r=1,dfs≤dfanddds≤ds−1,dds>dsorxf,d,yf,d≤0orxf,d,yf,d≥xmaxxf+yf−xd−yd+0.8,other,where dfs and dds represent the distance between fighter and SAM and between decoy and SAM, respectively, ds represents the attack range of the SAM, df represents the attack range of the fighter, xd and yd represent the coordinate X and Y of the decoy.


#### 5.3.1. Offline Training

Similarly, the policy and value function network of PPO are trained on three different random seeds, and the cumulative rewards of each episode in the training process are recorded and compared with the intelligent planning model based on A2C and TRPO to obtain the episode rewards learning curve, as shown in Figure10.Figure 10
Learning curves for the fighter-decoy scenario. The shaded region represents a standard deviation of average evaluation over three trails.It can be seen from the figure that the A2C and TRPO models have large variance, low training stability, and poor convergence, while the PPO model proposed in this paper obtains higher cumulative episode rewards, a smooth and stable learning curve, and excellent convergence performance.


#### 5.3.2. Online Planning

The trained PPO model is tested in the environment to verify the planning performance of the model. The results are as follows:In Figure11(a), the decoy flies directly to the SAM to attract SAM radar tracking and attacking, while the fighter waits for an opportunity to fly around. In Figure 11(b), the fighter utilizes the time interval between the SAM tracking, locking on to the decoy, and completing the attack quickly, successfully destroying the SAM and the targets. The fighter sacrifices the decoy, but completes the mission, which shows that the PPO-based intelligent planning model has certain tactical cooperative planning performance.Figure 11
(a) Decoy entry early; (b) Sam tracks decoy, and fighter takes the opportunity to attack the Sam and target. When the Sam and target are destroyed, they turn black.
(a)(b)


### 5.4. Robustness Studies

The robustness of the intelligent planning model is tested next, and certain randomness is added to the training environment. As shown in Figure12, the starting positions of fighter, decoy, and SAM can change randomly in a certain surrounding area to test the generalization performance and robustness of the trained model in the unknown environment.Figure 12
SEAD environment with different initial positions.The test results are shown in Figure13. In Figure 13, the initial position of the fighter is (20, 20), that of the decoy (85, 95), and that of the SAM (55, 60). Upon being input into the trained model, it is found that the fighter will still intelligently wait for the decoy to enter the SAM range first, and then take the opportunity to attack quickly and successfully destroy the SAM and target, as shown in Figure 13. Therefore, this shows that the model has certain robustness and generalization performance to unknown situations, can adapt to an uncertain environment, and has strong practical application value.Figure 13
(a) Decoy enters early; (b) Sam tracks and attacks decoy, and fighter takes the opportunity to attack the Sam and target. When the decoy, Sam, and target are destroyed, they turn black.
(a)(b)


### 5.5. Ablation Studies

Finally, the effectiveness of different training tricks are compared; that is, the performance of models using all tricks, and the advantage function normalization, layer orthogonal initialization, value function normalization, adaptive learning rate, and adaptive clip. The ablation studies are completed. The results are shown in Figure14.Figure 14
Learning-curve comparison of different tricks.As it can be seen from Figure14, the episode rewards using all training tricks are higher. The advantage function normalization will greatly improve the early convergence speed of the model, layer orthogonal initialization can improve the final performance of the model, and other tricks have relatively little impact on model performance.


## 5.1. Experimental Setup and Training Tricks

In this paper, the SEAD environment is a square area of 100 × 100 km2. The fighter position is (40, 40), the attack range is 30 km, the jammer position is (30, 30), the attack range is 50 km, the SAM position is (80, 80), and the attack range is 40 km. After jamming, the SAM attack range is reduced to 25 km, as shown in Figure 7. In the experiment, the above range is reduced 100 times for normalization, which is easy for neural network training and can prevent gradient vanishing. The simulation environment uses Python 3.6 and PyCharm. The intelligent planning experiments of three different scenarios are completed separately. Compared with other classical DRL algorithms, robustness and ablation studies are carried out to verify the algorithm performance.Figure 7
SEAD environment in present work.The hyperparameter settings in this work are shown in Table5. The neural network adopts orthogonal initialization, the optimizer is Adam [33], and other training parameters are described in Section 5.1.1.Table 5
Hyperparameter settings.
ParameterValuemax_episode300learning_rate0.0003Clip0.2Gamma0.995Lambda0.95max_step_per_round1000coeff_loss_policy1coeff_loss_value0.5coeff_loss_entropy0.01minibatch_size64num_seed3


### 5.1.1. State Space

The status of the fighter, jammer, and SAM is defined as the coordinate position in 2-D space, which are continuous values and represented byxf,yf, xj,yj, and xs,ys.(1) Advantage function normalization. The advantage function value is normalized to improve the training stability and policy learning skills. The formula is as follows:(18)A∗=A−μAσA.(2) Value function normalization. Similarly, the value function loss is also normalized. The policy training skills are studied and compared in the subsequent simulation. The formula is as follows:(19)Vloss∗=∑iNVi−Ri26TσRi.


### 5.1.2. Adaptive Adjustment Parameters

(1) Adaptive learning rate. In the early training stage, a larger learning rate is adopted to accelerate convergence, and in the later training stage, a smaller learning rate is adopted to find the optimal value. The formula is as follows:(20)lr=LR∗1−i_EPMAX_EP.(2) Adaptive clip value. The clip adaptive change is consistent with the change of learning rate. A larger clip value is allowed to accelerate the policy update in the early training stage, and a smaller clip value is used in the later stage to ensure the policy update is stable. The formula is as follows:(21)clip=CLIP×1−i_EPMAX_EP.


## 5.1.1. State Space

The status of the fighter, jammer, and SAM is defined as the coordinate position in 2-D space, which are continuous values and represented byxf,yf, xj,yj, and xs,ys.(1) Advantage function normalization. The advantage function value is normalized to improve the training stability and policy learning skills. The formula is as follows:(18)A∗=A−μAσA.(2) Value function normalization. Similarly, the value function loss is also normalized. The policy training skills are studied and compared in the subsequent simulation. The formula is as follows:(19)Vloss∗=∑iNVi−Ri26TσRi.


## 5.1.2. Adaptive Adjustment Parameters

(1) Adaptive learning rate. In the early training stage, a larger learning rate is adopted to accelerate convergence, and in the later training stage, a smaller learning rate is adopted to find the optimal value. The formula is as follows:(20)lr=LR∗1−i_EPMAX_EP.(2) Adaptive clip value. The clip adaptive change is consistent with the change of learning rate. A larger clip value is allowed to accelerate the policy update in the early training stage, and a smaller clip value is used in the later stage to ensure the policy update is stable. The formula is as follows:(21)clip=CLIP×1−i_EPMAX_EP.


## 5.2. Intelligent Planning for Fighter-Jammer Scenario

In Experiment 1, we set up a fighter-jammer scenario, as depicted in Section 5.1. Therefore, the fighter has to be able to safely destroy the SAM under jamming to complete the mission.Therefore, in this work, the intelligent planning model is used to simulate the 300 episodes to train the model, and use the trained model for online inferencing to test the intelligent planning performance and cooperative attack performance.


### 5.2.1. Offline Training

Three different random seeds are used to train the PPO policy and value function network. We record the average timesteps reward in the training process, and compare it with the classical advantage actor-critic (A2C) and TRPO algorithms to obtain the cumulative rewards learning curve, as shown in Figure8.Figure 8
Learning curves for the fighter-jammer scenario. The shaded region represents the standard deviation of average evaluation over three trails.As it can be seen from the above figure, the A2C model has a large variance and the TRPO model has poor convergence performance. However, the PPO model used in this work has higher episode rewards, more stable training, smaller episode rewards variance, and good robustness, so its performance is better.


### 5.2.2. Online Planning

The trained model is used to input the initial situation information of the environment, test the online inferencing performance of the model, and obtain the cooperative attack process, as shown in Figure9.Figure 9
(a) Fighter and jammer flight cooperatively; (b) jammer jamming the Sam to reduce the weapon range, and the fighter takes the opportunity to attack the Sam. When the Sam is destroyed, it turns black.
(a)(b)It can be seen from Figure9(a) that the fighter and jammer cleverly complete a cooperative attack. Before the jammer successfully jams, the fighter does not enter the detection range of the SAM or start an attack for its own safety. As can be seen from Figure 9(b), when the jammer jams the SAM and degrades the detection range of the SAM, the fighter attack quickly and successfully destroys the SAM, which reflects strong intelligent cooperative planning performance.


## 5.2.1. Offline Training

Three different random seeds are used to train the PPO policy and value function network. We record the average timesteps reward in the training process, and compare it with the classical advantage actor-critic (A2C) and TRPO algorithms to obtain the cumulative rewards learning curve, as shown in Figure8.Figure 8
Learning curves for the fighter-jammer scenario. The shaded region represents the standard deviation of average evaluation over three trails.As it can be seen from the above figure, the A2C model has a large variance and the TRPO model has poor convergence performance. However, the PPO model used in this work has higher episode rewards, more stable training, smaller episode rewards variance, and good robustness, so its performance is better.


## 5.2.2. Online Planning

The trained model is used to input the initial situation information of the environment, test the online inferencing performance of the model, and obtain the cooperative attack process, as shown in Figure9.Figure 9
(a) Fighter and jammer flight cooperatively; (b) jammer jamming the Sam to reduce the weapon range, and the fighter takes the opportunity to attack the Sam. When the Sam is destroyed, it turns black.
(a)(b)It can be seen from Figure9(a) that the fighter and jammer cleverly complete a cooperative attack. Before the jammer successfully jams, the fighter does not enter the detection range of the SAM or start an attack for its own safety. As can be seen from Figure 9(b), when the jammer jams the SAM and degrades the detection range of the SAM, the fighter attack quickly and successfully destroys the SAM, which reflects strong intelligent cooperative planning performance.


## 5.3. Intelligent Planning for Fighter-Decoy Scenario

To further test the intelligent planning performance of the model proposed in this paper, a cooperative attack for fighter-decoy scenario is designed in which the SAM attack range is 20 km and that of the fighter is also 20 km, which could not directly and safely destroy the SAM and target. The decoy is a low-cost expendable UAV, which could be sacrificed. Therefore, the fighter has to attack with the decoy cooperatively, sacrificing the decoy first and utilizing the interval between the SAM attacks on the decoy to destroy the SAM and target to complete the mission. Therefore, the reward function is modified as follows:(22)r=1,dfs≤dfanddds≤ds−1,dds>dsorxf,d,yf,d≤0orxf,d,yf,d≥xmaxxf+yf−xd−yd+0.8,other,where dfs and dds represent the distance between fighter and SAM and between decoy and SAM, respectively, ds represents the attack range of the SAM, df represents the attack range of the fighter, xd and yd represent the coordinate X and Y of the decoy.


### 5.3.1. Offline Training

Similarly, the policy and value function network of PPO are trained on three different random seeds, and the cumulative rewards of each episode in the training process are recorded and compared with the intelligent planning model based on A2C and TRPO to obtain the episode rewards learning curve, as shown in Figure10.Figure 10
Learning curves for the fighter-decoy scenario. The shaded region represents a standard deviation of average evaluation over three trails.It can be seen from the figure that the A2C and TRPO models have large variance, low training stability, and poor convergence, while the PPO model proposed in this paper obtains higher cumulative episode rewards, a smooth and stable learning curve, and excellent convergence performance.


### 5.3.2. Online Planning

The trained PPO model is tested in the environment to verify the planning performance of the model. The results are as follows:In Figure11(a), the decoy flies directly to the SAM to attract SAM radar tracking and attacking, while the fighter waits for an opportunity to fly around. In Figure 11(b), the fighter utilizes the time interval between the SAM tracking, locking on to the decoy, and completing the attack quickly, successfully destroying the SAM and the targets. The fighter sacrifices the decoy, but completes the mission, which shows that the PPO-based intelligent planning model has certain tactical cooperative planning performance.Figure 11
(a) Decoy entry early; (b) Sam tracks decoy, and fighter takes the opportunity to attack the Sam and target. When the Sam and target are destroyed, they turn black.
(a)(b)


## 5.3.1. Offline Training

Similarly, the policy and value function network of PPO are trained on three different random seeds, and the cumulative rewards of each episode in the training process are recorded and compared with the intelligent planning model based on A2C and TRPO to obtain the episode rewards learning curve, as shown in Figure10.Figure 10
Learning curves for the fighter-decoy scenario. The shaded region represents a standard deviation of average evaluation over three trails.It can be seen from the figure that the A2C and TRPO models have large variance, low training stability, and poor convergence, while the PPO model proposed in this paper obtains higher cumulative episode rewards, a smooth and stable learning curve, and excellent convergence performance.


## 5.3.2. Online Planning

The trained PPO model is tested in the environment to verify the planning performance of the model. The results are as follows:In Figure11(a), the decoy flies directly to the SAM to attract SAM radar tracking and attacking, while the fighter waits for an opportunity to fly around. In Figure 11(b), the fighter utilizes the time interval between the SAM tracking, locking on to the decoy, and completing the attack quickly, successfully destroying the SAM and the targets. The fighter sacrifices the decoy, but completes the mission, which shows that the PPO-based intelligent planning model has certain tactical cooperative planning performance.Figure 11
(a) Decoy entry early; (b) Sam tracks decoy, and fighter takes the opportunity to attack the Sam and target. When the Sam and target are destroyed, they turn black.
(a)(b)


## 5.4. Robustness Studies

The robustness of the intelligent planning model is tested next, and certain randomness is added to the training environment. As shown in Figure12, the starting positions of fighter, decoy, and SAM can change randomly in a certain surrounding area to test the generalization performance and robustness of the trained model in the unknown environment.Figure 12
SEAD environment with different initial positions.The test results are shown in Figure13. In Figure 13, the initial position of the fighter is (20, 20), that of the decoy (85, 95), and that of the SAM (55, 60). Upon being input into the trained model, it is found that the fighter will still intelligently wait for the decoy to enter the SAM range first, and then take the opportunity to attack quickly and successfully destroy the SAM and target, as shown in Figure 13. Therefore, this shows that the model has certain robustness and generalization performance to unknown situations, can adapt to an uncertain environment, and has strong practical application value.Figure 13
(a) Decoy enters early; (b) Sam tracks and attacks decoy, and fighter takes the opportunity to attack the Sam and target. When the decoy, Sam, and target are destroyed, they turn black.
(a)(b)


## 5.5. Ablation Studies

Finally, the effectiveness of different training tricks are compared; that is, the performance of models using all tricks, and the advantage function normalization, layer orthogonal initialization, value function normalization, adaptive learning rate, and adaptive clip. The ablation studies are completed. The results are shown in Figure14.Figure 14
Learning-curve comparison of different tricks.As it can be seen from Figure14, the episode rewards using all training tricks are higher. The advantage function normalization will greatly improve the early convergence speed of the model, layer orthogonal initialization can improve the final performance of the model, and other tricks have relatively little impact on model performance.


## 6. Conclusions

In this paper, we propose an end-to-end DRL-based UAV intelligent mission planning method. First, a SEAD mission is selected as the research object, and mission planning is described as a sequential decision-making problem. Then, the SEAD mission intelligent planning model based on the PPO algorithm is established, including the design of UAV state space, action space, and reward function. Three policy training tricks, namely domain randomization, maximizing policy entropy, and parameter sharing, are introduced, and the intelligent planning general architecture is constructed. Finally, two experiments analyses, robustness studies, and an ablation experiment are completed. We conclude that the intelligent planning model based on deep reinforcement learning, which adopts an end-to-end architecture and offline training and online inferencing, can adapt to the dynamic situation, and it is advanced and valuable. In future work, this end-to-end method will be extended to large-scale complex combat scenarios, and the problem of multi-agent cooperative planning will be studied in depth.



---

*Source: 3551508-2022-03-31.xml*