# Intelligent Online Multiconstrained Reentry Guidance Based on Hindsight Experience Replay

Authors: Qingji Jiang; Xiaogang Wang; Yuliang Bai; Yu Li
Journal: International Journal of Aerospace Engineering
 (2023)

---

## Abstract

Traditional guidance algorithms for hypersonic glide vehicles face the challenge of real-time requirements and robustness to multiple deviations or tasks. In this paper, an intelligent online multiconstrained reentry guidance is proposed to strikingly reduce computational burden and enhance the effectiveness with multiple constraints. First, the simulation environment of reentry including dynamics, multiconstraints, and control variables is built. Different from traditional decoupling methods, the bank angle command including its magnitude and sign is designed as the sole guidance variable. Secondly, a policy neural network is designed to output end-to-end guidance commands. By transforming the reentry process into a Markov Decision Process (MDP), the policy network can be trained by deep reinforcement learning (DRL). To address the sparse reward issue caused by multiconstraints, the improved Hindsight Experience Replay (HER) method is adaptively combined with Deep Deterministic Policy Gradient (DDPG) algorithm by transforming multiconstraints into multigoals. As a result, the novel training algorithm can realize higher utilization of failed data and improve the rate of convergence. Finally, simulations for typical scenes show that the policy network in the proposed guidance can output effective commands in much less time than the traditional method. The guidance is robust to initial bias, different targets, and online aerodynamic deviation.


---

## Body

## 1. Introduction

From the last few decades until now, hypersonic glide vehicles (HGV) have attracted the attention of researchers [1–5] due to their high velocity and wide flight airspace. Usually, cooperative guidance has been widely studied as an effective threat to valuable targets represented by hypersonic vehicles [6–10]. HGV shows its superiority against cooperation for its large flight envelope, consisting of the boost phase, initial descend phase, glide phase, and dive phase. To achieve long-range penetration, the glide phase plays an important role. For different mission requirements, the guidance law of the glide phase presents different results. This paper will focus on the flight mission in that HGV flies to the neighborhood of the target at a certain height, and the total time of the glide phase is constrained.Undergoing years of development, the most popular methods of reentry guidance can be summarized in reference tracking guidance [11–22] and predictor-corrector guidance [23–29]. As far back as 1978, Harpold and Graves [11] described the theoretical basis and guidance architecture of reference tracking guidance, which has been successfully applied in the guidance of space shuttles and other reentry vehicles. The reference tracking guidance is composed of offline trajectory optimization and online trajectory tracking. Commonly, the optimal or suboptimal trajectory is calculated by planning or optimization and stored in an onboard computer before the flight. Once entering the glide phase, the trajectory tracker generates guidance commands according to errors between the actual flight and the stored standard trajectory. Not taking deviation and disturbance into account, the trajectory planning problem can be seen as an optimal control problem, which can be solved by the Legendre pseudospectral method [13, 14], improved Gauss pseudospectral method [15, 16], etc. Other commonly used optimization methods in reentry flight mainly include particle swarm optimization [17] and multiphase convex programming [18]. The trajectory tracking methods mainly include linear quadratic regulator [19], sliding mode control [20], nonlinear geometric method [21], and fuzzy approximation method [22]. Reference tracking guidance is easy to realize, and the method does not consume much onboard computing resource. Nevertheless, the method is easily affected by initial state changes and kinds of disturbance such as aerodynamic errors, which means its robustness is not good in some situations.Oppositely, predictor-corrector guidance requires no reference trajectory, which makes it flexible in more flight missions with initial errors and onboard disturbance. The task information stored in the onboard computer is merely terminal constraints. The predictor computes the terminal states by integrating dynamics equations. The corrector calculates the command of AOA (angle of attack) and bank angle to correct the errors between the calculated terminal states and terminal constraints. By repeating the guidance flow, the final errors of terminal constraints are limited to a very small range. Based on different prediction methods, predictor-corrector guidance is classified as analytical predictor-corrector guidance (APCG) and numerical predictor-corrector guidance (NPCG). Analytical prediction reduces the computational work by making some assumptions and simplifying equations of flight states. Kluever [23] proposed a guidance method for the skip-entry phase based on closed-form expressions that are derived from a matched asymptotic expansion analysis. Yu and Chen [24] innovatively presented a method for the derivation of analytical solutions to hypersonic gliding problems by transforming the equations into a linear system with variable coefficients and solving it based on spectral decomposition. Zhang et al. [25] established an auxiliary geocentric rotation frame, simplified the complex dynamics, and got a series of analytical expressions taking energy as the argument. Though the calculation of this kind of prediction is rapid, it faces the problem of insufficient accuracy. Numerical prediction is widely used due to its high accuracy. Joshi et al. [27] adopted NPCG at each guidance cycle to generate a feasible trajectory and presented a guidance algorithm that can satisfy path and terminal constraints. Lu [28] summarized the methods of predecessors and developed a unified predictor-corrector guidance method, based on a simple and robust numerical predictor with an altitude-rate-dependent, which is suitable for kinds of reentry vehicles. Cheng et al. [29] designed a deep neural network in place of the numerical integration and provided an intelligent multiconstrained predictor-corrector guidance, which can be seen as an improvement of NPCG. Methods based on NPCG improve the accuracy, meanwhile causing the problem of intensive computation. Limited by the computing power of the onboard computer, NPCG is not operative in practical use, especially in tasks with a short guidance cycle. Both the reference tracking guidance and predictor-corrector guidance have advantages and disadvantages. If there is no deviation or disturbance, the reentry problem can be solved under the framework of trajectory planning, which is widely studied [30–33]. However, the complex online deviation and rapidity requirement call for an intelligent online method.In recent years, deep reinforcement learning (DRL) has made a great breakthrough [34–36]. Deep Q-learning (DQN) [35] was first proposed in 2013, which shows great performance in dealing with continuous states in the game field. Then, the Deep Deterministic Policy Gradient (DDPG) [36] further expands the action space from discrete to continuous, which aroused the rapid development of DRL [37–39]. With the strong representation ability of deep learning (DL) and the natural decision-making ability of reinforcement learning (RL), DRL is widely applied in various fields [40–42]. The sparse reward problem is common in DRL tasks, which describes that the agent may only receive a reward signal at the terminal time. To improve the efficiency of training and avoid the sparse reward, based on universal value function approximators [43], Andrychowicz et al. [44] innovatively proposed the Hindsight Experience Replay (HER) method. The HER method replaces the initial goal of the agent with an achieved goal, which greatly leads to abundant reward signals. The original intention of this design decides that the HER method is suitable for those tasks with multigoals. Prianto et al. [45] applied HER to soft actor-critic used for path planning for multiarm manipulators, and the trained agent can generate the shortest path for arbitrary scenarios. Manela and Biess [46] combined HER with curriculum learning on three challenging throwing tasks, which shows wonderful performance in multiple goals and sparse reward functions. Considering that the terminal constraints in the reentry task are similar to multigoals, we can explore the HER method in the DRL guidance frame.This paper is aimed at finding a real-time plan and guidance scheme for reentry with the help of deep reinforcement learning. Compared with previous studies, the main contributions of this paper are as follows:(1)
The problem of reentry with multiple (path and terminal) constraints is transformed into a Markov Decision Process, which is the precondition of solving it in DRL. Different from the decoupling method in most previous studies, the guidance command of bank angle including magnitude and sign is combined as an action. This makes the guidance logic concise(2)
The HER method is introduced and improved in the training of guidance commands. The HER condition is presented creatively according to the peculiarities of the reentry problem. The flow of this method is easy to generalize to similar guidance problems with other additional constraints(3)
The guidance network is trained offline and called online. The guidance scheme provides an end-to-end guidance command which is calculated by the well-trained network rather than complex guidance schemes. The engineering implementation of the proposed algorithm is simple, and the real-time performance is excellentThe remainder of the paper is organized as follows. Section2 describes the reentry problem with multiconstraints and the bounds of control variables. Section 3 introduces the basic theory of DRL and the benefit of the HER method. Section 4 proposes the intelligent multiconstrained guidance algorithm and gives the architecture of two kinds of neural networks. Section 5 gives the simulations and analysis to verify the efficiency of the method. Finally, the conclusions of this paper are in Section 6.


## 2. Problem Formulation

### 2.1. Reentry Dynamics Equations

Considering the earth’s rotation, the dynamics equations of the hypersonic glide vehicle are given as follows:(1)V̇=−Dm−gsinγ+ω2rcosϕsinγcosϕ−cosγsinϕcosψ,γ̇=LcosσmV−gV−Vrcosγ+2ωcosϕsinψ+ω2rcosϕVcosϕcosγ+sinγsinϕcosψ,ψ̇=LsinσmVcosγ+Vrcosγsinψtanϕ−2ωtanγcosϕcosψ−sinϕ+ω2rsinϕcosϕsinψVcosγ,ṙ=Vsinγ,θ̇=Vcosγsinψrcosϕ,ϕ̇=Vcosγcosψr,where V represents the velocity, γ represents the flight path angle, ψ represents the velocity heading angle, r represents the distance between the Earth center and the vehicle, θ represents the longitude, ϕ represents the latitude, m represents the vehicle mass, g represents the gravitational acceleration, ω represents the self-rotation rate of the Earth, and σ represents the bank angle. The expressions of lift L and drag force D are
(2)D=12ρV2CDSm,L=12ρV2CLSm,where ρ represents the atmospheric density, Sm represents the reference area, and CD and CL represent the drag coefficient and lift coefficient, respectively, which can be obtained by two interpolation functions for AOA and the Mach number.ρ is a function of height H (H=r−Re, Re is the radius of the Earth):
(3)ρ=ρ0e−βH.Given an initial state, the trajectory can be obtained by integration based on Equation (1), command of AOA and bank angle.


### 2.2. Multiple Constraints

To complete the flight mission, the HGV must satisfy all the path constraints and terminal constraints.


#### 2.2.1. Path Constraints

During the reentry flight, the path constraints can be expressed as(4)Q̇=kQρ0.5V3.15<Q̇max,q=12ρV2<qmax,n=D2+L2m<nmax,where Q̇ is the heating rate of the stagnation point, q is the dynamic pressure, n is the aerodynamic load, and kQ is a constant. Q̇max, qmax, and nmax denote the maximum limits, respectively.Apart from the three hard constraints, there is a soft QEGC (quasiequilibrium glide condition) constraint, which can make the reentry trajectory smooth:(5)Lcosσ−g+V2r≥0.According to Equations (2)–(5), we can obtain the three lower bounds and one upper bound of H vs. V, in which the constraint of the upper bound is soft.
(6)H≥2βlnkQρ00.5V3.15Q̇max,H≥1βlnρ0V22qmax,H≥1βlnCD2+CL2ρ0V2Sm2nmaxmg,H≤1βlnCLρ0V2Smr2mgr−V2.According to Equation (6), the H-V corridor in the glide phase is obtained. Only if the flight trajectory of HGV is limited in the corridor, the path constraints can be satisfied. Intuitively, the H-V corridor is shown in Figure 1.Figure 1
Height-velocity corridor of HGV during the reentry flight.


#### 2.2.2. Terminal Constraints

Assume that HGV is required to reach the vicinity of the target at a specified height, the terminal constraints (height errorΔH, distance error Δd) can be summarized as
(7)ΔH=Hf−Hre<ΔHtd,Δd=Rearccossinϕfsinϕtar+cosϕfcosϕtarcosθf−θtar<Δdtd,where Hf, ϕf, and θf represent the final height, latitude, and longitude at the terminal time, Hre is the required height, ϕtar and θtar represent the latitude and longitude of the target, and ΔHtd and Δdtd are the thresholds of height and distance.In many practical application scenes, the terminal constraint on velocity is not so tight, which means that the velocity constraint takes the form of(8)Vf>Vre,where Vf is the terminal velocity and Vre is the required minimum velocity.Compared to traditional reentry tasks, we consider more terminal constraints in different tasks. For example, to arrive at the required timeTre, there is another constraint:
(9)ΔT=Tf−Tre<ΔTtd,where Tf is the flight time and ΔTtd is the threshold of time.After this, we give the criterion for the success of a reentry flight. In cases where all the path constraints are satisfied, if the terminal statesΔd<3×105m, ΔH<2000m, ΔT<20s, and Vf>2500m/s, then the flight mission is successful.Similarly, if the reentry task requires HGV to arrive at a desired terminal approach angle, the relevant constraint condition is adaptively formulated:(10)Δψ=ψLOS−ψapproach<Δψtd,where ψLOS is the LOS (line of sight) angle, ψapproach is the desired approach angle, and Δψtd is the angle threshold.


#### 2.2.3. Switch Condition

As is shown in Figure2, HGV cannot begin sliding from an arbitrary initial status so there is a switch condition. The switch condition refers to the constraint condition at the junction of the initial descent phase and glide phase.Figure 2
The flight phases of HGV.Referring to [1], the switch condition is defined as that the absolute value of the difference between the slope of the QEGC at the current point and the actual slope of r-V is less than a small preselected value.
(11)drdV−drdVQEGC<δ0,where δ0 is a small value.The first slope can be derived from Equation (1):
(12)drdV=Vsinγ−D/m−gsinγ.The QEGC equation is rewritten as(13)ρV2CLSm2mcosσ+V2cosγr−gcosγ=0.According to Equation (13), the derivate of ρ to V is obtained:
(14)dρdV=dρdrdrdV=−βρdrdV.Both sides of Equation (13) take a derivative to velocity, and then, we can obtain
(15)ρV2CLSmcosσ2m2V−βdhdV+2Vr−V2r2dhdV+2grdhdV=0.Substituting Equation (13) into (15), then we can obtain
(16)gcosγ−V2cosγr2V−βdhdV+2Vr=drdVV2r2−2gr.Suppose thatcosγ≈1, Equation (16) is simplified as
(17)2gV=drdVV2r2−2gr+βg−V2r.Then, the second slope is obtained:(18)drdVQEGC=2g/VV2/r2−2g/r+βg−V2/r.Substituting Equations (12) and (18) into (11), the switch condition is obtained.


### 2.3. Control Variables

During the reentry process, the trajectory is decided by the AOAα and the bank angle σ. Commonly, the AOA profile is set as the following piecewise linear function:
(19)α=αmax,V≥V1,αbest+αmax−αbestV1−V2V−V2,V2≤V<V1,αbest,V<V2,where αmax is the max AOA limited by the aerodynamic and αbest is the best AOA when the ratio of lift-to-drag takes the maximum value, which is helpful to increase the flight range of HGV.Once the AOA profile is determined, the only control variable is the bank angleσ. According to the QEGC constraint and the three hard path constraints, the upper magnitude bound of σ can be derived as follows:
(20)σ<arccosg−V2r2mV4.3kQ2CLSmQ̇max2=σQ̇max,σ<arccosg−V2rmCLSmqmax=σqmax,σ<arccosg−V2rCD2+CL2CLgnmax=σnmax.The traditional approach to generating guidance command is to calculate the magnitude ofσ iteratively to satisfy longitudinal constraint and change the sign of σ in the horizontal plane in the heading angle corridor to fly HGV toward the target. However, the application of neural networks allows us to generate both the magnitude and sign of σ at the same time. If the output of a policy network is a∈−1,1, the command of σ is as follows:
(21)σcmd=signaσmin+aσmax−σmin,where σmax=maxσQ̇max,σqmax,σnmax, σmin=0.


## 2.1. Reentry Dynamics Equations

Considering the earth’s rotation, the dynamics equations of the hypersonic glide vehicle are given as follows:(1)V̇=−Dm−gsinγ+ω2rcosϕsinγcosϕ−cosγsinϕcosψ,γ̇=LcosσmV−gV−Vrcosγ+2ωcosϕsinψ+ω2rcosϕVcosϕcosγ+sinγsinϕcosψ,ψ̇=LsinσmVcosγ+Vrcosγsinψtanϕ−2ωtanγcosϕcosψ−sinϕ+ω2rsinϕcosϕsinψVcosγ,ṙ=Vsinγ,θ̇=Vcosγsinψrcosϕ,ϕ̇=Vcosγcosψr,where V represents the velocity, γ represents the flight path angle, ψ represents the velocity heading angle, r represents the distance between the Earth center and the vehicle, θ represents the longitude, ϕ represents the latitude, m represents the vehicle mass, g represents the gravitational acceleration, ω represents the self-rotation rate of the Earth, and σ represents the bank angle. The expressions of lift L and drag force D are
(2)D=12ρV2CDSm,L=12ρV2CLSm,where ρ represents the atmospheric density, Sm represents the reference area, and CD and CL represent the drag coefficient and lift coefficient, respectively, which can be obtained by two interpolation functions for AOA and the Mach number.ρ is a function of height H (H=r−Re, Re is the radius of the Earth):
(3)ρ=ρ0e−βH.Given an initial state, the trajectory can be obtained by integration based on Equation (1), command of AOA and bank angle.


## 2.2. Multiple Constraints

To complete the flight mission, the HGV must satisfy all the path constraints and terminal constraints.


### 2.2.1. Path Constraints

During the reentry flight, the path constraints can be expressed as(4)Q̇=kQρ0.5V3.15<Q̇max,q=12ρV2<qmax,n=D2+L2m<nmax,where Q̇ is the heating rate of the stagnation point, q is the dynamic pressure, n is the aerodynamic load, and kQ is a constant. Q̇max, qmax, and nmax denote the maximum limits, respectively.Apart from the three hard constraints, there is a soft QEGC (quasiequilibrium glide condition) constraint, which can make the reentry trajectory smooth:(5)Lcosσ−g+V2r≥0.According to Equations (2)–(5), we can obtain the three lower bounds and one upper bound of H vs. V, in which the constraint of the upper bound is soft.
(6)H≥2βlnkQρ00.5V3.15Q̇max,H≥1βlnρ0V22qmax,H≥1βlnCD2+CL2ρ0V2Sm2nmaxmg,H≤1βlnCLρ0V2Smr2mgr−V2.According to Equation (6), the H-V corridor in the glide phase is obtained. Only if the flight trajectory of HGV is limited in the corridor, the path constraints can be satisfied. Intuitively, the H-V corridor is shown in Figure 1.Figure 1
Height-velocity corridor of HGV during the reentry flight.


### 2.2.2. Terminal Constraints

Assume that HGV is required to reach the vicinity of the target at a specified height, the terminal constraints (height errorΔH, distance error Δd) can be summarized as
(7)ΔH=Hf−Hre<ΔHtd,Δd=Rearccossinϕfsinϕtar+cosϕfcosϕtarcosθf−θtar<Δdtd,where Hf, ϕf, and θf represent the final height, latitude, and longitude at the terminal time, Hre is the required height, ϕtar and θtar represent the latitude and longitude of the target, and ΔHtd and Δdtd are the thresholds of height and distance.In many practical application scenes, the terminal constraint on velocity is not so tight, which means that the velocity constraint takes the form of(8)Vf>Vre,where Vf is the terminal velocity and Vre is the required minimum velocity.Compared to traditional reentry tasks, we consider more terminal constraints in different tasks. For example, to arrive at the required timeTre, there is another constraint:
(9)ΔT=Tf−Tre<ΔTtd,where Tf is the flight time and ΔTtd is the threshold of time.After this, we give the criterion for the success of a reentry flight. In cases where all the path constraints are satisfied, if the terminal statesΔd<3×105m, ΔH<2000m, ΔT<20s, and Vf>2500m/s, then the flight mission is successful.Similarly, if the reentry task requires HGV to arrive at a desired terminal approach angle, the relevant constraint condition is adaptively formulated:(10)Δψ=ψLOS−ψapproach<Δψtd,where ψLOS is the LOS (line of sight) angle, ψapproach is the desired approach angle, and Δψtd is the angle threshold.


### 2.2.3. Switch Condition

As is shown in Figure2, HGV cannot begin sliding from an arbitrary initial status so there is a switch condition. The switch condition refers to the constraint condition at the junction of the initial descent phase and glide phase.Figure 2
The flight phases of HGV.Referring to [1], the switch condition is defined as that the absolute value of the difference between the slope of the QEGC at the current point and the actual slope of r-V is less than a small preselected value.
(11)drdV−drdVQEGC<δ0,where δ0 is a small value.The first slope can be derived from Equation (1):
(12)drdV=Vsinγ−D/m−gsinγ.The QEGC equation is rewritten as(13)ρV2CLSm2mcosσ+V2cosγr−gcosγ=0.According to Equation (13), the derivate of ρ to V is obtained:
(14)dρdV=dρdrdrdV=−βρdrdV.Both sides of Equation (13) take a derivative to velocity, and then, we can obtain
(15)ρV2CLSmcosσ2m2V−βdhdV+2Vr−V2r2dhdV+2grdhdV=0.Substituting Equation (13) into (15), then we can obtain
(16)gcosγ−V2cosγr2V−βdhdV+2Vr=drdVV2r2−2gr.Suppose thatcosγ≈1, Equation (16) is simplified as
(17)2gV=drdVV2r2−2gr+βg−V2r.Then, the second slope is obtained:(18)drdVQEGC=2g/VV2/r2−2g/r+βg−V2/r.Substituting Equations (12) and (18) into (11), the switch condition is obtained.


## 2.2.1. Path Constraints

During the reentry flight, the path constraints can be expressed as(4)Q̇=kQρ0.5V3.15<Q̇max,q=12ρV2<qmax,n=D2+L2m<nmax,where Q̇ is the heating rate of the stagnation point, q is the dynamic pressure, n is the aerodynamic load, and kQ is a constant. Q̇max, qmax, and nmax denote the maximum limits, respectively.Apart from the three hard constraints, there is a soft QEGC (quasiequilibrium glide condition) constraint, which can make the reentry trajectory smooth:(5)Lcosσ−g+V2r≥0.According to Equations (2)–(5), we can obtain the three lower bounds and one upper bound of H vs. V, in which the constraint of the upper bound is soft.
(6)H≥2βlnkQρ00.5V3.15Q̇max,H≥1βlnρ0V22qmax,H≥1βlnCD2+CL2ρ0V2Sm2nmaxmg,H≤1βlnCLρ0V2Smr2mgr−V2.According to Equation (6), the H-V corridor in the glide phase is obtained. Only if the flight trajectory of HGV is limited in the corridor, the path constraints can be satisfied. Intuitively, the H-V corridor is shown in Figure 1.Figure 1
Height-velocity corridor of HGV during the reentry flight.


## 2.2.2. Terminal Constraints

Assume that HGV is required to reach the vicinity of the target at a specified height, the terminal constraints (height errorΔH, distance error Δd) can be summarized as
(7)ΔH=Hf−Hre<ΔHtd,Δd=Rearccossinϕfsinϕtar+cosϕfcosϕtarcosθf−θtar<Δdtd,where Hf, ϕf, and θf represent the final height, latitude, and longitude at the terminal time, Hre is the required height, ϕtar and θtar represent the latitude and longitude of the target, and ΔHtd and Δdtd are the thresholds of height and distance.In many practical application scenes, the terminal constraint on velocity is not so tight, which means that the velocity constraint takes the form of(8)Vf>Vre,where Vf is the terminal velocity and Vre is the required minimum velocity.Compared to traditional reentry tasks, we consider more terminal constraints in different tasks. For example, to arrive at the required timeTre, there is another constraint:
(9)ΔT=Tf−Tre<ΔTtd,where Tf is the flight time and ΔTtd is the threshold of time.After this, we give the criterion for the success of a reentry flight. In cases where all the path constraints are satisfied, if the terminal statesΔd<3×105m, ΔH<2000m, ΔT<20s, and Vf>2500m/s, then the flight mission is successful.Similarly, if the reentry task requires HGV to arrive at a desired terminal approach angle, the relevant constraint condition is adaptively formulated:(10)Δψ=ψLOS−ψapproach<Δψtd,where ψLOS is the LOS (line of sight) angle, ψapproach is the desired approach angle, and Δψtd is the angle threshold.


## 2.2.3. Switch Condition

As is shown in Figure2, HGV cannot begin sliding from an arbitrary initial status so there is a switch condition. The switch condition refers to the constraint condition at the junction of the initial descent phase and glide phase.Figure 2
The flight phases of HGV.Referring to [1], the switch condition is defined as that the absolute value of the difference between the slope of the QEGC at the current point and the actual slope of r-V is less than a small preselected value.
(11)drdV−drdVQEGC<δ0,where δ0 is a small value.The first slope can be derived from Equation (1):
(12)drdV=Vsinγ−D/m−gsinγ.The QEGC equation is rewritten as(13)ρV2CLSm2mcosσ+V2cosγr−gcosγ=0.According to Equation (13), the derivate of ρ to V is obtained:
(14)dρdV=dρdrdrdV=−βρdrdV.Both sides of Equation (13) take a derivative to velocity, and then, we can obtain
(15)ρV2CLSmcosσ2m2V−βdhdV+2Vr−V2r2dhdV+2grdhdV=0.Substituting Equation (13) into (15), then we can obtain
(16)gcosγ−V2cosγr2V−βdhdV+2Vr=drdVV2r2−2gr.Suppose thatcosγ≈1, Equation (16) is simplified as
(17)2gV=drdVV2r2−2gr+βg−V2r.Then, the second slope is obtained:(18)drdVQEGC=2g/VV2/r2−2g/r+βg−V2/r.Substituting Equations (12) and (18) into (11), the switch condition is obtained.


## 2.3. Control Variables

During the reentry process, the trajectory is decided by the AOAα and the bank angle σ. Commonly, the AOA profile is set as the following piecewise linear function:
(19)α=αmax,V≥V1,αbest+αmax−αbestV1−V2V−V2,V2≤V<V1,αbest,V<V2,where αmax is the max AOA limited by the aerodynamic and αbest is the best AOA when the ratio of lift-to-drag takes the maximum value, which is helpful to increase the flight range of HGV.Once the AOA profile is determined, the only control variable is the bank angleσ. According to the QEGC constraint and the three hard path constraints, the upper magnitude bound of σ can be derived as follows:
(20)σ<arccosg−V2r2mV4.3kQ2CLSmQ̇max2=σQ̇max,σ<arccosg−V2rmCLSmqmax=σqmax,σ<arccosg−V2rCD2+CL2CLgnmax=σnmax.The traditional approach to generating guidance command is to calculate the magnitude ofσ iteratively to satisfy longitudinal constraint and change the sign of σ in the horizontal plane in the heading angle corridor to fly HGV toward the target. However, the application of neural networks allows us to generate both the magnitude and sign of σ at the same time. If the output of a policy network is a∈−1,1, the command of σ is as follows:
(21)σcmd=signaσmin+aσmax−σmin,where σmax=maxσQ̇max,σqmax,σnmax, σmin=0.


## 3. Deep Deterministic Policy Gradient with Hindsight Experience Replay

### 3.1. Deep Reinforcement Learning

Based on Markov Decision Process (MDP), standard reinforcement learning consists of an environment, an agent, and their interactions. The environment is initiated as a states1∈S, which can describe its initial features. The agent observes the state of the environment and chooses an action a1∈A. The action affects the environment to transfer to the next state s2 with a certain probability p1 and generates a reward r1∈R to the agent. Then, the agent interacts with the environment repeatedly, so there is a sequence of tuples <si,ai,pi,si+1,ri>. The agent’s goal is to increase its accumulated rewards.
(22)Gt=∑i=t∞λiri,where λ is the discount factor, which is used to balance the current and future reward.Action-value functionQπst,at is set to evaluate how good the action at is: Qπst,at=EGtst,at. And if the policy π is optimal, Qπst,at is the optimal action-value function, which satisfies the Bellman equation:
(23)Q∗s,a=Ers,a+λmaxa′∈AQ∗s′,a′.In DRL, if the policyπ is deterministic, it can be considered as a mapping from states to actions, commonly in a form of neural network with the parameter θπ.


### 3.2. Deep Deterministic Policy Gradient (DDPG)

The baseline algorithm used for the training of neural networks is Deep Deterministic Policy Gradient (DDPG), which belongs to the family of model-free policy gradient DRL methods. There are four neural networks in DDPG, which are called the online actor networkπsθπ, the target actor network π′sθπ′, the online critic network Qs,a∣θQ, and the target critic network Q′s,a∣θQ′. The online actor network parameterized by θπ exports actions of the agent, and the online critic network parameterized by θQ is designed to approximate action-value function Qs,a. The actual behavior policy a of the agent is presented from π with an added noise N.The parameterθQ is optimized by decreasing the loss:
(24)LθQ=Erst,at+λQ′st+1,π′st+1θπ′θQ′−Qst,atθQ2.The actorπsθπ is updated by the theory of policy gradient:
(25)∇θπJ=Est~ξ∇θπQs,aθQs=st,a=πst,θπ=Est~ξ∇aQs,aθQ∇θππsθπs=st,where ξ is the distribution of the state.The existence of the target actor network and the target critic network can make the process of training more stable. Besides, the data is stored in a replay buffer to reduce the relevance between experiences and deal with the experiences effectively.


### 3.3. Hindsight Experience Replay (HER)

The key idea behind Hindsight Experience Replay (HER) is derived from universal value function approximators (UVFA) [43]. In UVFA, there is a goal space G. For any goal g∈G, there is a pseudoreward Rgs,a,s′:S×A⟶R. The first state-goal pair in every episode is sampled from a distribution ps0,g, and in the same episode, the goal g is fixed. At every step in an episode, the agent gets an action from the current state and goal π:S×G⟶A. On this premise, the traditional Q-function is expanded to Qst,at,g=ERt∣st,at,g.In a task where the agent needs to achieve multiple goals, the designed reward is quite likely to be sparse. To deal with sparse rewards, an effective trick is to exploit the failed data samples too. In the method of HER, the states of the agent are expanded to combinations of states and goals. The essence of HER rests with designing a virtual goalg′, which is achieved in a failed trajectory. For example, there is an RL task in Figure 3, where the agent has a goal position. Only if the agent arrives at the goal gM,1 it gets a reward 1, otherwise 0. In most episodes, the agent gets 0 rewards. It is quite difficult for conventional RL algorithms to achieve good results for the sparse reward problem especially when M and N are large numbers. In the HER method, if the final position is 3,N, we can define the position as a virtual goal g′, and then, the final reward Rt∣st,at,g=0, but Rt∣st,at,g′=1. By doing this, the algorithm can learn from failed experiences, and the reward is changed from sparse to dense.Figure 3
Simple example of HER.With the help of virtual goals, the efficiency of experience is improved, which makes the convergence of neural networks significantly ameliorated.


## 3.1. Deep Reinforcement Learning

Based on Markov Decision Process (MDP), standard reinforcement learning consists of an environment, an agent, and their interactions. The environment is initiated as a states1∈S, which can describe its initial features. The agent observes the state of the environment and chooses an action a1∈A. The action affects the environment to transfer to the next state s2 with a certain probability p1 and generates a reward r1∈R to the agent. Then, the agent interacts with the environment repeatedly, so there is a sequence of tuples <si,ai,pi,si+1,ri>. The agent’s goal is to increase its accumulated rewards.
(22)Gt=∑i=t∞λiri,where λ is the discount factor, which is used to balance the current and future reward.Action-value functionQπst,at is set to evaluate how good the action at is: Qπst,at=EGtst,at. And if the policy π is optimal, Qπst,at is the optimal action-value function, which satisfies the Bellman equation:
(23)Q∗s,a=Ers,a+λmaxa′∈AQ∗s′,a′.In DRL, if the policyπ is deterministic, it can be considered as a mapping from states to actions, commonly in a form of neural network with the parameter θπ.


## 3.2. Deep Deterministic Policy Gradient (DDPG)

The baseline algorithm used for the training of neural networks is Deep Deterministic Policy Gradient (DDPG), which belongs to the family of model-free policy gradient DRL methods. There are four neural networks in DDPG, which are called the online actor networkπsθπ, the target actor network π′sθπ′, the online critic network Qs,a∣θQ, and the target critic network Q′s,a∣θQ′. The online actor network parameterized by θπ exports actions of the agent, and the online critic network parameterized by θQ is designed to approximate action-value function Qs,a. The actual behavior policy a of the agent is presented from π with an added noise N.The parameterθQ is optimized by decreasing the loss:
(24)LθQ=Erst,at+λQ′st+1,π′st+1θπ′θQ′−Qst,atθQ2.The actorπsθπ is updated by the theory of policy gradient:
(25)∇θπJ=Est~ξ∇θπQs,aθQs=st,a=πst,θπ=Est~ξ∇aQs,aθQ∇θππsθπs=st,where ξ is the distribution of the state.The existence of the target actor network and the target critic network can make the process of training more stable. Besides, the data is stored in a replay buffer to reduce the relevance between experiences and deal with the experiences effectively.


## 3.3. Hindsight Experience Replay (HER)

The key idea behind Hindsight Experience Replay (HER) is derived from universal value function approximators (UVFA) [43]. In UVFA, there is a goal space G. For any goal g∈G, there is a pseudoreward Rgs,a,s′:S×A⟶R. The first state-goal pair in every episode is sampled from a distribution ps0,g, and in the same episode, the goal g is fixed. At every step in an episode, the agent gets an action from the current state and goal π:S×G⟶A. On this premise, the traditional Q-function is expanded to Qst,at,g=ERt∣st,at,g.In a task where the agent needs to achieve multiple goals, the designed reward is quite likely to be sparse. To deal with sparse rewards, an effective trick is to exploit the failed data samples too. In the method of HER, the states of the agent are expanded to combinations of states and goals. The essence of HER rests with designing a virtual goalg′, which is achieved in a failed trajectory. For example, there is an RL task in Figure 3, where the agent has a goal position. Only if the agent arrives at the goal gM,1 it gets a reward 1, otherwise 0. In most episodes, the agent gets 0 rewards. It is quite difficult for conventional RL algorithms to achieve good results for the sparse reward problem especially when M and N are large numbers. In the HER method, if the final position is 3,N, we can define the position as a virtual goal g′, and then, the final reward Rt∣st,at,g=0, but Rt∣st,at,g′=1. By doing this, the algorithm can learn from failed experiences, and the reward is changed from sparse to dense.Figure 3
Simple example of HER.With the help of virtual goals, the efficiency of experience is improved, which makes the convergence of neural networks significantly ameliorated.


## 4. Design of Intelligent Guidance and Training of Neural Networks

### 4.1. Overall Design

In this section, the HER-DDPG-based guidance is proposed. First of all, we normalize the reentry simulation into two segments (scenario initialization and guidance step) and open two corresponding interfaces to the intelligent algorithm. In the segment of scenario initialization, the position and velocity of HGV are randomly initialized within a certain range. The simulation starts from the specified point at the initial phase and continues to the switch point decided by Equation (11). After entering the glide phase, at each decision step later, HGV receives the guidance command of bank angle via the interface to the intelligent algorithm and calculates AOA according to the profile given by Equation (19). The simulation proceeds until distance error Δd<Δdtd or the velocity Vf<Vre or any path constraint is not satisfied.After constructing the two segments, the transformation from simulation to MDP is accomplished. The kinematical parameters of HGV are mapped to states, and the guidance command of bank angle is designed as the action. Based on the theory of HER, the terminal constraints are transformed into multigoals and the expanded states are defined. The reward function is designed as a combination of several linear functions of terminal height, terminal distance error, and flight time error, which is used for generating dense signals of policy gradient. Finally, based on HER and DDPG, the algorithm goes into operation as is shown in Figure4. When the neural networks converge, the training process is terminated.Figure 4
The overall design of the algorithm.


### 4.2. Markov Decision Process Variables

The fundamental variables in MDP are defined as follows.


#### 4.2.1. States

The basic state of the HGV agent is defined ass=V,γ,ψ,r,θ,ϕ, which can uniquely express the features of the motion.


#### 4.2.2. Goals

Goals describe the terminal constraints of HGV with a certain fixed tolerance of position and time. The initial goal is defined asg=θtar,ϕtar,Hre,Tre. In this paper, Δdtd=300km, if the final distance error Δd<300km then the terminal distance constraint is satisfied. If the loose constraint Δd<ΔdHER=2000km, we decide that the HER condition is met.The HER condition is shown in Figure5, which means that despite its failure of arriving at the given target, HGV has arrived at the terminal position successfully. Here, we define θtarHER, ϕtarHER, HreHER, and TreHER as target longitude, target latitude, required height, and required time on the HER condition. The intention of HER is to efficiently increase the sampled data of failed episodes, which means that they may be successful in other target tasks. Therefore, the required height and time are modified as the actual values at the terminal time when the errors concerning initial goals are in a specifically designed range. The concrete values of them in this paper are given as
(26)θtarHER=θf,ϕtarHER=ϕfHreHER=Hf,ΔH≤8km,HreHER=Hre,ΔH>8km,TreHER=Tf,ΔT≤200s,TreHER=Tre,ΔT>200s.Figure 5
HER condition of HGV.Then, the new goal is defined asgHER=θtarHER,ϕtarHER,HreHER,TreHER.In this case, the new terminal errors can be expressed as(27)ΔHHER=Hf−HreHER,ΔdHER=RearccossinϕfsinϕtarHER+cosϕfcosϕtarHERcosθf−θtarHER,ΔTHER=Tf−TtarHER.By doing this, lots of failed data is transformed into successful data with new terminal constraints. The reward signal is more abundant in the training process, and the average reward can increase. In DRL training, the more abundant the reward signal, the faster the rate of convergence.


#### 4.2.3. Expanded States

The expanded state can be defined as(28)se=sg=V,γ,ψ,r,θ,ϕ,θtar,ϕtar,Hre,Tre.On the HER condition, the expanded state is modified accordingly:(29)seHER=sgHER=V,γ,ψ,r,θ,ϕ,θtarHER,ϕtarHER,HreHER,TreHER.


#### 4.2.4. Rewards

In the paper where HER is proposed [44], the success of the task is merely relative to the position of a robotic arm, so it is feasible to learn from rewards that are sparse and binary. However, the number of terminal constraints in the reentry task is so up to 4 that the binary reward is no longer suitable. For the reentry task, the priorities of satisfying different terminal constraints are not at the same level. Intuitively, the range constraint is the most important, for it requires the agent to adjust both the magnitude and the sign of the bank angle to arrive in the vicinity of the target. Secondly, the height constraint is also important. After the two constraints are under consideration, the time-related reward is added. Considering the following dive phase after the glide phase, the constraint thresholds are defined: Δdtd=3×105m, ΔHtd=2000m, and ΔTtd=20s.Piecewise linear functions are applied to design the range-relative rewardRR, height-relative reward RH, and time-relative reward RT:
(30)RR=0+0−−903−70×105Δd−3×105,Δd>3×105,0,Δd≤3×105,(31)RH=60+60−100−2000ΔH−0,ΔH≤2000,10+10−52000−5000ΔH−2000,2000<ΔH≤5000,5+5−05000−10000ΔH−5000,5000<ΔH≤10000,0,ΔH>10000,(32)RT=RH=80+80−00−20ΔT−0,ΔT≤20,0+0−−1020−100ΔT−20,20<ΔT≤100,−10+−10−−12100−500ΔT−100,100<ΔT≤500,−12+−12−−20500−10000ΔT−10000,ΔT>500,The 3 subrewards are shown in Figure6.Figure 6
The curves of subrewards.Then, the total rewardR is designed as follows:
(33)R=RR,Δd>3×105,RR+RH,Δd≤3×105,ΔH≥2000,RR+RH+RT,Δd<3×105,ΔH<2000.RR is designed to reduce the terminal distance error in the range of Δdtd, which means that if the distance error Δd>300km then the subreward RR<0, and if Δd≤300km, then the total reward is determined by RH and RT. RH is designed to reduce the terminal height error, so the smaller the ΔH, the larger the RH. And in the whole training process, the number of samples with larger ΔH is more than smaller ΔH, so as ΔH decreases, the slopes of piecewise linear functions increase. Finally, only if the height error ΔH<ΔHtd is the subreward RT added in R. The design of RT is similar, so the smaller the ΔT, the larger the RT. And as ΔT decreases, the slopes of piecewise linear functions increase too.If the terminal statesΔd<3×105m, ΔH<2000m, ΔT<20s, and Vf>2500m/s, then the flight mission is successful. According to Equations (30)–(33), the reward of a successful episode is in the range of 10~140. For some successful episodes, the average reward value will be in the range of 10 to 140, which depends on the final data distribution of ΔH and ΔT.


#### 4.2.5. Actions

Different from the widely used decoupling methods in current guidance algorithms, the magnitude and sign command of the bank angle are provided at the same time in this paper. Commonly actionA is calculated by a policy network. In DDPG, the output of the policy network is a continuous value A∈−1,1. The action is mapped to the bank angle command in the form of Equation (21).


### 4.3. Algorithm Flow for Training

Based on HER, the training algorithm of multiconstrained reentry guidance is given in Algorithm1.Algorithm 1: Training of multiconstrained reentry guidance based on HER.
Randomly initialize parameters of actor networkπsθπ and critic network Qs,a∣θQ.Initialize target actorπ′sθπ′ and target critic Q′s,a∣θQ′ with θπ′⟵θπ, θQ′⟵θQ.Initialize basic replay bufferRBforepisode=1, NedoInitialize an HER replay bufferRHER and a random process noise N for explorationInitialize an initial goalg=θtar,ϕtar,Hre,Tre randomlyRun the Scenario Initialization Function, sample a basic states0=V,γ,ψ,r,θ,ϕfort=0, NTdoCombine basic state and goal to expanded stateset=stgSample an action from actor and noise:at=πsetθπ+NtExecute the action in the Policy Step Function and observe a new statest+1Combine basic new state and goal to expand new stateset+1=st+1gStore the transitionset,at,rt,set+1 in RB and RHERif the episode is doneJudge whether the HER condition is met and record itend ifend forif the HER condition is metfort=0, NT in RHERdoCalculategHER=θtarHER,ϕtarHER,HreHER,TreHER and ΔHHER, ΔdHER, ΔTHERRecalculate rewardrtHER according to Equations (30)–(33)Combine basic states and goal to expand states:setHER=setgHER,set+1HER=set+1gHERStore the transitionsetHER,at,rtHER,set+1HER in RBend forClear data inRHERend iffort=0, NtraindoSample a minibatchB from the replay buffer RBUpdate critic by Equation (24), update actor by Equation (25)Update target network periodically:θπ′⟵τθπ+1−τθπ′, θQ′⟵τθQ+1−τθQ′end forend for


### 4.4. Neural Network Design

The designed actor network model and critic network model are shown in Figures7 and 8. And the details of every layer are listed in Tables 1 and 2.Figure 7
The architecture of the actor network.Figure 8
The architecture of the critic network.Table 1
The architecture of the actor network.
NumberLayer typeLayer nodesActivation function1Dense10×300Leaky ReLU2Dense300×600Leaky RELU3Batch normalization600—4Dense600×400Leaky ReLU5Dense400×1—6Batch normalization1TanhTable 2
The architecture of the critic network.
NumberLayer typeLayer nodesActivation function1Dense11×300Leaky ReLU2Dense300×600Leaky RELU3Batch normalization600—4Dense600×500Leaky ReLU5Dense500×1Tanh


## 4.1. Overall Design

In this section, the HER-DDPG-based guidance is proposed. First of all, we normalize the reentry simulation into two segments (scenario initialization and guidance step) and open two corresponding interfaces to the intelligent algorithm. In the segment of scenario initialization, the position and velocity of HGV are randomly initialized within a certain range. The simulation starts from the specified point at the initial phase and continues to the switch point decided by Equation (11). After entering the glide phase, at each decision step later, HGV receives the guidance command of bank angle via the interface to the intelligent algorithm and calculates AOA according to the profile given by Equation (19). The simulation proceeds until distance error Δd<Δdtd or the velocity Vf<Vre or any path constraint is not satisfied.After constructing the two segments, the transformation from simulation to MDP is accomplished. The kinematical parameters of HGV are mapped to states, and the guidance command of bank angle is designed as the action. Based on the theory of HER, the terminal constraints are transformed into multigoals and the expanded states are defined. The reward function is designed as a combination of several linear functions of terminal height, terminal distance error, and flight time error, which is used for generating dense signals of policy gradient. Finally, based on HER and DDPG, the algorithm goes into operation as is shown in Figure4. When the neural networks converge, the training process is terminated.Figure 4
The overall design of the algorithm.


## 4.2. Markov Decision Process Variables

The fundamental variables in MDP are defined as follows.


### 4.2.1. States

The basic state of the HGV agent is defined ass=V,γ,ψ,r,θ,ϕ, which can uniquely express the features of the motion.


### 4.2.2. Goals

Goals describe the terminal constraints of HGV with a certain fixed tolerance of position and time. The initial goal is defined asg=θtar,ϕtar,Hre,Tre. In this paper, Δdtd=300km, if the final distance error Δd<300km then the terminal distance constraint is satisfied. If the loose constraint Δd<ΔdHER=2000km, we decide that the HER condition is met.The HER condition is shown in Figure5, which means that despite its failure of arriving at the given target, HGV has arrived at the terminal position successfully. Here, we define θtarHER, ϕtarHER, HreHER, and TreHER as target longitude, target latitude, required height, and required time on the HER condition. The intention of HER is to efficiently increase the sampled data of failed episodes, which means that they may be successful in other target tasks. Therefore, the required height and time are modified as the actual values at the terminal time when the errors concerning initial goals are in a specifically designed range. The concrete values of them in this paper are given as
(26)θtarHER=θf,ϕtarHER=ϕfHreHER=Hf,ΔH≤8km,HreHER=Hre,ΔH>8km,TreHER=Tf,ΔT≤200s,TreHER=Tre,ΔT>200s.Figure 5
HER condition of HGV.Then, the new goal is defined asgHER=θtarHER,ϕtarHER,HreHER,TreHER.In this case, the new terminal errors can be expressed as(27)ΔHHER=Hf−HreHER,ΔdHER=RearccossinϕfsinϕtarHER+cosϕfcosϕtarHERcosθf−θtarHER,ΔTHER=Tf−TtarHER.By doing this, lots of failed data is transformed into successful data with new terminal constraints. The reward signal is more abundant in the training process, and the average reward can increase. In DRL training, the more abundant the reward signal, the faster the rate of convergence.


### 4.2.3. Expanded States

The expanded state can be defined as(28)se=sg=V,γ,ψ,r,θ,ϕ,θtar,ϕtar,Hre,Tre.On the HER condition, the expanded state is modified accordingly:(29)seHER=sgHER=V,γ,ψ,r,θ,ϕ,θtarHER,ϕtarHER,HreHER,TreHER.


### 4.2.4. Rewards

In the paper where HER is proposed [44], the success of the task is merely relative to the position of a robotic arm, so it is feasible to learn from rewards that are sparse and binary. However, the number of terminal constraints in the reentry task is so up to 4 that the binary reward is no longer suitable. For the reentry task, the priorities of satisfying different terminal constraints are not at the same level. Intuitively, the range constraint is the most important, for it requires the agent to adjust both the magnitude and the sign of the bank angle to arrive in the vicinity of the target. Secondly, the height constraint is also important. After the two constraints are under consideration, the time-related reward is added. Considering the following dive phase after the glide phase, the constraint thresholds are defined: Δdtd=3×105m, ΔHtd=2000m, and ΔTtd=20s.Piecewise linear functions are applied to design the range-relative rewardRR, height-relative reward RH, and time-relative reward RT:
(30)RR=0+0−−903−70×105Δd−3×105,Δd>3×105,0,Δd≤3×105,(31)RH=60+60−100−2000ΔH−0,ΔH≤2000,10+10−52000−5000ΔH−2000,2000<ΔH≤5000,5+5−05000−10000ΔH−5000,5000<ΔH≤10000,0,ΔH>10000,(32)RT=RH=80+80−00−20ΔT−0,ΔT≤20,0+0−−1020−100ΔT−20,20<ΔT≤100,−10+−10−−12100−500ΔT−100,100<ΔT≤500,−12+−12−−20500−10000ΔT−10000,ΔT>500,The 3 subrewards are shown in Figure6.Figure 6
The curves of subrewards.Then, the total rewardR is designed as follows:
(33)R=RR,Δd>3×105,RR+RH,Δd≤3×105,ΔH≥2000,RR+RH+RT,Δd<3×105,ΔH<2000.RR is designed to reduce the terminal distance error in the range of Δdtd, which means that if the distance error Δd>300km then the subreward RR<0, and if Δd≤300km, then the total reward is determined by RH and RT. RH is designed to reduce the terminal height error, so the smaller the ΔH, the larger the RH. And in the whole training process, the number of samples with larger ΔH is more than smaller ΔH, so as ΔH decreases, the slopes of piecewise linear functions increase. Finally, only if the height error ΔH<ΔHtd is the subreward RT added in R. The design of RT is similar, so the smaller the ΔT, the larger the RT. And as ΔT decreases, the slopes of piecewise linear functions increase too.If the terminal statesΔd<3×105m, ΔH<2000m, ΔT<20s, and Vf>2500m/s, then the flight mission is successful. According to Equations (30)–(33), the reward of a successful episode is in the range of 10~140. For some successful episodes, the average reward value will be in the range of 10 to 140, which depends on the final data distribution of ΔH and ΔT.


### 4.2.5. Actions

Different from the widely used decoupling methods in current guidance algorithms, the magnitude and sign command of the bank angle are provided at the same time in this paper. Commonly actionA is calculated by a policy network. In DDPG, the output of the policy network is a continuous value A∈−1,1. The action is mapped to the bank angle command in the form of Equation (21).


## 4.2.1. States

The basic state of the HGV agent is defined ass=V,γ,ψ,r,θ,ϕ, which can uniquely express the features of the motion.


## 4.2.2. Goals

Goals describe the terminal constraints of HGV with a certain fixed tolerance of position and time. The initial goal is defined asg=θtar,ϕtar,Hre,Tre. In this paper, Δdtd=300km, if the final distance error Δd<300km then the terminal distance constraint is satisfied. If the loose constraint Δd<ΔdHER=2000km, we decide that the HER condition is met.The HER condition is shown in Figure5, which means that despite its failure of arriving at the given target, HGV has arrived at the terminal position successfully. Here, we define θtarHER, ϕtarHER, HreHER, and TreHER as target longitude, target latitude, required height, and required time on the HER condition. The intention of HER is to efficiently increase the sampled data of failed episodes, which means that they may be successful in other target tasks. Therefore, the required height and time are modified as the actual values at the terminal time when the errors concerning initial goals are in a specifically designed range. The concrete values of them in this paper are given as
(26)θtarHER=θf,ϕtarHER=ϕfHreHER=Hf,ΔH≤8km,HreHER=Hre,ΔH>8km,TreHER=Tf,ΔT≤200s,TreHER=Tre,ΔT>200s.Figure 5
HER condition of HGV.Then, the new goal is defined asgHER=θtarHER,ϕtarHER,HreHER,TreHER.In this case, the new terminal errors can be expressed as(27)ΔHHER=Hf−HreHER,ΔdHER=RearccossinϕfsinϕtarHER+cosϕfcosϕtarHERcosθf−θtarHER,ΔTHER=Tf−TtarHER.By doing this, lots of failed data is transformed into successful data with new terminal constraints. The reward signal is more abundant in the training process, and the average reward can increase. In DRL training, the more abundant the reward signal, the faster the rate of convergence.


## 4.2.3. Expanded States

The expanded state can be defined as(28)se=sg=V,γ,ψ,r,θ,ϕ,θtar,ϕtar,Hre,Tre.On the HER condition, the expanded state is modified accordingly:(29)seHER=sgHER=V,γ,ψ,r,θ,ϕ,θtarHER,ϕtarHER,HreHER,TreHER.


## 4.2.4. Rewards

In the paper where HER is proposed [44], the success of the task is merely relative to the position of a robotic arm, so it is feasible to learn from rewards that are sparse and binary. However, the number of terminal constraints in the reentry task is so up to 4 that the binary reward is no longer suitable. For the reentry task, the priorities of satisfying different terminal constraints are not at the same level. Intuitively, the range constraint is the most important, for it requires the agent to adjust both the magnitude and the sign of the bank angle to arrive in the vicinity of the target. Secondly, the height constraint is also important. After the two constraints are under consideration, the time-related reward is added. Considering the following dive phase after the glide phase, the constraint thresholds are defined: Δdtd=3×105m, ΔHtd=2000m, and ΔTtd=20s.Piecewise linear functions are applied to design the range-relative rewardRR, height-relative reward RH, and time-relative reward RT:
(30)RR=0+0−−903−70×105Δd−3×105,Δd>3×105,0,Δd≤3×105,(31)RH=60+60−100−2000ΔH−0,ΔH≤2000,10+10−52000−5000ΔH−2000,2000<ΔH≤5000,5+5−05000−10000ΔH−5000,5000<ΔH≤10000,0,ΔH>10000,(32)RT=RH=80+80−00−20ΔT−0,ΔT≤20,0+0−−1020−100ΔT−20,20<ΔT≤100,−10+−10−−12100−500ΔT−100,100<ΔT≤500,−12+−12−−20500−10000ΔT−10000,ΔT>500,The 3 subrewards are shown in Figure6.Figure 6
The curves of subrewards.Then, the total rewardR is designed as follows:
(33)R=RR,Δd>3×105,RR+RH,Δd≤3×105,ΔH≥2000,RR+RH+RT,Δd<3×105,ΔH<2000.RR is designed to reduce the terminal distance error in the range of Δdtd, which means that if the distance error Δd>300km then the subreward RR<0, and if Δd≤300km, then the total reward is determined by RH and RT. RH is designed to reduce the terminal height error, so the smaller the ΔH, the larger the RH. And in the whole training process, the number of samples with larger ΔH is more than smaller ΔH, so as ΔH decreases, the slopes of piecewise linear functions increase. Finally, only if the height error ΔH<ΔHtd is the subreward RT added in R. The design of RT is similar, so the smaller the ΔT, the larger the RT. And as ΔT decreases, the slopes of piecewise linear functions increase too.If the terminal statesΔd<3×105m, ΔH<2000m, ΔT<20s, and Vf>2500m/s, then the flight mission is successful. According to Equations (30)–(33), the reward of a successful episode is in the range of 10~140. For some successful episodes, the average reward value will be in the range of 10 to 140, which depends on the final data distribution of ΔH and ΔT.


## 4.2.5. Actions

Different from the widely used decoupling methods in current guidance algorithms, the magnitude and sign command of the bank angle are provided at the same time in this paper. Commonly actionA is calculated by a policy network. In DDPG, the output of the policy network is a continuous value A∈−1,1. The action is mapped to the bank angle command in the form of Equation (21).


## 4.3. Algorithm Flow for Training

Based on HER, the training algorithm of multiconstrained reentry guidance is given in Algorithm1.Algorithm 1: Training of multiconstrained reentry guidance based on HER.
Randomly initialize parameters of actor networkπsθπ and critic network Qs,a∣θQ.Initialize target actorπ′sθπ′ and target critic Q′s,a∣θQ′ with θπ′⟵θπ, θQ′⟵θQ.Initialize basic replay bufferRBforepisode=1, NedoInitialize an HER replay bufferRHER and a random process noise N for explorationInitialize an initial goalg=θtar,ϕtar,Hre,Tre randomlyRun the Scenario Initialization Function, sample a basic states0=V,γ,ψ,r,θ,ϕfort=0, NTdoCombine basic state and goal to expanded stateset=stgSample an action from actor and noise:at=πsetθπ+NtExecute the action in the Policy Step Function and observe a new statest+1Combine basic new state and goal to expand new stateset+1=st+1gStore the transitionset,at,rt,set+1 in RB and RHERif the episode is doneJudge whether the HER condition is met and record itend ifend forif the HER condition is metfort=0, NT in RHERdoCalculategHER=θtarHER,ϕtarHER,HreHER,TreHER and ΔHHER, ΔdHER, ΔTHERRecalculate rewardrtHER according to Equations (30)–(33)Combine basic states and goal to expand states:setHER=setgHER,set+1HER=set+1gHERStore the transitionsetHER,at,rtHER,set+1HER in RBend forClear data inRHERend iffort=0, NtraindoSample a minibatchB from the replay buffer RBUpdate critic by Equation (24), update actor by Equation (25)Update target network periodically:θπ′⟵τθπ+1−τθπ′, θQ′⟵τθQ+1−τθQ′end forend for


## 4.4. Neural Network Design

The designed actor network model and critic network model are shown in Figures7 and 8. And the details of every layer are listed in Tables 1 and 2.Figure 7
The architecture of the actor network.Figure 8
The architecture of the critic network.Table 1
The architecture of the actor network.
NumberLayer typeLayer nodesActivation function1Dense10×300Leaky ReLU2Dense300×600Leaky RELU3Batch normalization600—4Dense600×400Leaky ReLU5Dense400×1—6Batch normalization1TanhTable 2
The architecture of the critic network.
NumberLayer typeLayer nodesActivation function1Dense11×300Leaky ReLU2Dense300×600Leaky RELU3Batch normalization600—4Dense600×500Leaky ReLU5Dense500×1Tanh


## 5. Simulations and Analysis

### 5.1. Simulation Settings

The HGV model used in this paper is the common aero vehicle (CAV-H). In Equation (1), m=907kg and g=9.8066m/s2. In Equation (2), Sm=0.4839m2. In Equation (3), ρ0=1.2250kg/m3, β=0.00014065, and Re=6378004m. In Equation (4), kQ=5×10−5, qmax=100kPa, nmax=3, and Q̇max=2000 kW/m2. In Equation (8), Vre=2500m/s. In Equation (19), αmax=20deg, αbest=10deg, V1=6000m/s, and V2=5000m/s. The initial longitude and latitude of HGV are 0 deg. The size of the integral step is 0.01 s, and the size of the guidance (policy) step is 150 s. All used random variables in Table 3 are subject to the uniform distribution.Table 3
Parameters of initial reentry point and target point.
Initial parametersValueInitial heightH060~64 kmInitial velocityV06200~6800 m/sInitial path angleσ0−0.02~−0.01 degTarget longitudeθtar60~65 degTarget latitudeϕtar−7~7 degRequired heightHre26~32 kmRequired timeTre1200~1500 sThe error of drag coefficientΔCD0~10%The error of lift coefficientΔCL0~10%After setting the above parameters, the networks are trained by Algorithm1.


### 5.2. Training Process of Neural Networks

Comparative experiments are made at the same time to verify the superiority of DDPG+HER method. DDPG algorithm [36] and Proximal Policy Optimization (PPO) algorithm [37] are selected as control groups. The hyperparameters of the three algorithms used in the training process are set in Table 4.Table 4
The hyperparameters in the training.
HyperparameterDDPGPPODDPG+HERDiscount factorλ0.990.990.99Batch size646464Replay buffer size20000—20000Actor learning rate10−410−310−4Critic learning rate10−310−310−3Target update rateτ0.001—0.001Maximum number of steps100010001000Exploration policyOU—N0,0.1GAE factor—0.98—Clip factor—0.2—After the training, the average returns and success count in the latest 100 episodes are shown in Figures9 and 10. The total episode number is 23161.Figure 9
Contrast curves of the average return in the latest 100 episodes.Figure 10
Contrast curves of the success count in the latest 100 episodes.It can be seen from Figure9 that the average return of the DDPG+HER method begins rising first and gets more rewards than other methods. The sparse reward problem is effectively alleviated. At the end time of training, the success rate of the DDPG+HER method in Figure 10 converges to 100%, which implies that the policy given by the actor is operative. The best success count of DDPG is less than 75 and the best success count of PPO is less than 60, which powerfully illustrates that the proposed DDPG+HER method is more effective in the training of the policy network.In Figure11, the loss of actor and critic in the DDPG+HER method is given.Figure 11
The loss of actor and critic in DDPG+HER.It can be seen from Figure11 that the loss of the critic is decreasing to zero along with the progress of training. Meanwhile, the magnitude of actor loss is increasing in the process, which means that the score of the agent is on a trend of improvement.


### 5.3. Random Single Trajectory Analysis

After the convergence of neural networks, the typical simulation is performed randomly. To evaluate the performance of the proposed guidance, a contrast simulation based on NPCG is also performed. The proposed method in this paper is called HERG for short.The terminal constraints are set as target longitudeλtar=63.505deg, latitude ϕtar=3.976deg, required height Hre=30.429km, and required time Tre=1370.0s. The errors of aerodynamics are set as ΔCD=3.5% and ΔCL=4.5%. For the trajectory of HERG, the terminal height is 30.755 km, and the terminal time is 1359.7 s. For the trajectory of NPCG, the terminal height is 32.916 km, and the terminal time is 1386.6 s.The results are shown in Figures12–14.Figure 12
Reentry trajectory of HGV.Figure 13
State-time curves.Figure 14
Path constraint-time curves.Figure12 shows that the HGV can arrive at the required target area. Figure 13 shows that the policy decision is executed 9 times, and terminal height and time are in the required range. The reverse number of the bank angle is reduced greatly for HERG than NPCG. And as is seen in Figure 14, all of the path constraints are satisfied.The simulations are performed in a system with a 12th Gen Intel Core i5-12600KF CPU and 16 GB RAM. In the NPCG method, the corrector works a total of 9 times. As the time of integration decreases, the computing time decreases. The computing time of NPCG is listed in Table5, and the total computing time is 6.431 s. In the HERG method, the policy network is called 9 times and the total computing time is 0.0937 s.Table 5
Computing time of NPCG.
Order123456789Time (s)3.3872.7962.4112.2202.0021.3971.0720.6280.291The contrast simulation results are shown in Table6. It is obvious that with similar terminal errors, the calculation time of the proposed method is less than 1/68 of NPCG. By contrast with NPCG, the proposed method makes a breakthrough in real-time performance.Table 6
Contrast simulation results.
MethodComputing time (s)Range error (km)Height error (km)Time error (s)NPCG6.431299.9982.48716.6HERG0.0937299.9990.32610.3


### 5.4. Verifications on Monte Carlo Simulations

To evaluate the effectiveness of the proposed guidance, 1000 Monte Carlo simulations are performed, whose initial states and errors of aerodynamic coefficients are set according to Table3. The results are shown in Figures 15–22.Figure 15
Height-velocity curve.Figure 16
Latitude-longitude curve.Figure 17
AOA-time curve.Figure 18
Bank angle-time curve.Figure 19
Heating rate-time curve.Figure 20
Dynamic pressure-time curve.Figure 21
Longitudinal overload-time curve.Figure 22
Lateral overload-time curve.As is seen in Figure15, all of the 1000 trajectories of the glide phase are in the range of the H-V corridor, and naturally, in Figures 16–19, all of the path constraints are satisfied. Moreover, the AOA and bank angle in different trajectories take on a continuous trend as is shown in Figures 17 and 18, which indicates that the output of the policy network is steady and robust.The statistical data of terminal constraints is shown in Table7.Table 7
Terminal constraints of the Monte Carlo simulations.
Terminal constraintsDistance error (km)Height error (km)Time error (s)Mean value299.8501.0849.907Minimum value299.6520.00020.0301Maximum value299.9991.99319.961The detailed data is shown in Figures23–25.Figure 23
Terminal height-time error.Figure 24
Histogram of height error.Figure 25
Histogram of time error.Figure23 shows the terminal time error and height error, which implies that all the time errors are in the required range of 20 s and all the height errors are in the required range of 2 km. Figures 24 and 25 show histograms of the terminal time error and height error, which implies that the proposed method can satisfy all the terminal constraints.


## 5.1. Simulation Settings

The HGV model used in this paper is the common aero vehicle (CAV-H). In Equation (1), m=907kg and g=9.8066m/s2. In Equation (2), Sm=0.4839m2. In Equation (3), ρ0=1.2250kg/m3, β=0.00014065, and Re=6378004m. In Equation (4), kQ=5×10−5, qmax=100kPa, nmax=3, and Q̇max=2000 kW/m2. In Equation (8), Vre=2500m/s. In Equation (19), αmax=20deg, αbest=10deg, V1=6000m/s, and V2=5000m/s. The initial longitude and latitude of HGV are 0 deg. The size of the integral step is 0.01 s, and the size of the guidance (policy) step is 150 s. All used random variables in Table 3 are subject to the uniform distribution.Table 3
Parameters of initial reentry point and target point.
Initial parametersValueInitial heightH060~64 kmInitial velocityV06200~6800 m/sInitial path angleσ0−0.02~−0.01 degTarget longitudeθtar60~65 degTarget latitudeϕtar−7~7 degRequired heightHre26~32 kmRequired timeTre1200~1500 sThe error of drag coefficientΔCD0~10%The error of lift coefficientΔCL0~10%After setting the above parameters, the networks are trained by Algorithm1.


## 5.2. Training Process of Neural Networks

Comparative experiments are made at the same time to verify the superiority of DDPG+HER method. DDPG algorithm [36] and Proximal Policy Optimization (PPO) algorithm [37] are selected as control groups. The hyperparameters of the three algorithms used in the training process are set in Table 4.Table 4
The hyperparameters in the training.
HyperparameterDDPGPPODDPG+HERDiscount factorλ0.990.990.99Batch size646464Replay buffer size20000—20000Actor learning rate10−410−310−4Critic learning rate10−310−310−3Target update rateτ0.001—0.001Maximum number of steps100010001000Exploration policyOU—N0,0.1GAE factor—0.98—Clip factor—0.2—After the training, the average returns and success count in the latest 100 episodes are shown in Figures9 and 10. The total episode number is 23161.Figure 9
Contrast curves of the average return in the latest 100 episodes.Figure 10
Contrast curves of the success count in the latest 100 episodes.It can be seen from Figure9 that the average return of the DDPG+HER method begins rising first and gets more rewards than other methods. The sparse reward problem is effectively alleviated. At the end time of training, the success rate of the DDPG+HER method in Figure 10 converges to 100%, which implies that the policy given by the actor is operative. The best success count of DDPG is less than 75 and the best success count of PPO is less than 60, which powerfully illustrates that the proposed DDPG+HER method is more effective in the training of the policy network.In Figure11, the loss of actor and critic in the DDPG+HER method is given.Figure 11
The loss of actor and critic in DDPG+HER.It can be seen from Figure11 that the loss of the critic is decreasing to zero along with the progress of training. Meanwhile, the magnitude of actor loss is increasing in the process, which means that the score of the agent is on a trend of improvement.


## 5.3. Random Single Trajectory Analysis

After the convergence of neural networks, the typical simulation is performed randomly. To evaluate the performance of the proposed guidance, a contrast simulation based on NPCG is also performed. The proposed method in this paper is called HERG for short.The terminal constraints are set as target longitudeλtar=63.505deg, latitude ϕtar=3.976deg, required height Hre=30.429km, and required time Tre=1370.0s. The errors of aerodynamics are set as ΔCD=3.5% and ΔCL=4.5%. For the trajectory of HERG, the terminal height is 30.755 km, and the terminal time is 1359.7 s. For the trajectory of NPCG, the terminal height is 32.916 km, and the terminal time is 1386.6 s.The results are shown in Figures12–14.Figure 12
Reentry trajectory of HGV.Figure 13
State-time curves.Figure 14
Path constraint-time curves.Figure12 shows that the HGV can arrive at the required target area. Figure 13 shows that the policy decision is executed 9 times, and terminal height and time are in the required range. The reverse number of the bank angle is reduced greatly for HERG than NPCG. And as is seen in Figure 14, all of the path constraints are satisfied.The simulations are performed in a system with a 12th Gen Intel Core i5-12600KF CPU and 16 GB RAM. In the NPCG method, the corrector works a total of 9 times. As the time of integration decreases, the computing time decreases. The computing time of NPCG is listed in Table5, and the total computing time is 6.431 s. In the HERG method, the policy network is called 9 times and the total computing time is 0.0937 s.Table 5
Computing time of NPCG.
Order123456789Time (s)3.3872.7962.4112.2202.0021.3971.0720.6280.291The contrast simulation results are shown in Table6. It is obvious that with similar terminal errors, the calculation time of the proposed method is less than 1/68 of NPCG. By contrast with NPCG, the proposed method makes a breakthrough in real-time performance.Table 6
Contrast simulation results.
MethodComputing time (s)Range error (km)Height error (km)Time error (s)NPCG6.431299.9982.48716.6HERG0.0937299.9990.32610.3


## 5.4. Verifications on Monte Carlo Simulations

To evaluate the effectiveness of the proposed guidance, 1000 Monte Carlo simulations are performed, whose initial states and errors of aerodynamic coefficients are set according to Table3. The results are shown in Figures 15–22.Figure 15
Height-velocity curve.Figure 16
Latitude-longitude curve.Figure 17
AOA-time curve.Figure 18
Bank angle-time curve.Figure 19
Heating rate-time curve.Figure 20
Dynamic pressure-time curve.Figure 21
Longitudinal overload-time curve.Figure 22
Lateral overload-time curve.As is seen in Figure15, all of the 1000 trajectories of the glide phase are in the range of the H-V corridor, and naturally, in Figures 16–19, all of the path constraints are satisfied. Moreover, the AOA and bank angle in different trajectories take on a continuous trend as is shown in Figures 17 and 18, which indicates that the output of the policy network is steady and robust.The statistical data of terminal constraints is shown in Table7.Table 7
Terminal constraints of the Monte Carlo simulations.
Terminal constraintsDistance error (km)Height error (km)Time error (s)Mean value299.8501.0849.907Minimum value299.6520.00020.0301Maximum value299.9991.99319.961The detailed data is shown in Figures23–25.Figure 23
Terminal height-time error.Figure 24
Histogram of height error.Figure 25
Histogram of time error.Figure23 shows the terminal time error and height error, which implies that all the time errors are in the required range of 20 s and all the height errors are in the required range of 2 km. Figures 24 and 25 show histograms of the terminal time error and height error, which implies that the proposed method can satisfy all the terminal constraints.


## 6. Conclusions

In this paper, an intelligent multiconstrained reentry guidance is developed to generate super real-time guidance commands with no need for a decoupling method. First, the magnitude and sign of the bank angle are combined as the sole guidance command rather than the form of longitudinal guidance and lateral guidance. Then, a policy network is constructed to output the guidance command with the help of DRL. Based on DDPG and HER, the terminal constraints are transformed into multigoals. The improved HER method shows more excellent performance in the training simulation. In simulations, the policy network in the proposed guidance can give commands online in less than 1/68 of NPCG time. The robustness to initial state bias and online aerodynamic errors is excellent. However, the deflection of bias and errors is limited in the range of the training phase. If online parameters are beyond the range, the policy network should be retrained. The relation between the scale of the network and the deflection range deserves further study, and the method proposed in this paper can be used as a reference.



---

*Source: 5883080-2023-03-06.xml*