# Two-Loop Acceleration Autopilot Design and Analysis Based on TD3 Strategy

Authors: Junfang Fan; Denghui Dou; Yi Ji; Ning Liu; Shiwei Chen; Huajie Yan; Junxian Li
Journal: International Journal of Aerospace Engineering
 (2023)

---

## Abstract

A two-loop acceleration autopilot is designed using the twin-delayed deep deterministic policy gradient (TD3) strategy to avoid the tedious design process of conventional tactical missile acceleration autopilots and the difficulty of meeting the performance requirements of the full flight envelope. First, a deep reinforcement learning model for the two-loop autopilot is developed. The flight state information serves as the state, the to-be-designed autopilot control parameters serve as the action, and a reward mechanism based on the stability margin index is designed. The TD3 strategy is subsequently used to offline learn the control parameters for the entire flight envelope. An autopilot control parameter fitting model that can be directly applied to the guidance loop is obtained. Finally, the obtained fitting model is combined with the impact angle constraint in the guidance system and verified online. The simulation results demonstrate that the autopilot based on the TD3 strategy can self-adjust the control parameters online based on the real-time flight state, ensuring system stability and achieving accurate acceleration command tracking.


---

## Body

## 1. Introduction

The autopilot is a critical component of the guidance and control system, capable of producing control forces and torques by driving actuators in response to control commands. Thus, the speed and direction of the missile’s flight can be changed, the stability of the missile’s centroid and attitude can be maintained, and the guided missile can hit the target based on the required flight trajectory and attitude [1, 2]. The key to designing an autopilot is identifying the control parameters that meet specific performance indices. Traditional autopilot design methods include pole placement [3, 4], optimal control [5, 6], sliding mode variable structure [7], event-triggered attitude control policy [8], and active disturbance rejection [9]. Due to the diversification of tactical missile combat scenarios and the intelligent development of missiles, the design of autopilots must meet more stringent requirements, such as improved mobility and stability at large attack angles and large-scale flight scenarios [10, 11]. However, traditional autopilot design methods select typical state feature points in the flight envelope to design the control parameters [12]. Consequently, it is difficult to meet the requirements for the entire control domain of the flight envelope.In recent years, deep learning (DL) and reinforcement learning (RL) have become hot topics in artificial intelligence technology, providing new design options for aircraft guidance and control systems [13–15]. The principle of RL originates from the process of intelligent species learning new things. For a specific task, agents learn through real-time interaction with the external environment and continuous trial and error; ultimately, a task-appropriate action strategy is obtained [16]. Deep reinforcement learning (DRL) combines the autonomous decision-making capability of RL with the feature extraction capability of deep neural networks (DNNs) and has demonstrated outstanding performance in high-dimensional data control problems. Unlike conventional control methods, DRL is less dependent on models and more transferable [17].In order to solve the strong coupling problem between the adaptive updating loop and the strict-feedback control loop in traditional control, a low-frequency learning structure consisting of a low-pass filter, a state estimator, and a multilayer perceptron (MLP) neural network was proposed in [18], which reduced the computation complexity and effectively avoided the update explosion problem. Literature [19] designed a two-loop autopilot based on a missile longitudinal channel model employing the RL principle and solved the quadratic optimal tracking control problem employing an actor-critic structure. The results demonstrated that the designed controller possessed an excellent tracking effect and dynamic performance. Different from literature [19], literature [20] transformed the aircraft control procedure into a Markov decision procedure. Utilizing the deep deterministic policy gradient (DDPG) algorithm [21], the optimal PID controller control parameters were determined iteratively. Compared to the LQR controller, the result of the PID controller exhibited better control effects and robustness. Similar to literature [20], based on the acceleration autopilot, literature [22] developed a control parameter design method for acceleration autopilots using the policy gradient algorithm [23]. In order to reduce the time and workload for identifying model information during the autopilot design process, literature [24] developed a learning-based design method for UAV autopilot using the DDPG algorithm by designing appropriate observation and reward functions. In addition, literature [25] made a comparative analysis of PID neural network controller and DDPG controller and also provided a conceptual proof that reinforcement learning can effectively solve the adaptive optimal control problem of nonlinear dynamic systems. However, most of the methods in [19, 20, 22, 24] and [25] were designed and analyzed based on nonglobal profile state rather than entire flight envelope, which led to insufficient consideration of global constraints and performance indicators. When there was great uncertainty in the flight environment, the autonomy and robustness of the nominal trajectory tracking guidance mode were poor.In order to ensure that the missile can be well controlled in the entire flight envelope, literature [26] used wavelet analysis to monitor the incremental RL’s attitude control stability online. Then, they adaptively modified the learning rate of the RL algorithm based on the gradient descent principle, which effectively enhanced the aircraft’s control stability under large-scale dynamic changes. Literature [27] remodeled the autopilot in the RL framework, trained a two-degree-of-freedom- (DOF-) linearized dynamic model for missiles using the DDPG algorithm. By taking into account different flight conditions and uncertainties in the aerodynamic coefficients, literature [27] validated the ability of the designed controller to maintain closed-loop stability in the presence of uncertain models. Literature [28] developed a method for aircraft control using the DDPG algorithm, and the complete flight control was achieved by controlling the position, speed, and attitude angle. By adding a PD controller, the stability in the early stage of training was effectively improved. Literature [29] proposed a model-free coupling dynamic controller design method for jet aircrafts capable of withstanding multiple types of faults. After offline training, adequate results were achieved under highly coupled maneuvers and were robust to various failure situations. To surmount the reliance on global position data in hostile surroundings where GPS was being attacked or disrupted, literature [30] designed a new GPS-free cooperative elliptical circling controller, in which not only the energy consumption was reduced but also the dependency on global position was removed when multiple nonholonomic vehicles moved together. Literature [31] fixes the autopilot structure as typical three-loop autopilot, and deep reinforcement learning is utilized to learn the autopilot gains, and the state-of-the-art deep deterministic policy gradient algorithm is utilized to learn an action policy that maps the observed states to the autopilot gains. Influenced by literature [31], literature [32] used the pole placement algorithm of a three-loop autopilot as the foundation and designed an intelligent training method for converting three-dimensional control parameters into one-dimensional design parameters using the proximal policy optimization algorithm. The simulation results demonstrated that its control effect was good. However, these design methods had some problems such as poor stability and low strategy learning efficiency. How to improve the practicability of deep reinforcement learning method in the field of autopilot has become an urgent problem to be solved.Motivated by the previous investigations, this paper employs the DRL principle to design a parameter tuning model for a two-loop autopilot using the TD3 algorithm [33], which has the following striking advantages over existing autopilot design schemes:
(1)
Contrasting to the previous alternatives [19, 20, 22, 24, 25] based on nonglobal profile state, herein the proposed method takes the entire flight envelope state space as the research object. The TD3 algorithm is used to offline learn the control parameters for the entire flight envelope, which can ensure the autopilot has good performance during the flight envelope. In addition, after offline training, a fitting model with the real-time flight state of the aircraft as the input and the autopilot parameters as the output is obtained through the policy network, which can ensure the rapid online tuning under the condition of large-scale flight state changes(2)
Different from the reward design mechanism in literatures [26–29] and [31], this paper considers the stability margin when designing the reward mechanism to ensure the autopilot has a desired margin space in the entire flight envelope, which makes the autopilot has stronger robustness. In addition, compared to [31, 32] which randomly selected states during training, this paper arranges the flight states in an orderly manner and then samples them sequentially, which can improve the strategy learning efficiency while ensuring the convergence of the algorithmThe remainder of this paper is organized as follows. Problem formulation is stated in Section2. The specific process of two-loop acceleration autopilot design method based on TD3 strategy is given in Section 3. Section 4 is the simulation analysis of the proposed method. Conclusion of this paper is described in Section 5.


## 2. Problem Description

### 2.1. Two-Loop Autopilot Model

This paper uses an air-to-ground guided missile as the research object. The missile model is illustrated in Figure1, where V represents the velocity, Fy/Fx/Fg represent the lift, resistance, and gravity, respectively, Mz is the pitching moment, θ denotes the ballistic inclination, ϑ is the pitch angle, α represents the angle of attack (AOA), δz is the elevator deflection angle, L denotes the reference length, D is the projectile diameter, c.p is the pressure center position of the whole missile, and c.g is the centroid position.Figure 1
Missile model and characteristic parameters.The expressions of aerodynamic force, moment, and gravity are as follows:(1)Fx=cxqdS,Fy=cyqdS,Mz=mzqdSL,Fg=mg,where cx, cy, and mz denote the resistance coefficient, lift coefficient, and pitching moment coefficient, respectively, S denotes the reference area, qd represents the incoming flow pressure (qd=ρV2/2, where ρ represents the air density at the flying altitude of the missile), m is the missile mass, and g is the gravitational acceleration.The projectile dynamics at the longitudinal plane can be described as follows:(2)α̇=ωz−P+cyαqdSmVα−cyδqdSmVδz+dα,ω̇z=mzω¯qdSLJzωz+mzαqdSLJzα+mzδqdSLJzδz+dω,ay=qdScyαα+cyδδzm+dq,where ωz represents the pitch angle rate, P is the axial thrust of the missile, cyα represents the derivative of the lift coefficient to the AOA, cyδ denotes the derivative of the lift coefficient to the elevator deflection angle, mzα represents the derivative of the pitching moment coefficient to the AOA, mzω¯ represents the derivative of the pitching moment coefficient to the dimensionless pitch angle rate, mzδ denotes the derivative of the pitching moment coefficient to the elevator deflection angle, Jz is the pitching rotational inertia, and dα, dω, and dq represent the aerodynamic noise interference. The aerodynamic noise obeys the following bounded Gaussian distribution:
(3)dα/ω/q~clipN0,σα/ω/q,−eα/ω/q,eα/ω/q,where eα/ω/q is the upper bound of noise, σα/ω/q represents the variance of the Gaussian distribution N0,σα/ω/q, and the clip function is defined as follows:
(4)clipx1,y1,z1=x1ify1≤x1≤z1,y1ifx1<y1,z1ifx1>z1.The research object of this paper is a thrust-free missile; thus,P=0. Figure 2 shows the two-loop acceleration autopilot model composed of an accelerometer and a velocity gyroscope [34]. In Figure 2, ayc denotes the acceleration command based on the guidance command, δzc is the actuator command with the actuator dynamic delay being temporarily ignored, and kA and kg are the control parameters to be designed.Figure 2
Two-loop autopilot structure.The geometric relationship between the two-dimensional missile plane and the target is illustrated in Figure3, where M represents the missile position, T is the target position, q denotes the line-of-sight angle of the missile and the target, φ is the lead angle, and R is the relative distance between missile and target.Figure 3
Missile-target geometric relationships.For stationary targets,(5)mV̇=−Fx−Fgsinθ,Ṙ=−Vcosφ,Rq̇=Vsinφ.It is well-known that the dynamics of the airframe change with the flight state during the flight process. To maintain control stability and accurate acceleration tracking throughout the entire guidance procedure, the control parameters must be continuously adjusted based on the flight state. A missile guidance control model was designed in combination with the impact angle constraint guidance (Figure4). In this paper, a two-loop autopilot with self-adjustable control parameters based on the model depicted in Figure 4 will be developed.Figure 4
Guidance control model.


### 2.2. Autopilot Parameter Design Method and Online Application Framework

The key to the design of the autopilot is to ensure that the control parameterskA and kg meet the expected performance indices of the control system, so that the aircraft can accurately and robustly track the guidance acceleration command, accelerate the response speed, and improve the damping of the missile. Among the frequency domain indices of the system, the amplitude margin hm and phase margin γm are often used as important indices to assess the control system performance. In general, the amplitude and phase margins of the autopilot should not be less than 6.5 dB and 40°, respectively. Therefore, in this paper, the amplitude margin hm and phase margin γm are selected as the performance indices that must be satisfied during the design process of the two-loop autopilot. According to Figure 2, the conventional open-loop system Gs is the transfer function from “in” to “out,” which is disconnected at the actuator. As a result, hm and γm can be calculated as follows:
(6)hm=1Gjωx,γm=180°+∠Gjωc,where ωx represents the phase crossover frequency, meeting ∠Gjωx=−π, and ωc is the cut-off frequency, meeting Gjωc=1.In this paper, the guidance control model described in Section2.1 is modeled by RL. The TD3 algorithm was utilized to offline learn the driving control parameters for each flight state across the entire flight envelope. At the same time, a MLP neural network is used to model the nonlinear relationship between different flight states and control parameters in the TD3 algorithm model. As depicted in Figure 5, the MLP neural network fitting can be directly applied to the missile guidance control loop. The corresponding autopilot control parameters can be self-adjusted online during flight based on the real-time flight state.Figure 5
Design and online application framework of the autopilot based on the TD3 algorithm.


## 2.1. Two-Loop Autopilot Model

This paper uses an air-to-ground guided missile as the research object. The missile model is illustrated in Figure1, where V represents the velocity, Fy/Fx/Fg represent the lift, resistance, and gravity, respectively, Mz is the pitching moment, θ denotes the ballistic inclination, ϑ is the pitch angle, α represents the angle of attack (AOA), δz is the elevator deflection angle, L denotes the reference length, D is the projectile diameter, c.p is the pressure center position of the whole missile, and c.g is the centroid position.Figure 1
Missile model and characteristic parameters.The expressions of aerodynamic force, moment, and gravity are as follows:(1)Fx=cxqdS,Fy=cyqdS,Mz=mzqdSL,Fg=mg,where cx, cy, and mz denote the resistance coefficient, lift coefficient, and pitching moment coefficient, respectively, S denotes the reference area, qd represents the incoming flow pressure (qd=ρV2/2, where ρ represents the air density at the flying altitude of the missile), m is the missile mass, and g is the gravitational acceleration.The projectile dynamics at the longitudinal plane can be described as follows:(2)α̇=ωz−P+cyαqdSmVα−cyδqdSmVδz+dα,ω̇z=mzω¯qdSLJzωz+mzαqdSLJzα+mzδqdSLJzδz+dω,ay=qdScyαα+cyδδzm+dq,where ωz represents the pitch angle rate, P is the axial thrust of the missile, cyα represents the derivative of the lift coefficient to the AOA, cyδ denotes the derivative of the lift coefficient to the elevator deflection angle, mzα represents the derivative of the pitching moment coefficient to the AOA, mzω¯ represents the derivative of the pitching moment coefficient to the dimensionless pitch angle rate, mzδ denotes the derivative of the pitching moment coefficient to the elevator deflection angle, Jz is the pitching rotational inertia, and dα, dω, and dq represent the aerodynamic noise interference. The aerodynamic noise obeys the following bounded Gaussian distribution:
(3)dα/ω/q~clipN0,σα/ω/q,−eα/ω/q,eα/ω/q,where eα/ω/q is the upper bound of noise, σα/ω/q represents the variance of the Gaussian distribution N0,σα/ω/q, and the clip function is defined as follows:
(4)clipx1,y1,z1=x1ify1≤x1≤z1,y1ifx1<y1,z1ifx1>z1.The research object of this paper is a thrust-free missile; thus,P=0. Figure 2 shows the two-loop acceleration autopilot model composed of an accelerometer and a velocity gyroscope [34]. In Figure 2, ayc denotes the acceleration command based on the guidance command, δzc is the actuator command with the actuator dynamic delay being temporarily ignored, and kA and kg are the control parameters to be designed.Figure 2
Two-loop autopilot structure.The geometric relationship between the two-dimensional missile plane and the target is illustrated in Figure3, where M represents the missile position, T is the target position, q denotes the line-of-sight angle of the missile and the target, φ is the lead angle, and R is the relative distance between missile and target.Figure 3
Missile-target geometric relationships.For stationary targets,(5)mV̇=−Fx−Fgsinθ,Ṙ=−Vcosφ,Rq̇=Vsinφ.It is well-known that the dynamics of the airframe change with the flight state during the flight process. To maintain control stability and accurate acceleration tracking throughout the entire guidance procedure, the control parameters must be continuously adjusted based on the flight state. A missile guidance control model was designed in combination with the impact angle constraint guidance (Figure4). In this paper, a two-loop autopilot with self-adjustable control parameters based on the model depicted in Figure 4 will be developed.Figure 4
Guidance control model.


## 2.2. Autopilot Parameter Design Method and Online Application Framework

The key to the design of the autopilot is to ensure that the control parameterskA and kg meet the expected performance indices of the control system, so that the aircraft can accurately and robustly track the guidance acceleration command, accelerate the response speed, and improve the damping of the missile. Among the frequency domain indices of the system, the amplitude margin hm and phase margin γm are often used as important indices to assess the control system performance. In general, the amplitude and phase margins of the autopilot should not be less than 6.5 dB and 40°, respectively. Therefore, in this paper, the amplitude margin hm and phase margin γm are selected as the performance indices that must be satisfied during the design process of the two-loop autopilot. According to Figure 2, the conventional open-loop system Gs is the transfer function from “in” to “out,” which is disconnected at the actuator. As a result, hm and γm can be calculated as follows:
(6)hm=1Gjωx,γm=180°+∠Gjωc,where ωx represents the phase crossover frequency, meeting ∠Gjωx=−π, and ωc is the cut-off frequency, meeting Gjωc=1.In this paper, the guidance control model described in Section2.1 is modeled by RL. The TD3 algorithm was utilized to offline learn the driving control parameters for each flight state across the entire flight envelope. At the same time, a MLP neural network is used to model the nonlinear relationship between different flight states and control parameters in the TD3 algorithm model. As depicted in Figure 5, the MLP neural network fitting can be directly applied to the missile guidance control loop. The corresponding autopilot control parameters can be self-adjusted online during flight based on the real-time flight state.Figure 5
Design and online application framework of the autopilot based on the TD3 algorithm.


## 3. DRL Design Method for the Autopilot

### 3.1. Markov Decision Process

The flight state of the missile at two adjacent moments can be approximately regarded as the transfer between the states under a given control command, and the state of the next moment is only related to the state at the current moment, so the flight state of the missile has Markov property. By discretizing the flight process in the time dimension, the guidance process of the missile can be approximately modeled as a discrete-time Markov chain (DTMC). When the guidance law is determined, the control parameters determine whether the autopilot can follow the guidance command. Therefore, the design process of the autopilot is to add decision-making command to the Markov chain of the missile flight process, so the design process of the missile autopilot can be modeled as a Markov decision process (MDP).MDP is a sequential decision process, which can be described by a 5-tupleS,A,P.,R.,γ. The specific model definition is as follows:
(7)MDPS:stateset,S=s1,s2,⋯sn,A:actionset,A=a1,a2,⋯an,Pst+1st,at:thestatetransitionfunctionoftheenvironment,Rst,at,st+1:therewardfunctionoftheenvironment,γ:discountfactor.In this paper, the agent cannot directly access the transition functionPst+1st,at and the reward function Rst,at,st+1 but can only obtain specific information about the state st, action at, and reward rt by interacting with the environment. The interaction process between the agent and the environment in the MDP is depicted in Figure 6.Figure 6
The interaction process between the agent and the environment in the MDP.The autopilot design problem is a continuous control problem. In each round, the agent observes the statest at each time t and decides the action at to be taken according to the current strategy. The strategy π is the mapping from the state to the action (π:a∼πs). Autopilot model gets the next state under the action and gets a reward from the environment. The trajectory generated by the control loop from s0 to the termination state sT is expressed as τ:s0,a0,r0,s1,a1,r1,⋯,s1,a1,r1.Formalizing the optimization objective, the reward is defined as the weighted sum of all rewards in the trajectory:(8)Rτ=∑t=0Tγtrt,where γ∈0,1 represents discount factor and T represents the state number of a scene data trajectory. The objective function Jτ is defined as the expectation of the trajectory return:
(9)Jτ=Eτ∼πRτ=Eτ∼π∑t=0Tγtrt.WithQst,at defined as the state action-value function of Jτ, the Behrman equation of the state action-value function can be expressed as
(10)Qst,at=Eπrt+γEπQst+1,at+1.Throughout the interaction between agent and environment, the optimal policy is continuously updated to maximize the state action-value function.(11)π∗=argmaxπQst,at.The Behrman optimal equation of the state action-value function can be obtained through the Behrman optimal principle:(12)Q∗st,at=Eπ∗rt+γEπ∗Q∗st+1,at+1.


### 3.2. TD3 Algorithm

The TD3 algorithm is an off-policy algorithm for continuous action space. Based on the DDPG algorithm, the actor- and critic-networks are simultaneously improved, thereby resolving the problem of the critic-network overestimating theQ value and enhancing the algorithm’s stability. To increase the agent’s ability to explore the environment, the TD3 algorithm chooses actions based on the current policy and exploration noise:
(13)at=clipπϕst+ϵ,amin,amax,where ϕ represents the actor-network parameter, amin is the lower limit of the action space, amax denotes the upper limit of the action space, and ϵ is the exploration noise which obeys the Gaussian distribution: ϵ~N0,σt. In addition, to ensure the algorithm convergence in the later stage, the Gaussian distribution variance σt obeys the following: σt+1=σt1−ε, where 0≤ε≤1 is the attenuation factor.The experience replay mechanism is adopted for the TD3 algorithm. The agent inputs the trial datast,at,rt,st+1 into the experience buffer pool R and then randomly selects the data si,ai,ri,si+1 of the batch N from R to train the network and update its parameters. To solve the overestimation problem of the Q value, the TD3 algorithm uses two sets of critic-networks to represent different Q values and estimates the target Q value using the following equation:
(14)yi=ri+γmink=1,2Qθk′si+1,a~i+1,where θk′k=1,2 is the parameters of the target critic-network and a~i+1 is expressed as follows:
(15)a~i+1=clipπϕ′si+1+ϵ~,amin,amax,where ϕ′ represents the parameter of the target actor-network. Random noise ϵ~ is introduced to enhance the stability of the target policy. The noise ϵ~ follows an independent distribution controlled by parameters different from those of the exploration noise ϵ, i.e., ϵ~~clipN0,σ~,−e,e, where e is the upper limit of the noise.The TD3 algorithm updates the critic-network parameterθk by minimizing the temporal difference (TD) error:
(16)θk⟵argminθk1N∑iyi−Qθksi,ai2,fork∈1,2,where θkk=1,2 is the critic-network parameter.To reduce the number of incorrect updates and improve the algorithm’s stability, the TD3 algorithm does not update the actor-network until theQ value becomes stable. Therefore, the actor-network in the TD3 algorithm has a slightly lower update frequency than the critic-network. Through a deterministic policy gradient, the TD3 algorithm updates the actor-network parameters.
(17)∇ϕJπϕ≈1N∑i∇aQθ1si,aa=πϕsi∇ϕπϕsi.To ensure the stability of training the neural network, the soft update strategy is adopted for the target network parameters:(18)θk′⟵μθk+1−μθk′fork∈1,2,ϕ′⟵μϕ+1−μϕ′,where 0<μ<1 is a smoothing constant that represents the update speed. The pseudocode of the TD3 algorithm is given in Algorithm 1.Algorithm 1: TD3 algorithm.
TD3: Twin Delayed Deep Deterministic Policy GradientRandomly initialize the network parametersθ1,θ2, and ϕInitialize the target network parametersθ1′⟵θ1,θ2′⟵θ2,ϕ′⟵ϕInitialize the replay bufferRfor episode = 1, M doInitialize an exploration noiseє~N0,σ for action explorationReceive the initial environmental state quantitys0for t = 1, T doSelect an action according to the current policy and exploration noise:at=clipπϕst+є,amin,amaxExecute the actionat and observe the reward rt and the next state st+1Store the explored transition arrayst,at,rt,st+1 in RExtract sample datasi,ai,ri,si+1 of the batch N from Ra~i+1=clipπϕ′si+1+є~,amin,amax,є~~clipN0,σ~,−c,cyi=ri+γmink=1,2Qθk′si+1,a~i+1Update the critic-network parameters:θk⟵argminθk1/N∑iyi−Qθksi,ai2fork∈1,2if t mod d thenUpdate the actor-network parameters through deterministic policy gradients:∇ϕJπϕ≈1/N∑i∇aQθ1si,aa=πϕsi∇ϕπϕsiUpdate the target network:θk′⟵μθk+1−μθk′fork∈1,2ϕ′⟵μϕ+1−μϕ′end ifend forend for


### 3.3. RL Model for Autopilot Design

To use the TD3 algorithm to solve the problem of two-loop autopilot parameter design, it is necessary to transform the autopilot design process into an RL problem and to design the various components of the RL model, i.e., state, action, and reward. At the same time, the algorithm’s network structure and hyperparameters must be determined to ensure its effectiveness.The design of the two-loop autopilot requires information regarding the flight state feature points in the full flight envelope. The Mach number andα can affect the change of aerodynamic characteristics, and the flight altitude and Mach number can affect the change of dynamic pressure, so the state vector can be designed as st:H,Ma,αT, where Ma represents the Mach number and H represents the flight height. The objective of the design of a two-loop autopilot is that the control parameters satisfy the performance index requirements. Thus, the action vector is designed as at:kA,kgT.The reward signal is the objective that must be maximized by the agent. A reasonable reward mechanism is crucial for training effectiveness. The frequency domain stability margin of the control system is selected as the performance index, and the following reward function is designed:(19)r=−k1hm−hm02+k2γm−γm02,where hm0 is the expected magnitude margin, γm0 represents the expected phase margin, and k1 and k2 are the influence weights of the magnitude and phase margins on the reward, respectively, meeting 0≤k1, k2≤1, and k1+k2=1.As shown in Figure7, both the actor- and critic-networks of the TD3 algorithm employ an MLP neural network with four layers, and an activation function is connected behind the neurons of each layer besides the input layer. To prevent control input saturation, the tanh function activates the actor-network of the TD3 algorithm:
(20)Tanhx=ex−e−xex+e−x.Figure 7
TD3 algorithm network layer structure.The critic-network is activated by the ReLu function:(21)ReLux=xifx>0,0ifx≤0.The dimensions of each layer of the actor- and critic-networks are defined in Table1.Table 1
Dimensions of each layer of the actor- and critic-networks.
LayerActor-networkCritic-networkInput layer3 (state dimension)5 (state dimension+action dimension)Hidden layer 1256256Hidden layer 2256256Output layer2 (action dimension)1 (action value function dimension)The inputs of the actor- and critic-networks include state vectors. The state vectors must be normalized to eliminate the effect of input data dimensions on the neural network training procedure. In this paper, the state vectors were processed by (0,1) normalization:(22)H¯=H−HminHmax−Hmin,M¯a=Ma−MaminMamax−Mamin,α¯=α−αminαmax−αmin,where H¯, M¯a, and α¯ are the normalized network input values, in which H∈Hmin,Hmax, Ma∈Mamin,Mamax, and α∈αmin,αmax.Since the tanh activation function limits the output of the actor-network to the range (-1, 1), the output value of the actor-network needs to be denormalized to obtain the action vector value:(23)kA=k¯AkAmax−kAmin+kAmax+kAmin2,kg=k¯gkgmax−kgmin+kgmax+kgmin2,where k¯A and k¯g are the output values of the actor-network, kAmax,kAmin are the maximum and minimum values of the control parameters, respectively, and kgmax,kgmin are the maximum and minimum values of the control parameters, respectively.To prevent the overfitting problem, both the actor- and critic-networks are trained by the Adam optimizer withL2 regularization. The hyperparameter settings significantly affect the performance of the TD3 algorithm. Table 2 shows the hyperparameters suitable for the application scope of this paper.Table 2
TD3 algorithm hyperparameters.
ParameterValueMaximum permissible episodes1600Maximum permissible steps of each episode572Actor-network learning rate10-4Critic-network learning rate10-4Regularization constant6×10−3Discounting factorγ0.99Sampling sizeN256Varianceσt for the initial exploration noise ϵ0.25Variance fading factorε0.005Varianceσ~ for random noise ϵ~0.2Upper limite of the random noise ϵ~0.25


### 3.4. Offline Training Framework

Before training, computational fluid dynamics (CFD) software was used to calculate the missile’s pneumatic parameters throughout its entire flight envelope, and a library of pneumatic parameters was compiled.In the offline training process, the flight speedV can be obtained using the following atmospheric model:
(24)ρ=ρ0e−g/287.05TH,c=1.4×287.05T,where c represents acoustic velocity and ρ0=1.225kg/m3 is the atmospheric density in the sea level. The correlation between the temperature T and the troposphere H can be approximately expressed as
(25)T=T0−0.0065H,where T0=288.15K is the reference temperature at sea level.Firstly, the actor-network and the critic-network are initialized. Then, the experiment begins, the initial statest is selected and brought into the library of pneumatic parameters to obtain the corresponding pneumatic parameters, and the dynamic coefficient of the two-loop autopilot is computed. In this paper, thrust P=0, so the dynamic coefficient is defined as follows: aα=−mzαqSL/Jz, aω=−mzω¯qSL/Jz, bα=cyαqS/mV, bδ=cyδqS/mV+dα, and aδ=−mzδqSL/Jz+dω.According to the structure of the two-loop autopilot, the open-loop transfer functionGs can be derived as follows [34]:
(26)Gs=M2s2+M1s+M0s2/ωm2+2μms/ωm+1,where ωm=aα+aωbα and μm=aω+bα/2ωm. M0, M1, and M2 are expressed as follows:
(27)M0=aδbα−aαbδkg+kAVaα+aωbα,M1=kgaδ−kAaωbδV+aδbα−aαbδckAaα+aωbα,M2=kAcaδ−bδVaα+aωbα.At the same time, the flight statest determined the control parameters kA and kg through the actor-network and exploration noise ϵ and output the action at. Then, the system stability margin is calculated according to Equation (6) and Equation (26), and the real-time reward rt is calculated according to Equation (19).It is worth noting that the offline design of the autopilot parameters in the whole flight envelope is not a traditional sequential decision-making problem but can be regarded as a problem to obtain the optimal value of control parameters in the flight envelope. Therefore, the acquisition of the next flight statest+1 in the interaction with the environment needs to be designed. After several tests, it is determined that, compared to random sampling for training, the orderly arrangement of flight states and sequential sampling can improve the efficiency of the TD3 algorithm and the rate of convergence of the reward function. Therefore, in this study, all flight states in the library of pneumatic parameters are sequentially sorted. The ranking rule is fixed H first, and then, Ma and α are determined in turn. The next flight state st+1 is determined according to the ranking rule.The explored datast,at,rt,st+1 are imported into the experience buffer pool R. Then, the data si,ai,ri,si+1 of the batch N are selected from the experience buffer pool R to train the network according to the TD3 algorithm. It should be noted that this sampling method does not affect the neural network’s ability to generalize, as the network model can “remember” the order of the samples. This is because the TD3 algorithm employs the experience replay mechanism, and random selection of data from the experience pool during training breaks the correlation between sequences. The offline training framework for the two-loop autopilot parameter design based on the TD3 algorithm is illustrated in Figure 8.Figure 8
Two-loop autopilot offline training framework based on the TD3 algorithm.


## 3.1. Markov Decision Process

The flight state of the missile at two adjacent moments can be approximately regarded as the transfer between the states under a given control command, and the state of the next moment is only related to the state at the current moment, so the flight state of the missile has Markov property. By discretizing the flight process in the time dimension, the guidance process of the missile can be approximately modeled as a discrete-time Markov chain (DTMC). When the guidance law is determined, the control parameters determine whether the autopilot can follow the guidance command. Therefore, the design process of the autopilot is to add decision-making command to the Markov chain of the missile flight process, so the design process of the missile autopilot can be modeled as a Markov decision process (MDP).MDP is a sequential decision process, which can be described by a 5-tupleS,A,P.,R.,γ. The specific model definition is as follows:
(7)MDPS:stateset,S=s1,s2,⋯sn,A:actionset,A=a1,a2,⋯an,Pst+1st,at:thestatetransitionfunctionoftheenvironment,Rst,at,st+1:therewardfunctionoftheenvironment,γ:discountfactor.In this paper, the agent cannot directly access the transition functionPst+1st,at and the reward function Rst,at,st+1 but can only obtain specific information about the state st, action at, and reward rt by interacting with the environment. The interaction process between the agent and the environment in the MDP is depicted in Figure 6.Figure 6
The interaction process between the agent and the environment in the MDP.The autopilot design problem is a continuous control problem. In each round, the agent observes the statest at each time t and decides the action at to be taken according to the current strategy. The strategy π is the mapping from the state to the action (π:a∼πs). Autopilot model gets the next state under the action and gets a reward from the environment. The trajectory generated by the control loop from s0 to the termination state sT is expressed as τ:s0,a0,r0,s1,a1,r1,⋯,s1,a1,r1.Formalizing the optimization objective, the reward is defined as the weighted sum of all rewards in the trajectory:(8)Rτ=∑t=0Tγtrt,where γ∈0,1 represents discount factor and T represents the state number of a scene data trajectory. The objective function Jτ is defined as the expectation of the trajectory return:
(9)Jτ=Eτ∼πRτ=Eτ∼π∑t=0Tγtrt.WithQst,at defined as the state action-value function of Jτ, the Behrman equation of the state action-value function can be expressed as
(10)Qst,at=Eπrt+γEπQst+1,at+1.Throughout the interaction between agent and environment, the optimal policy is continuously updated to maximize the state action-value function.(11)π∗=argmaxπQst,at.The Behrman optimal equation of the state action-value function can be obtained through the Behrman optimal principle:(12)Q∗st,at=Eπ∗rt+γEπ∗Q∗st+1,at+1.


## 3.2. TD3 Algorithm

The TD3 algorithm is an off-policy algorithm for continuous action space. Based on the DDPG algorithm, the actor- and critic-networks are simultaneously improved, thereby resolving the problem of the critic-network overestimating theQ value and enhancing the algorithm’s stability. To increase the agent’s ability to explore the environment, the TD3 algorithm chooses actions based on the current policy and exploration noise:
(13)at=clipπϕst+ϵ,amin,amax,where ϕ represents the actor-network parameter, amin is the lower limit of the action space, amax denotes the upper limit of the action space, and ϵ is the exploration noise which obeys the Gaussian distribution: ϵ~N0,σt. In addition, to ensure the algorithm convergence in the later stage, the Gaussian distribution variance σt obeys the following: σt+1=σt1−ε, where 0≤ε≤1 is the attenuation factor.The experience replay mechanism is adopted for the TD3 algorithm. The agent inputs the trial datast,at,rt,st+1 into the experience buffer pool R and then randomly selects the data si,ai,ri,si+1 of the batch N from R to train the network and update its parameters. To solve the overestimation problem of the Q value, the TD3 algorithm uses two sets of critic-networks to represent different Q values and estimates the target Q value using the following equation:
(14)yi=ri+γmink=1,2Qθk′si+1,a~i+1,where θk′k=1,2 is the parameters of the target critic-network and a~i+1 is expressed as follows:
(15)a~i+1=clipπϕ′si+1+ϵ~,amin,amax,where ϕ′ represents the parameter of the target actor-network. Random noise ϵ~ is introduced to enhance the stability of the target policy. The noise ϵ~ follows an independent distribution controlled by parameters different from those of the exploration noise ϵ, i.e., ϵ~~clipN0,σ~,−e,e, where e is the upper limit of the noise.The TD3 algorithm updates the critic-network parameterθk by minimizing the temporal difference (TD) error:
(16)θk⟵argminθk1N∑iyi−Qθksi,ai2,fork∈1,2,where θkk=1,2 is the critic-network parameter.To reduce the number of incorrect updates and improve the algorithm’s stability, the TD3 algorithm does not update the actor-network until theQ value becomes stable. Therefore, the actor-network in the TD3 algorithm has a slightly lower update frequency than the critic-network. Through a deterministic policy gradient, the TD3 algorithm updates the actor-network parameters.
(17)∇ϕJπϕ≈1N∑i∇aQθ1si,aa=πϕsi∇ϕπϕsi.To ensure the stability of training the neural network, the soft update strategy is adopted for the target network parameters:(18)θk′⟵μθk+1−μθk′fork∈1,2,ϕ′⟵μϕ+1−μϕ′,where 0<μ<1 is a smoothing constant that represents the update speed. The pseudocode of the TD3 algorithm is given in Algorithm 1.Algorithm 1: TD3 algorithm.
TD3: Twin Delayed Deep Deterministic Policy GradientRandomly initialize the network parametersθ1,θ2, and ϕInitialize the target network parametersθ1′⟵θ1,θ2′⟵θ2,ϕ′⟵ϕInitialize the replay bufferRfor episode = 1, M doInitialize an exploration noiseє~N0,σ for action explorationReceive the initial environmental state quantitys0for t = 1, T doSelect an action according to the current policy and exploration noise:at=clipπϕst+є,amin,amaxExecute the actionat and observe the reward rt and the next state st+1Store the explored transition arrayst,at,rt,st+1 in RExtract sample datasi,ai,ri,si+1 of the batch N from Ra~i+1=clipπϕ′si+1+є~,amin,amax,є~~clipN0,σ~,−c,cyi=ri+γmink=1,2Qθk′si+1,a~i+1Update the critic-network parameters:θk⟵argminθk1/N∑iyi−Qθksi,ai2fork∈1,2if t mod d thenUpdate the actor-network parameters through deterministic policy gradients:∇ϕJπϕ≈1/N∑i∇aQθ1si,aa=πϕsi∇ϕπϕsiUpdate the target network:θk′⟵μθk+1−μθk′fork∈1,2ϕ′⟵μϕ+1−μϕ′end ifend forend for


## 3.3. RL Model for Autopilot Design

To use the TD3 algorithm to solve the problem of two-loop autopilot parameter design, it is necessary to transform the autopilot design process into an RL problem and to design the various components of the RL model, i.e., state, action, and reward. At the same time, the algorithm’s network structure and hyperparameters must be determined to ensure its effectiveness.The design of the two-loop autopilot requires information regarding the flight state feature points in the full flight envelope. The Mach number andα can affect the change of aerodynamic characteristics, and the flight altitude and Mach number can affect the change of dynamic pressure, so the state vector can be designed as st:H,Ma,αT, where Ma represents the Mach number and H represents the flight height. The objective of the design of a two-loop autopilot is that the control parameters satisfy the performance index requirements. Thus, the action vector is designed as at:kA,kgT.The reward signal is the objective that must be maximized by the agent. A reasonable reward mechanism is crucial for training effectiveness. The frequency domain stability margin of the control system is selected as the performance index, and the following reward function is designed:(19)r=−k1hm−hm02+k2γm−γm02,where hm0 is the expected magnitude margin, γm0 represents the expected phase margin, and k1 and k2 are the influence weights of the magnitude and phase margins on the reward, respectively, meeting 0≤k1, k2≤1, and k1+k2=1.As shown in Figure7, both the actor- and critic-networks of the TD3 algorithm employ an MLP neural network with four layers, and an activation function is connected behind the neurons of each layer besides the input layer. To prevent control input saturation, the tanh function activates the actor-network of the TD3 algorithm:
(20)Tanhx=ex−e−xex+e−x.Figure 7
TD3 algorithm network layer structure.The critic-network is activated by the ReLu function:(21)ReLux=xifx>0,0ifx≤0.The dimensions of each layer of the actor- and critic-networks are defined in Table1.Table 1
Dimensions of each layer of the actor- and critic-networks.
LayerActor-networkCritic-networkInput layer3 (state dimension)5 (state dimension+action dimension)Hidden layer 1256256Hidden layer 2256256Output layer2 (action dimension)1 (action value function dimension)The inputs of the actor- and critic-networks include state vectors. The state vectors must be normalized to eliminate the effect of input data dimensions on the neural network training procedure. In this paper, the state vectors were processed by (0,1) normalization:(22)H¯=H−HminHmax−Hmin,M¯a=Ma−MaminMamax−Mamin,α¯=α−αminαmax−αmin,where H¯, M¯a, and α¯ are the normalized network input values, in which H∈Hmin,Hmax, Ma∈Mamin,Mamax, and α∈αmin,αmax.Since the tanh activation function limits the output of the actor-network to the range (-1, 1), the output value of the actor-network needs to be denormalized to obtain the action vector value:(23)kA=k¯AkAmax−kAmin+kAmax+kAmin2,kg=k¯gkgmax−kgmin+kgmax+kgmin2,where k¯A and k¯g are the output values of the actor-network, kAmax,kAmin are the maximum and minimum values of the control parameters, respectively, and kgmax,kgmin are the maximum and minimum values of the control parameters, respectively.To prevent the overfitting problem, both the actor- and critic-networks are trained by the Adam optimizer withL2 regularization. The hyperparameter settings significantly affect the performance of the TD3 algorithm. Table 2 shows the hyperparameters suitable for the application scope of this paper.Table 2
TD3 algorithm hyperparameters.
ParameterValueMaximum permissible episodes1600Maximum permissible steps of each episode572Actor-network learning rate10-4Critic-network learning rate10-4Regularization constant6×10−3Discounting factorγ0.99Sampling sizeN256Varianceσt for the initial exploration noise ϵ0.25Variance fading factorε0.005Varianceσ~ for random noise ϵ~0.2Upper limite of the random noise ϵ~0.25


## 3.4. Offline Training Framework

Before training, computational fluid dynamics (CFD) software was used to calculate the missile’s pneumatic parameters throughout its entire flight envelope, and a library of pneumatic parameters was compiled.In the offline training process, the flight speedV can be obtained using the following atmospheric model:
(24)ρ=ρ0e−g/287.05TH,c=1.4×287.05T,where c represents acoustic velocity and ρ0=1.225kg/m3 is the atmospheric density in the sea level. The correlation between the temperature T and the troposphere H can be approximately expressed as
(25)T=T0−0.0065H,where T0=288.15K is the reference temperature at sea level.Firstly, the actor-network and the critic-network are initialized. Then, the experiment begins, the initial statest is selected and brought into the library of pneumatic parameters to obtain the corresponding pneumatic parameters, and the dynamic coefficient of the two-loop autopilot is computed. In this paper, thrust P=0, so the dynamic coefficient is defined as follows: aα=−mzαqSL/Jz, aω=−mzω¯qSL/Jz, bα=cyαqS/mV, bδ=cyδqS/mV+dα, and aδ=−mzδqSL/Jz+dω.According to the structure of the two-loop autopilot, the open-loop transfer functionGs can be derived as follows [34]:
(26)Gs=M2s2+M1s+M0s2/ωm2+2μms/ωm+1,where ωm=aα+aωbα and μm=aω+bα/2ωm. M0, M1, and M2 are expressed as follows:
(27)M0=aδbα−aαbδkg+kAVaα+aωbα,M1=kgaδ−kAaωbδV+aδbα−aαbδckAaα+aωbα,M2=kAcaδ−bδVaα+aωbα.At the same time, the flight statest determined the control parameters kA and kg through the actor-network and exploration noise ϵ and output the action at. Then, the system stability margin is calculated according to Equation (6) and Equation (26), and the real-time reward rt is calculated according to Equation (19).It is worth noting that the offline design of the autopilot parameters in the whole flight envelope is not a traditional sequential decision-making problem but can be regarded as a problem to obtain the optimal value of control parameters in the flight envelope. Therefore, the acquisition of the next flight statest+1 in the interaction with the environment needs to be designed. After several tests, it is determined that, compared to random sampling for training, the orderly arrangement of flight states and sequential sampling can improve the efficiency of the TD3 algorithm and the rate of convergence of the reward function. Therefore, in this study, all flight states in the library of pneumatic parameters are sequentially sorted. The ranking rule is fixed H first, and then, Ma and α are determined in turn. The next flight state st+1 is determined according to the ranking rule.The explored datast,at,rt,st+1 are imported into the experience buffer pool R. Then, the data si,ai,ri,si+1 of the batch N are selected from the experience buffer pool R to train the network according to the TD3 algorithm. It should be noted that this sampling method does not affect the neural network’s ability to generalize, as the network model can “remember” the order of the samples. This is because the TD3 algorithm employs the experience replay mechanism, and random selection of data from the experience pool during training breaks the correlation between sequences. The offline training framework for the two-loop autopilot parameter design based on the TD3 algorithm is illustrated in Figure 8.Figure 8
Two-loop autopilot offline training framework based on the TD3 algorithm.


## 4. Simulation Analysis

The sample space for the two-loop autopilot parameter training process was determined based on the application scene of the selected missile in this project. Subsequently, the TD3 algorithm is used to train the model offline, and the training results are analyzed. Finally, in conjunction with the impact angle constraint guidance problem, the neural network fitting model obtained after training was directly implemented in the guidance control loop for trajectory simulation to demonstrate its performance.


### 4.1. Autopilot Design Simulation Based on the TD3 Algorithm

First, the design state space of the autopilot parameters is determined:S:H,Ma,αT, H∈0m,3000m, Ma∈0.3,1, and α∈−12°,12°. Then, the design action space of the autopilot parameters is set as follows: at:kA,kgT, kA∈0,0.01, and kg∈0,0.5.Subsequently, the expected performance indices are selected,hm0=8.5dB, γm0=80°, k1=0.5, and k2=0.5, and the hyperparameters are set according to Table 2 for offline training. The change process of the episode cumulative reward during the offline training process is exhibited in Figure 9.Figure 9
Episode cumulative reward change curve.It can be seen that compared with random sampling, sequential sampling improved the strategy learning efficiency of the algorithm, and the stable value of cumulative reward was higher than random sampling. This is because random sampling in the training process leads to no correlation in state transfer, which makes the agent unable to determine how the state transfers through learning, so that there will be a large difference in each calculation of theQ value, and it is difficult to obtain a stable Q value. However, with sequential sampling, the state transition is determined, which greatly reduces the learning difficulty of the agent and enables faster convergence of cumulative reward. In the early stages of training, the cumulative reward value was small, with relatively large fluctuation due to the random selection of actions within the set range. Nevertheless, as the training progressed, the agent gradually chose better actions; as a result, the cumulative reward value gradually increased, and the fluctuation decreased. At 600 epochs of training, the cumulative reward value of sequential sampling was greater than -60 but still exhibited an upward trend. When the training reached 1550 epochs, the cumulative reward value was stable at approximately -9.9, with a small fluctuation range. It can be considered that the training process achieved the ideal results.The design results of three characteristic points in the flight envelope are selected for analysis.The feature points 1 and 2 in Table3 are selected according to the predicted trajectory. To verify the generalization ability of the algorithm, feature point 3 with a negative AOA is selected for analysis. Table 4 lists the design results and stability margin of the TD3 algorithm for the autopilot control parameters at the feature points.Table 3
Flight state at three different feature points.
H (m)Maα (°)Feature point 16000.56Feature point 22000.42Feature point 316000.6-2Table 4
Design results of the TD3 algorithm at the feature points.
Feature point 1Feature point 2Feature point 3kA0.0015900.0024850.001822kg0.103440.121090.108623kdc3.66463.53543.5850hm8.66948.50048.5971γm79.975879.789879.8691According to Table4, the autopilot design results at the feature points met the design requirements of the amplitude and phase margins. Subsequently, the pole placement [27] method is used to determine the autopilot control parameters at the feature points, the damping coefficient μ is taken as 0.7, and the natural frequency ω is taken as 35 rad/s. The design results are listed in Table 5.Table 5
Pole assignment results at the feature points.
Feature point 1Feature point 2Feature point 3kA0.0038240.006630.004038kg0.084710.103980.08859kdc2.07891.93122.1368hm3.60453.18593.8796γm59.042455.609260.9565The autopilot actuator adopts the second-order actuator model:(28)Wδs=ωact2s2+2μactωact+ωact2,where ωact=220rad/s represents the actuator natural frequency and μact=0.65 is the damping coefficient.The design results of the control parameters at the feature points are incorporated into the two-loop autopilot model, and Figure10 depicts the step response curves. According to the simulation results, the performance of the autopilot designed by the TD3 algorithm was superior to that obtained by the conventional pole placement method. Under the RL framework, the autopilot designed by the TD3 algorithm can obtain the control parameters that satisfy the ideal performance indices through autonomous learning, and the actuator’s performance requirements are more reasonable.Figure 10
Step response curves at the feature points.
(a)
Autopilot step response curve at the feature points(b)
Actuator response curve at the feature points


### 4.2. Online Application of the Autopilot Parameter Fitting Model

After offline training of the TD3 algorithm, the MLP neural network fitting model with the flight stateH,Ma,a as input and the autopilot control parameters kA,kg as the output in the full flight envelope is obtained by the actor-network. This model can be directly incorporated into the guidance control loop to automatically adjust the autopilot control parameters based on the current flight state. To further validate the effectiveness and adaptability of the proposed method, a flight experiment was designed with the guidance and impact angle constraint control problem in mind. The three-DOF motion equation in the plumb plane from Ref. [35] is chosen as the model for the theoretical flight simulation. To ensure the end impact angle, proportional guidance plus an offset term is required to meet the end impact angle constraint:
(29)ayc=Nyq̇+abV,where ab represents the control item to be designed and Ny is the proportional coefficient, for which Ny≥2. The following relationship can be easily obtained:
(30)θ̇=Nyq̇+ab.Assuming thattf is the end time of guidance, Equation (30) can be integrated on the interval t,tf, and the following relationship can be obtained:
(31)θtf=θt+Nqtf−qt+∫ttfabdt.Then,θd is set to a specified impact angle, and it must meet
(32)θtf=qtf=θd.From Equations (31) and (32), the following relationship can be deduced:
(33)∫tftabdt=N−1θd+θt−Nqt.Consequently,ηt=∫tftabdt and ηtf=0 can be obtained. It is known that, as long as η converges to zero during the guidance process, the impact angle requirements can be met. Therefore, the following equation can be formulated [36]:
(34)η̇=−KVcosφRη,where K is an adjustable parameter, K≥0. Based on Equation (33), the following relationship can be obtained:
(35)η̇=θ̇−Nyq̇=aycVy−Nyq̇.The offset proportional guidance relationship with the impact angle constraint can be obtained by combining Equations (34) and (35):
(36)ayc=NyVq̇−KV2cosφRη.Based on the structure of the two-loop autopilot, the expression for the actuator command can be deduced as follows:(37)δzc=−ayc−ayKA−kgωz.Set the initial missile position to (X0 = 0 m, Y0 = 500 m), the initial pitch angle to ϑ0 = 0°, and the initial velocity to V0 = 200 m/s. Set the target position to (Xt = 2000 m, Yt = 0 m), Ny =3, K = 3, and the acceleration command is constrained to [-50 m/s2, 50 m/s2]. The desired attack angles are set to -15°, -45°, and -80°, respectively. The simulation results are obtained and exhibited in Figure 11.Figure 11
Control ballistic simulation results.
(a)
Flight trajectory curve(b)
Speed change curve(c)
AOA curve(d)
Trajectory inclination curve(e)
Elevator deflection variation curve(f)
Acceleration tracking curveAccording to simulation results, the missile can consistently strike the target at the preset impact angle. Figure11(f) illustrates that during the flight process, the control parameters obtained by the online adjustment of the real-time flight state via the neural network fitting model are able to effectively implement acceleration tracking. This indicates that the autopilot parameter fitting model trained by the TD3 algorithm is robust and capable of completing the flight task specified.Figure12 depicts the change processes of the control parameters and stability margins.Figure 12
Variation curves of control parameters and stability margins.
(a)
ka variation curves(b)
kg variation curves(c)
Amplitude margin variation curve(d)
Phase margin variation curveFigure12 demonstrates that the autopilot parameter fitting model trained by the TD3 algorithm can self-adjust the autopilot control parameters online based on the real-time flight state, ensuring that the amplitude margin hm and phase margin γm of the control system are within the expected range. In addition, it has strong generalizability and can be implemented in the guidance control loop to implement acceleration instruction tracking.


## 4.1. Autopilot Design Simulation Based on the TD3 Algorithm

First, the design state space of the autopilot parameters is determined:S:H,Ma,αT, H∈0m,3000m, Ma∈0.3,1, and α∈−12°,12°. Then, the design action space of the autopilot parameters is set as follows: at:kA,kgT, kA∈0,0.01, and kg∈0,0.5.Subsequently, the expected performance indices are selected,hm0=8.5dB, γm0=80°, k1=0.5, and k2=0.5, and the hyperparameters are set according to Table 2 for offline training. The change process of the episode cumulative reward during the offline training process is exhibited in Figure 9.Figure 9
Episode cumulative reward change curve.It can be seen that compared with random sampling, sequential sampling improved the strategy learning efficiency of the algorithm, and the stable value of cumulative reward was higher than random sampling. This is because random sampling in the training process leads to no correlation in state transfer, which makes the agent unable to determine how the state transfers through learning, so that there will be a large difference in each calculation of theQ value, and it is difficult to obtain a stable Q value. However, with sequential sampling, the state transition is determined, which greatly reduces the learning difficulty of the agent and enables faster convergence of cumulative reward. In the early stages of training, the cumulative reward value was small, with relatively large fluctuation due to the random selection of actions within the set range. Nevertheless, as the training progressed, the agent gradually chose better actions; as a result, the cumulative reward value gradually increased, and the fluctuation decreased. At 600 epochs of training, the cumulative reward value of sequential sampling was greater than -60 but still exhibited an upward trend. When the training reached 1550 epochs, the cumulative reward value was stable at approximately -9.9, with a small fluctuation range. It can be considered that the training process achieved the ideal results.The design results of three characteristic points in the flight envelope are selected for analysis.The feature points 1 and 2 in Table3 are selected according to the predicted trajectory. To verify the generalization ability of the algorithm, feature point 3 with a negative AOA is selected for analysis. Table 4 lists the design results and stability margin of the TD3 algorithm for the autopilot control parameters at the feature points.Table 3
Flight state at three different feature points.
H (m)Maα (°)Feature point 16000.56Feature point 22000.42Feature point 316000.6-2Table 4
Design results of the TD3 algorithm at the feature points.
Feature point 1Feature point 2Feature point 3kA0.0015900.0024850.001822kg0.103440.121090.108623kdc3.66463.53543.5850hm8.66948.50048.5971γm79.975879.789879.8691According to Table4, the autopilot design results at the feature points met the design requirements of the amplitude and phase margins. Subsequently, the pole placement [27] method is used to determine the autopilot control parameters at the feature points, the damping coefficient μ is taken as 0.7, and the natural frequency ω is taken as 35 rad/s. The design results are listed in Table 5.Table 5
Pole assignment results at the feature points.
Feature point 1Feature point 2Feature point 3kA0.0038240.006630.004038kg0.084710.103980.08859kdc2.07891.93122.1368hm3.60453.18593.8796γm59.042455.609260.9565The autopilot actuator adopts the second-order actuator model:(28)Wδs=ωact2s2+2μactωact+ωact2,where ωact=220rad/s represents the actuator natural frequency and μact=0.65 is the damping coefficient.The design results of the control parameters at the feature points are incorporated into the two-loop autopilot model, and Figure10 depicts the step response curves. According to the simulation results, the performance of the autopilot designed by the TD3 algorithm was superior to that obtained by the conventional pole placement method. Under the RL framework, the autopilot designed by the TD3 algorithm can obtain the control parameters that satisfy the ideal performance indices through autonomous learning, and the actuator’s performance requirements are more reasonable.Figure 10
Step response curves at the feature points.
(a)
Autopilot step response curve at the feature points(b)
Actuator response curve at the feature points


## 4.2. Online Application of the Autopilot Parameter Fitting Model

After offline training of the TD3 algorithm, the MLP neural network fitting model with the flight stateH,Ma,a as input and the autopilot control parameters kA,kg as the output in the full flight envelope is obtained by the actor-network. This model can be directly incorporated into the guidance control loop to automatically adjust the autopilot control parameters based on the current flight state. To further validate the effectiveness and adaptability of the proposed method, a flight experiment was designed with the guidance and impact angle constraint control problem in mind. The three-DOF motion equation in the plumb plane from Ref. [35] is chosen as the model for the theoretical flight simulation. To ensure the end impact angle, proportional guidance plus an offset term is required to meet the end impact angle constraint:
(29)ayc=Nyq̇+abV,where ab represents the control item to be designed and Ny is the proportional coefficient, for which Ny≥2. The following relationship can be easily obtained:
(30)θ̇=Nyq̇+ab.Assuming thattf is the end time of guidance, Equation (30) can be integrated on the interval t,tf, and the following relationship can be obtained:
(31)θtf=θt+Nqtf−qt+∫ttfabdt.Then,θd is set to a specified impact angle, and it must meet
(32)θtf=qtf=θd.From Equations (31) and (32), the following relationship can be deduced:
(33)∫tftabdt=N−1θd+θt−Nqt.Consequently,ηt=∫tftabdt and ηtf=0 can be obtained. It is known that, as long as η converges to zero during the guidance process, the impact angle requirements can be met. Therefore, the following equation can be formulated [36]:
(34)η̇=−KVcosφRη,where K is an adjustable parameter, K≥0. Based on Equation (33), the following relationship can be obtained:
(35)η̇=θ̇−Nyq̇=aycVy−Nyq̇.The offset proportional guidance relationship with the impact angle constraint can be obtained by combining Equations (34) and (35):
(36)ayc=NyVq̇−KV2cosφRη.Based on the structure of the two-loop autopilot, the expression for the actuator command can be deduced as follows:(37)δzc=−ayc−ayKA−kgωz.Set the initial missile position to (X0 = 0 m, Y0 = 500 m), the initial pitch angle to ϑ0 = 0°, and the initial velocity to V0 = 200 m/s. Set the target position to (Xt = 2000 m, Yt = 0 m), Ny =3, K = 3, and the acceleration command is constrained to [-50 m/s2, 50 m/s2]. The desired attack angles are set to -15°, -45°, and -80°, respectively. The simulation results are obtained and exhibited in Figure 11.Figure 11
Control ballistic simulation results.
(a)
Flight trajectory curve(b)
Speed change curve(c)
AOA curve(d)
Trajectory inclination curve(e)
Elevator deflection variation curve(f)
Acceleration tracking curveAccording to simulation results, the missile can consistently strike the target at the preset impact angle. Figure11(f) illustrates that during the flight process, the control parameters obtained by the online adjustment of the real-time flight state via the neural network fitting model are able to effectively implement acceleration tracking. This indicates that the autopilot parameter fitting model trained by the TD3 algorithm is robust and capable of completing the flight task specified.Figure12 depicts the change processes of the control parameters and stability margins.Figure 12
Variation curves of control parameters and stability margins.
(a)
ka variation curves(b)
kg variation curves(c)
Amplitude margin variation curve(d)
Phase margin variation curveFigure12 demonstrates that the autopilot parameter fitting model trained by the TD3 algorithm can self-adjust the autopilot control parameters online based on the real-time flight state, ensuring that the amplitude margin hm and phase margin γm of the control system are within the expected range. In addition, it has strong generalizability and can be implemented in the guidance control loop to implement acceleration instruction tracking.


## 5. Conclusions

In this paper, the TD3 algorithm is used to design a two-loop autopilot for the full flight envelope that can be deployed directly to the guidance control loop, allowing online self-adjustment of the control parameters. First, a deep reinforcement learning model is constructed, with the flight stateH,Ma,αT taken as the state and the control parameters to be designed kA,kgT taken as the actions. According to the amplitude and phase margins in the frequency domain index of the control system, a reasonable reward mechanism is designed. The TD3 algorithm is then used offline to learn the control parameters of the complete flight envelope, and an MLP neural network fitting model is obtained. Finally, a verification study for the designed autopilot is conducted in combination with the impact angle constraint guidance problem. The results indicate that the TD3 algorithm-based autopilot can satisfy the performance index requirements. The fitting model can self-adjust the control parameters of the autopilot based on the real-time flight state, ensuring that the control system’s stability margin is within the expected range and can accurately track acceleration. In addition, the fitting model has strong generalization capability and robustness, eliminating the robustness problem of the conventional overload autopilot, which is insufficiently robust to meet the performance requirements for the entire flight envelope. The method proposed in this paper for designing two-loop autopilots based on the TD3 algorithm is also applicable to the design of three-loop autopilots.



---

*Source: 5759135-2023-03-23.xml*