# Attitude Control with Auxiliary Structure Based on Adaptive Dynamic Programming for Reentry Vehicles

Authors: Xu Li; Zhongtao Cheng; Bo Wang; Yongji Wang; Lei Liu
Journal: Complexity
 (2020)

---

## Abstract

This paper presents an attitude control scheme combined with adaptive dynamic programming (ADP) for reentry vehicles with high nonlinearity and disturbances. Firstly, the nonlinear attitude dynamics is divided into inner and outer loops according to the time scale separation and the cascade control principle, and a general sliding mode control method is employed to construct the main controllers for the double loops. Considering the shortage of main controllers in handling nonlinearity and sudden disturbances, an ADP structure is introduced into the outer attitude loop as an auxiliary. And the ADP structure utilizes neural network estimators to minimize the cost function and generate optimal signals through online learning, so as to compensate defect of the main controllers’ adaptability speed and accuracy. Then, the stability is analyzed by the Lyapunov method, and the parameter selection strategy of the ADP structure is derived to guide implementation. In addition, this paper puts forward skills to speed up ADP training. Finally, simulation results show that the control strategy with ADP possesses stronger adaptability and faster response than that without ADP for the nonlinear vehicle system.


---

## Body

## 1. Introduction

Attitude control for reentry vehicles has been a hotspot in the field of aerospace. The complex operating conditions and the high nonlinearity of vehicles themselves bring great challenges to attitude control. Fortunately, around these focuses, researchers continue to explore and ameliorate control schemes, developing a series of available control technologies.For the control of space vehicles, some schemes have been investigated one after another. Some linear control methods, such as linear parameter varying (LPV) [1] and linear quadratic regulator (LQR), focus on linearizing the aircraft model. However, due to the highly nonlinear and coupling dynamic characteristics, to be honest, the capabilities of these linear control methods on actual nonlinear coupling vehicles are limited. Besides, some nonlinear control methods are widely employed, such as nonlinear dynamic inversion [2], sliding mode control, and backstepping method [3, 4]. Although these nonlinear control techniques can also effectively deal with the nonlinear nature of vehicles, they will still be slightly embarrassed and lack adaptability in the face of complex and changeable disturbances if without other auxiliary means. Therefore, in the recent development of vehicle control, more and more adaptive technologies have been favoured by researchers [5].For the purpose of ameliorating the robustness of the controller by designing adaptive mechanism, observer-based adaptive control technology and other intelligent methods (as adaptive fuzzy control and iterative learning) have emerged one after another [6–8]. Especially, in recent years, thanks to the vigorous development of new artificial intelligence, reinforcement learning (RL) has attracted more and more attention, which has shown strong performance in solving adaptive and optimal control problems [9–11]. In the control domain, reinforcement learning is transformed into approximate or adaptive dynamic programming (ADP), which learns by interacting with the environment to determine what optimal actions to take to minimize a cost function over a period of time [12]. One of the core approaches is the critic-action (CA) design, which approximates the cost function and obtains the optimal actions by solving the Hamilton–Jacobi–Bellman equation with function estimators [13]. ADP contains a variety of structural classifications, including heuristic [14], dual heuristic [15], and action-dependent dynamic programming (ADHDP), etc., which have been made preliminary explorations and achievements in the field of vehicle control [16]. Specifically, Luo et al. developed a direct heuristic dynamic programming (dHDP) for longitudinal control of hypersonic vehicles and introduced fuzzy neural networks to enhance the learning ability and robustness of dHDP [17]. There is also an application of ADDHP to study the optimal control of attitude maneuver for three-axis spacecraft [18]. Some creative researchers improve ADP by redefining the two optimization objectives and apply ADP to the in-orbit reconfiguration of the vehicle attitude system under multitask constraints through dual optimization indexes [19]. Moreover, ADP can be associated with traditional methods, such as nonlinear filter [20] and sliding mode control [21], to implement a data-driven ADHDP auxiliary control scheme for the speed and altitude system of an air-breathing hypersonic vehicle [21]. In [22], a switching adaptive active anti-interference control technique based on reduced-order observer technique and ADP is proposed, considering the parameter uncertainty and external disturbance of variable structure near-space vehicles. Furthermore, aiming at the guidance and control problem of the vertical take-off and landing (VTOL) system with multivariable disturbances, an online kernel DHP robust control strategy based on the sparse kernel theory is designed for VTOL vehicles [23]. Most of the above control strategies with ADP utilize neural network estimators to approximate the cost function and optimal control law online, while Zhou et al. creatively put forward an incremental ADP (iADP) combining the advantages of the incremental control method and ADP [24]. This iADP is based on Markov decision-making process and Bellman optimal principle to directly derive the explicit expression of optimal control law, greatly simplifying the design process of ADP, and successfully exploited to satellite [25] and aircraft [26]. Similarly, Sun and van Kampen also come up with an incremental model-based DHP technology for vehicle control, replacing the model network in traditional DHP with an incremental model [27, 28].In a word, the development of ADP in the field of vehicle control is rapidly deepening and expanding [16], but as far as the current literature is concerned, ADP is still rarely applied to the control of all three channels’ attitude angles of the vehicle. Moreover, most of the literature rarely mentions the internal weight convergence, parameter selection, training speed, and other issues of ADP based on critic-action networks, but these are problems to be concerned about. Therefore, this paper contributes to employ the ADP framework to the control of all three-channel attitude angles of a reentry vehicle. Inspired by the ADP as an auxiliary controller [21], this paper presents a framework combining conventional controller and ADP, and ADP is as the auxiliary means to enhance the rapidity and adaptivity of the whole attitude system. In addition, the internal convergence of the ADP structure and its parameter selection rules are discussed in depth. Aiming at the implementation problem, this paper considers the improvement measures to speed up ADP training, which will be provided to interested researchers for future discussion.The rest of this paper is organized as follows. Firstly, the nonlinear dynamics of the three-channel attitude control system of the reentry vehicle is established in Section2. Then, in Section 3, the control strategy based on the dual-loop main controller plus ADP is elaborated in detail. In Section 4, some issues about implementation are taken into consideration. Finally, the simulations and conclusions are presented in Sections 5 and 6, respectively.


## 2. Nonlinear Model

To describe the attitude change of the reentry phase, we give the rotation equations of the vehicle around the center of mass, including rotation dynamics and attitude kinematics. They determine the attitude angles of the vehicle around the center of mass and the angular rate of the three channels during the flight. Considering the influence of Earth rotation on attitude control, a three degree of freedom nonlinear attitude model in the body coordinate system can be obtained [29]:(1)α˙=−pcosαtanβ+q−rsinαtanβ+sinμcosβ−ϕ˙sinχsinϑ+χ˙cosϑ+θ˙+ΩEcosϕcosχsinϑ−sinϕcosϑ−cosμcosβϑ˙−ϕ˙cosχ−θ˙+ΩEcosϕsinχ,β˙=psinα−rcosα+sinμϑ˙−ϕ˙cosχ+θ˙+ΩEcosϕsinχ+cosμ−θ˙+ΩEcosϕcosχsinϑ−sinϕcosϑ+χ˙cosϑ−ϕ˙sinχsinϑ,μ˙=−pcosαcosβ−qsinβ−rsinαcosβ+α˙sinβ−χ˙sinϑ−ϕ˙sinχcosϑ+θ˙+ΩEsinϕsinϑ+cosϕcosχcosϑ,p˙=I=MxIxxIzz−Ixz2+IxzMzIxxIzz−Ixz2+Ixx−Iyy+IzzIxzIxxIz−Ixz2pq+Iyy−IzzIzz−Ixz2IxxIzz−Ixz2qr,q˙=MyIyy+IxzIyyr2−p2+Izz−IxxIyypr,r˙=IxzMxIxxIzz−Ixz2+IxxMzIxxIzz+Ixx−IyyIxx−Ixz2IxxIzz−Ixz2pq+Iyy−Ixx−IzzIxzIxxIzz−Ixz2qr,where α,β,andμ represent the angle of attack, sideslip, and bank angle, respectively; p,q,andr are the roll, pitch, and yaw rate, respectively. And Mx,My,andMz denote the roll, pitch, and yaw control torques, respectively; Iiji=x,y,z;j=x,y,z is rotational inertia. ϕ,θ,χ,andϑ are longitude, latitude, heading angle, and flight path angle, respectively; ΩE is the Earth rotation angular velocity.In actual control, vehicles can be regarded as an ideal rigid body. Considering that the rotation rate of the Earth is far less than that of vehicles, the rotation of the Earth is ignored. Besides, orbital motion is much slower than attitude motion, so the orbital motion terms of vehicles are described asϕ˙=θ˙=ϑ˙=χ˙=0. Finally, simplified dynamics can be obtained:(2)α˙=−pcosαtanβ+q−rsinαtanβ,β˙=psinα−rcosα,μ˙=−pcosαcosβ−qsinβ−rsinαcosβ.Above attitude kinematics equation (2) is abbreviated as(3)γ˙=Γ⋅ω,where γ=α,β,μT∈R3 and ω=p,q,rT∈R3. Γ∈R3×3 are defined as(4)Γ=−cosαtanβ1−sinαtanβsinα0−cosα−cosαcosβ−sinβ−sinαcosβ.Similarly, rotational dynamics can be simplified as(5)ω˙=−I−1ΩIω+I−1Mc,where I∈R3×3 denotes inertial matrix; Mc=Mx,My,MzT∈R3 is a vector of control torques. Ω∈R3×3 and I∈R3×3 are defined as(6)I=Ixx−Ixy−Ixz−IxyIyy−Iyz−Ixz−IyzIzz,Ω=0−rqr0−p−qp0.If there exist external disturbances,d1 and d2 are introduced into the vehicle system as follows:(7)γ˙=Γ⋅ω+d1,ω˙=−I−1ΩIω+I−1Mc+d2,where d1∈R3 and d2∈R3 represent external disturbances.Obviously, the attitude tracking control problem of the reentry vehicles can be described as(8)limt⟶∞α−αd=0,limt⟶∞β−βd=0,limt⟶∞μ−μd=0.


## 3. Controller Design

In the previous section, the nominal attitude model of the reentry vehicle has been established by equations (3) and (5), which can be reorganized as equations (9a) and (9b). This section will devise a controller with an auxiliary according to this vehicle model:(9a)γ˙=Γ⋅ω,(9b)ω˙=−I−1ΩIω+I−1Mc.It is well known that the attitude angles change more slowly than the angular rate. Therefore, according to the principle of time scale separation and cascade control, equations (9a) and (9b) can be divided into attitude angle slow loop equation (9a) and angle rate fast loop equation (9b), also known as outer loop and inner loop, respectively. In this section, the ADP-based controller will be presented, and the overall control strategy is shown in Figure 1.Figure 1
Dual-loop control structure based on ADP.As shown in Figure1, there are two control loops. The outer loop is an attitude control loop with two controllers. The controller 1 generates the main angular rate instruction ωs according to the guidance instruction γd, and the ADP controller outputs the control instruction uADP according to the attitude angle error; both of which together yield the angular rate ωc. Then, ωc is a reference instruction for the inner angular rate loop so that the controller 2 of the inner loop generates the control torque Mc, which acts on the vehicle to output the actual attitude angles and complete the control task.In this paper, the inner controller 1 and outer loop controller 2 are implemented based on conventional sliding mode control and serve as the main controllers. To increase the performance of the main controller of the outer loop, the ADP controller acts as an auxiliary and adopts an action-dependent structure such as ADHDP. Note that ADHDP belongs to the category of ADP, so it is called ADP in this paper. The output of the ADP serves as a supplementary reference signal for the inner loop. The focus of this paper is to discuss the auxiliary role of ADP structure. Of course, the main controllers can also choose other methods to design, but how to select the main controllers is not the focus of this paper. It should be pointed out that only the ADP auxiliary controller is introduced into the outer loop, mainly because the outer loop variable is the attitude angle and the inner loop variable is the angular rate, and the attitude angle changes slowly than the angular rate. Therefore, in each iteration, the iterative speed of the ADP is more easily matched with the update speed of the main controller 1. Perhaps we can similarly introduce the ADP auxiliary controller with the same structure into the inner loop, and its rationality and effectiveness will be researched and verified in future work.In the following subsections: according to cascade control strategy, the outer loop controllers are first designed, including the main controller 1 and the ADP-based auxiliary controller. After the reference command signalωc is obtained by the outer loop controllers, the inner loop controller 2 is presented.


### 3.1. Outer Loop Controllers

#### 3.1.1. Main Controller 1

The control objective of the outer loop is to operate the actual attitude angleγ to track γd within the desired accuracy. First, take the tracking error eγ=γd−γ∈R3. The sliding switching surface Sγ∈R3 of the outer loop can be selected as(10)Sγ=Sγ1,Sγ2,Sγ3T=eγ+ργ∫0teγdτ,where ργ=diagργ1,ργ2,ργ3∈R3×3 and ργi>0,i=1,2,3 are the parameters to be designed [30]. Obviously, on the sliding surface Sγ=0, the tracking error eγ can be guaranteed to converge uniformly, that is,(11)Sγ=eγ+ργ∫0teγdτ=0.In order to ensure the asymptotic convergence of the outer loop tracking error to the sliding surface, the virtual control law must be designed. First, take the derivative ofSγ as(12)S˙γ=e˙γ+ργeγ=γ˙d−γ˙+ργeγ=γ˙d−Γω+ργeγ.Take the following Lyapunov function:(13)L1=12SγTSγ>0,and the derivative of L1 is as(14)L˙1=SγTS˙γ.By Lyapunov stability,L˙1<0 has to be guaranteed. Therefore, the sliding mode approach law can be chosen as(15)S˙γ=−τγsignSγ,where designed parameter τγ>0 and signSγ=signSγ1,signSγ2,signSγ3T denotes a sign function.According to equations (12) and (15), there exists(16)γ˙d−Γω+ργeγ=−τγsignSγ.So, the virtual control law of the outer loop can be obtained as follows:(17)ωs=Γ−1γ˙d+ργeγ+τγsignSγ.In order to avoid or reduce the sliding mode chattering caused by the sign function in equation (17), a smooth continuous function can be adopted instead of the sign function. Because the saturation function is one of the most simple and effective ways, the virtual control law is redesigned as follows:(18)ωs=Γ−1γ˙d+ργeγ+τγsatSγξγ,where satSγ/ξγ=satSγ1/ξγ,satSγ2/ξγ,satSγ3/ξγT denotes a saturation function with width ξγ>0 as follows:(19)satSγiξγi=1,2,3=1,Sγi>ξγ,Sγiξγ,Sγi≤ξγ,−1,Sγi<−ξγ.Therefore, according to control law equation (18), the attitude angles can track the commands, and the error eγ uniformly converges. Next, ωs will be provided as the main reference signal to the inner loop.


#### 3.1.2. ADP Auxiliary Controller

The idea of ADP is to take advantage of the function estimators to approximate the performance index functions and control strategies that meet the principle of optimality. By designing a critic-action structure, the critic network approximates the performance indexJ (the cost function) and J is defined as the forward accumulation of the utility function U with the discount factor λ [20, 21]:(20)Jk=∑i=k∞λi−kUi,where U is usually defined as a quadratic. It can be seen that the cost function is also a quadratic convex function, with only a local minimum and at the same time a global minimum. The action network obtains the optimal control law u∗ by minimizing J:(21)u∗k=argminukUk+λJk+1.In this paper, only the auxiliary ADP controller is added to the outer loop to compensate for the attitude angle error generated by the main controller 1. ADP outputsuADP (uADP has the same dimension as ωs), and the sum of uADP and ωs inputs as a reference instruction to the inner loop. Obviously, the ADP controller is sensitive to the attitude angle error. It can be imagined that ADP will start to work when a certain error occurs; when the error meets the threshold requirements, the ADP does not need to work, which will balance the loss in accuracy and calculation speed. However, this does not seem to be the focus of this paper. It may be discussed in future research, such as the selection and optimization of the threshold.In Figure2, ADP adopts a network structure based on ADHDP, which includes an action network, a critic network, and attitude model (9a). The input of ADP is the attitude error, and the action network generates the control signal uADP. At the same time, the critic network approximates J. The specific design of each network is given below.Figure 2
Structure of the ADP controller.(1) Critic Network. In Figure 3, the critic network uses a single hidden-layer BP neural network with six input nodes, M hidden nodes, and one output node. The input contains the attitude angle error Δγ and uADP generated by the action network. The output is the estimated J^ of the cost function J. Wc1∈RM×6 is the weight matrix of the input layer to the hidden layer and Wc1jii=1,…,6;j=1,…,M represents the weight of the i-th input node to the j-th hidden node. Wc2∈R1×M is the weight matrix from the hidden-to-output layer, and Wc2j,j=1,…,M represents the connection weight of the j-th hidden node to the output. Ch1∈RM×1 and Ch2∈RM×1 are the input and output vectors of hidden nodes, respectively. The active functions of the hidden layer and the output layer are a bipolar sigmoid function and linear function, respectively. The attitude error is as follows:(22)Δγ=Δα,Δβ,ΔμT=αd−α,βd−β,μd−μT.Figure 3
Structure of the critic network.The input of the critic network isINc∈R6×1 as(23)INc=Δγ;ΔuADP=Δα,Δβ,Δμ,uADP1,uADP2,uADP3T.The training of the critic network consists of two parts, one is the forward calculation, and the other is the error backpropagation of updating network weights. The forward process of stepk is(24)Ch1jk=∑i=16INcik⋅Wc1jik,j=1,2,…,M,Ch2jk=1−e−Ch1jk1+e−Ch1jk,j=1,2,…,M,J^k=∑j=1MCh2jk⋅Wc2jk,j=1,2,…,M.Equation (24) can be rewritten in matrix form as(25)Ch1k=Wc1⋅INc,Ch2k=1−e−Ch1k1+e−Ch1k,J^k=Wc2k⋅Ch2k.Based on the Bellman optimality principle, the critic network approximates the cost function of the system. The actualJk is defined as the cumulative return from the current state to the future:(26)Jk=∑i=k∞λi−kUi,where λ∈0,1 is a discount factor or forgetting factor, indicating the influence of the future state on the current strategy. U is the utility function at each step, which is defined as a quadratic:(27)Uk=ΔγTkΛΔγk,Λ=diagσ,σ,σ,σ>0.The following errorEc can be defined, and the critic network can approximate J by minimizing Ec:(28)Ec=12ec2,ec=J^k−Uk+1−λJ^k+1.Therefore, network weights can be updated through backpropagation ofEc.(2) Updating the Weights Wc2. Using the gradient descent method, let ΔWc2 be the gradient, so(29)Wc2k+1=Wc2k+ΔWc2k,where each component of ΔWc2 is represented as(30)ΔWc2jk=ζck⋅−∂Eck∂Wc2jk,j=1,…,M=−ζck∂Eck∂J^k∂J^k∂Wc2jk,j=1,…,M=−ζck⋅eck⋅Ch2jk,j=1,…,M,where ζck∈0,1 is the learning rate. Equation (30) is combined and rewritten into a matrix form as(31)ΔWc2k=−ζck⋅eck⋅Ch2Tk.(3) Updating the Weights Wc1. Similarly, let ΔWc1 be the gradient, so(32)Wc1k+1=Wc1k+ΔWc1k,ΔWc1jik=ζck⋅−∂Eck∂Wc1jik,i=1,…,6j=1,…,M=−ζck∂Eck∂J^k∂J^k∂Ch2jk∂Ch2jk∂Ch1jk∂Ch1jk∂Wc1jik=−ζck⋅eck⋅Wc2jk⋅121−Ch2j2k⋅INcik.Combine the above formula into a simplified matrix form as follows:(33)ΔWc1k=−12⋅ζck⋅eck⋅Wc2Tk×1−Ch2k×Ch2k⋅INcTk,where the symbol “×” represents the Hadamard product of two matrices, that is, bitwise multiplication; “⋅” represents the ordinary multiplication of matrices. These symbols appearing in the later parts of this paper possess the same meaning.(4) Action Network. As shown in Figure 4, the action network adopts a single hidden-layer BP neural network with three input nodes, N hidden nodes, and three output nodes. The network’s input is INa=Δγ∈R3×1, and output is uADP∈R3×1. Other parameters are defined similarly to the critic network. The active functions of the hidden and output layer are a bipolar sigmoid function and linear function, respectively.Figure 4
Structure of the action network.The training of the action network also includes forward calculation and error backpropagation. Firstly, the forward process is briefly presented as(34)Ah1k=Wa1⋅INa,Ah2k=1−e−Ah1k1+e−Ah1k,uADPk=Wa2k⋅Ah2k.The action network generates an optimal control strategy by minimizing the system cost functionJ. This goal can be achieved by minimizing the defined error Ea:(35)Eak=12ea2k,eak=J^k.(5) Updating the Weights Wa2. With the gradient descent method, the update process of Wa2 is(36)Wa2k+1=Wa2k+ΔWa2k,ΔWa2k=ζak⋅−∂Eak∂Wa2k,where ζak represents the learning rate. The connection weight from the j-th hidden node to the ith output node is denoted as Wa2iji=1,2,3;j=1,…,N, so(37)Wa2ijk+1=Wa2ijk+ΔWa2ijk,i=1,2,3j=1,…,N,(38)ΔWa2ijk=ζak⋅−∂Eak∂eak∂eak∂J^k∂J^k∂Wa2ijk=ζak⋅−eak∂J^k∂uADPik∂uADPik∂Wa2ijk=−ζak⋅eak⋅∂J^k∂uADPik⋅Ah2jk.The middle term∂J^k/∂uADPik in equation (38) indicates that the path of the backpropagated signal passes through the critic network when training the action network [31]. Furthermore, by the output and input of the critic network, ∂J^k/∂uADPik can be obtained:(39)∂J^k∂uADPk=∂J^k∂Ch2k∂Ch2k∂Ch1k∂Ch1k∂uADPk.So,(40)∂J^k∂uADPik=∑j=1M∂J^k∂Ch2jk∂Ch2jk∂Ch1jk∂Ch1jk∂uADPik,i=1,2,3=∑j=1MWc2jk⋅121−Ch2j2k⋅Wc1j,i+3k=12⋅Wc1:,i+3Tk⋅Wc2Tk×1−Ch2k×Ch2k,where Wc1:,i+3 represents the (i + 3)-th column of Wc1. Equation (40) can be rewritten in matrix form:(41)∂J^k∂uADPk=12⋅Wc1uADPTk⋅Wc2Tk×1−Ch2k×Ch2k,where Wc1uADP=Wc1:,4:6 represents columns 4 to 6 of Wc1, that is, the connection weights of the three input nodes corresponding to uADP and all hidden nodes in the critic network. From equations (37)–(41), ΔWa2 can be deduced as(42)ΔWa2k=ζak⋅−∂Eak∂eak∂eak∂J^k∂J^k∂Wa2k=−ζak⋅eak⋅∂J^k∂uADPk⋅Ah2Tk=−12⋅ζak⋅eak⋅Wc1uADPTk⋅Wc2Tk×1−Ch2k×Ch2k⋅Ah2Tk.(6) Updating the Weights Wa1. Similar to the Wa2, the update of Wa1 is(43)Wa1k+1=Wa1k+ΔWa1k,(44)ΔWa1k=ζak⋅−∂Eak∂Wa1k=ζak⋅−∂Eak∂eak∂eak∂J^k∂J^k∂uADPk∂uADPk∂Ah2k∂Ah2k∂Ah1k∂Ah1k∂Wa1k=−ζak⋅eak⋅Wa2Tk⋅∂J^k∂uADPk×12⋅1−Ah2k×Ah2k⋅INaTk.Substituting equation (41) into equation (44), ΔWa1 can be easily obtained.So far, the training process is completed. And the optimal control signaluADP output by the action network will be combined with ωs output by outer loop main controller 1, that is(45)ωc=ωs+uADP,where the angular rate signal ωc∈R3×1 will be input as the reference command of the inner loop controller 2, and the control torque Mc output by controller 2 will operate the vehicle to complete the attitude control task.


### 3.2. Inner Loop Controller

To ensure that the actual angular rateω can stably track the expected reference angular rate ωc, similar to controller 1, the sliding variable is selected for inner loop controller 2 as follows:(46)Sω=Sω1,Sω2,Sω3T=eω+ρω∫0teωdτ,where eω=ωc−ω∈R3×1 and ρω=diagρω1,ρω2,ρω3∈R3×3 with ρωi>0,i=1,2,3. In order to ensure the inner loop tracking error eω asymptotically converges to the sliding surface Sω=0, the actual control law M has to be designed.The derivative ofSω is(47)S˙ω=e˙ω+ρωeω=ω˙c+I−1ΩIω−I−1Mc+ρωeω.Take the following Lyapunov functionL2:(48)L2=12SωTSω>0,L˙2=SωTS˙ω.By Lyapunov stability,L˙2<0 has to be guaranteed. Therefore, the dynamics S˙ω can be chosen as(49)S˙ω=−τωsignSω,where designed parameter τω>0 and signSω=signSω1,signSω2,signSω3T denotes a sign function.According to equations (47) and (49), there exists(50)ω˙c+I−1ΩIω−I−1Mc+ρωeω=−τωsignSω.So, the actual control law of the inner loop can be obtained as follows:(51)Mc=Iω˙c+ΩIω+Iρωeω+τωI⋅signSω.Similarly, a continuous saturation function is chosen to replace the sign function to reduce the chattering. Therefore, the actual control law is rewritten as follows:(52)Mc=Iω˙c+ΩIω+Iρωeω+τωI⋅satSωξω.where satSω/ξω=satSω1/ξω,satSω2/ξω,satSω3/ξωT denotes a saturation function with width ξω>0.Therefore, for actual control law as equation (52), L˙2<0 holds. That is, the actual attitude angular rate ω converges asymptotically to the expected angular rate ωc.


## 3.1. Outer Loop Controllers

### 3.1.1. Main Controller 1

The control objective of the outer loop is to operate the actual attitude angleγ to track γd within the desired accuracy. First, take the tracking error eγ=γd−γ∈R3. The sliding switching surface Sγ∈R3 of the outer loop can be selected as(10)Sγ=Sγ1,Sγ2,Sγ3T=eγ+ργ∫0teγdτ,where ργ=diagργ1,ργ2,ργ3∈R3×3 and ργi>0,i=1,2,3 are the parameters to be designed [30]. Obviously, on the sliding surface Sγ=0, the tracking error eγ can be guaranteed to converge uniformly, that is,(11)Sγ=eγ+ργ∫0teγdτ=0.In order to ensure the asymptotic convergence of the outer loop tracking error to the sliding surface, the virtual control law must be designed. First, take the derivative ofSγ as(12)S˙γ=e˙γ+ργeγ=γ˙d−γ˙+ργeγ=γ˙d−Γω+ργeγ.Take the following Lyapunov function:(13)L1=12SγTSγ>0,and the derivative of L1 is as(14)L˙1=SγTS˙γ.By Lyapunov stability,L˙1<0 has to be guaranteed. Therefore, the sliding mode approach law can be chosen as(15)S˙γ=−τγsignSγ,where designed parameter τγ>0 and signSγ=signSγ1,signSγ2,signSγ3T denotes a sign function.According to equations (12) and (15), there exists(16)γ˙d−Γω+ργeγ=−τγsignSγ.So, the virtual control law of the outer loop can be obtained as follows:(17)ωs=Γ−1γ˙d+ργeγ+τγsignSγ.In order to avoid or reduce the sliding mode chattering caused by the sign function in equation (17), a smooth continuous function can be adopted instead of the sign function. Because the saturation function is one of the most simple and effective ways, the virtual control law is redesigned as follows:(18)ωs=Γ−1γ˙d+ργeγ+τγsatSγξγ,where satSγ/ξγ=satSγ1/ξγ,satSγ2/ξγ,satSγ3/ξγT denotes a saturation function with width ξγ>0 as follows:(19)satSγiξγi=1,2,3=1,Sγi>ξγ,Sγiξγ,Sγi≤ξγ,−1,Sγi<−ξγ.Therefore, according to control law equation (18), the attitude angles can track the commands, and the error eγ uniformly converges. Next, ωs will be provided as the main reference signal to the inner loop.


### 3.1.2. ADP Auxiliary Controller

The idea of ADP is to take advantage of the function estimators to approximate the performance index functions and control strategies that meet the principle of optimality. By designing a critic-action structure, the critic network approximates the performance indexJ (the cost function) and J is defined as the forward accumulation of the utility function U with the discount factor λ [20, 21]:(20)Jk=∑i=k∞λi−kUi,where U is usually defined as a quadratic. It can be seen that the cost function is also a quadratic convex function, with only a local minimum and at the same time a global minimum. The action network obtains the optimal control law u∗ by minimizing J:(21)u∗k=argminukUk+λJk+1.In this paper, only the auxiliary ADP controller is added to the outer loop to compensate for the attitude angle error generated by the main controller 1. ADP outputsuADP (uADP has the same dimension as ωs), and the sum of uADP and ωs inputs as a reference instruction to the inner loop. Obviously, the ADP controller is sensitive to the attitude angle error. It can be imagined that ADP will start to work when a certain error occurs; when the error meets the threshold requirements, the ADP does not need to work, which will balance the loss in accuracy and calculation speed. However, this does not seem to be the focus of this paper. It may be discussed in future research, such as the selection and optimization of the threshold.In Figure2, ADP adopts a network structure based on ADHDP, which includes an action network, a critic network, and attitude model (9a). The input of ADP is the attitude error, and the action network generates the control signal uADP. At the same time, the critic network approximates J. The specific design of each network is given below.Figure 2
Structure of the ADP controller.(1) Critic Network. In Figure 3, the critic network uses a single hidden-layer BP neural network with six input nodes, M hidden nodes, and one output node. The input contains the attitude angle error Δγ and uADP generated by the action network. The output is the estimated J^ of the cost function J. Wc1∈RM×6 is the weight matrix of the input layer to the hidden layer and Wc1jii=1,…,6;j=1,…,M represents the weight of the i-th input node to the j-th hidden node. Wc2∈R1×M is the weight matrix from the hidden-to-output layer, and Wc2j,j=1,…,M represents the connection weight of the j-th hidden node to the output. Ch1∈RM×1 and Ch2∈RM×1 are the input and output vectors of hidden nodes, respectively. The active functions of the hidden layer and the output layer are a bipolar sigmoid function and linear function, respectively. The attitude error is as follows:(22)Δγ=Δα,Δβ,ΔμT=αd−α,βd−β,μd−μT.Figure 3
Structure of the critic network.The input of the critic network isINc∈R6×1 as(23)INc=Δγ;ΔuADP=Δα,Δβ,Δμ,uADP1,uADP2,uADP3T.The training of the critic network consists of two parts, one is the forward calculation, and the other is the error backpropagation of updating network weights. The forward process of stepk is(24)Ch1jk=∑i=16INcik⋅Wc1jik,j=1,2,…,M,Ch2jk=1−e−Ch1jk1+e−Ch1jk,j=1,2,…,M,J^k=∑j=1MCh2jk⋅Wc2jk,j=1,2,…,M.Equation (24) can be rewritten in matrix form as(25)Ch1k=Wc1⋅INc,Ch2k=1−e−Ch1k1+e−Ch1k,J^k=Wc2k⋅Ch2k.Based on the Bellman optimality principle, the critic network approximates the cost function of the system. The actualJk is defined as the cumulative return from the current state to the future:(26)Jk=∑i=k∞λi−kUi,where λ∈0,1 is a discount factor or forgetting factor, indicating the influence of the future state on the current strategy. U is the utility function at each step, which is defined as a quadratic:(27)Uk=ΔγTkΛΔγk,Λ=diagσ,σ,σ,σ>0.The following errorEc can be defined, and the critic network can approximate J by minimizing Ec:(28)Ec=12ec2,ec=J^k−Uk+1−λJ^k+1.Therefore, network weights can be updated through backpropagation ofEc.(2) Updating the Weights Wc2. Using the gradient descent method, let ΔWc2 be the gradient, so(29)Wc2k+1=Wc2k+ΔWc2k,where each component of ΔWc2 is represented as(30)ΔWc2jk=ζck⋅−∂Eck∂Wc2jk,j=1,…,M=−ζck∂Eck∂J^k∂J^k∂Wc2jk,j=1,…,M=−ζck⋅eck⋅Ch2jk,j=1,…,M,where ζck∈0,1 is the learning rate. Equation (30) is combined and rewritten into a matrix form as(31)ΔWc2k=−ζck⋅eck⋅Ch2Tk.(3) Updating the Weights Wc1. Similarly, let ΔWc1 be the gradient, so(32)Wc1k+1=Wc1k+ΔWc1k,ΔWc1jik=ζck⋅−∂Eck∂Wc1jik,i=1,…,6j=1,…,M=−ζck∂Eck∂J^k∂J^k∂Ch2jk∂Ch2jk∂Ch1jk∂Ch1jk∂Wc1jik=−ζck⋅eck⋅Wc2jk⋅121−Ch2j2k⋅INcik.Combine the above formula into a simplified matrix form as follows:(33)ΔWc1k=−12⋅ζck⋅eck⋅Wc2Tk×1−Ch2k×Ch2k⋅INcTk,where the symbol “×” represents the Hadamard product of two matrices, that is, bitwise multiplication; “⋅” represents the ordinary multiplication of matrices. These symbols appearing in the later parts of this paper possess the same meaning.(4) Action Network. As shown in Figure 4, the action network adopts a single hidden-layer BP neural network with three input nodes, N hidden nodes, and three output nodes. The network’s input is INa=Δγ∈R3×1, and output is uADP∈R3×1. Other parameters are defined similarly to the critic network. The active functions of the hidden and output layer are a bipolar sigmoid function and linear function, respectively.Figure 4
Structure of the action network.The training of the action network also includes forward calculation and error backpropagation. Firstly, the forward process is briefly presented as(34)Ah1k=Wa1⋅INa,Ah2k=1−e−Ah1k1+e−Ah1k,uADPk=Wa2k⋅Ah2k.The action network generates an optimal control strategy by minimizing the system cost functionJ. This goal can be achieved by minimizing the defined error Ea:(35)Eak=12ea2k,eak=J^k.(5) Updating the Weights Wa2. With the gradient descent method, the update process of Wa2 is(36)Wa2k+1=Wa2k+ΔWa2k,ΔWa2k=ζak⋅−∂Eak∂Wa2k,where ζak represents the learning rate. The connection weight from the j-th hidden node to the ith output node is denoted as Wa2iji=1,2,3;j=1,…,N, so(37)Wa2ijk+1=Wa2ijk+ΔWa2ijk,i=1,2,3j=1,…,N,(38)ΔWa2ijk=ζak⋅−∂Eak∂eak∂eak∂J^k∂J^k∂Wa2ijk=ζak⋅−eak∂J^k∂uADPik∂uADPik∂Wa2ijk=−ζak⋅eak⋅∂J^k∂uADPik⋅Ah2jk.The middle term∂J^k/∂uADPik in equation (38) indicates that the path of the backpropagated signal passes through the critic network when training the action network [31]. Furthermore, by the output and input of the critic network, ∂J^k/∂uADPik can be obtained:(39)∂J^k∂uADPk=∂J^k∂Ch2k∂Ch2k∂Ch1k∂Ch1k∂uADPk.So,(40)∂J^k∂uADPik=∑j=1M∂J^k∂Ch2jk∂Ch2jk∂Ch1jk∂Ch1jk∂uADPik,i=1,2,3=∑j=1MWc2jk⋅121−Ch2j2k⋅Wc1j,i+3k=12⋅Wc1:,i+3Tk⋅Wc2Tk×1−Ch2k×Ch2k,where Wc1:,i+3 represents the (i + 3)-th column of Wc1. Equation (40) can be rewritten in matrix form:(41)∂J^k∂uADPk=12⋅Wc1uADPTk⋅Wc2Tk×1−Ch2k×Ch2k,where Wc1uADP=Wc1:,4:6 represents columns 4 to 6 of Wc1, that is, the connection weights of the three input nodes corresponding to uADP and all hidden nodes in the critic network. From equations (37)–(41), ΔWa2 can be deduced as(42)ΔWa2k=ζak⋅−∂Eak∂eak∂eak∂J^k∂J^k∂Wa2k=−ζak⋅eak⋅∂J^k∂uADPk⋅Ah2Tk=−12⋅ζak⋅eak⋅Wc1uADPTk⋅Wc2Tk×1−Ch2k×Ch2k⋅Ah2Tk.(6) Updating the Weights Wa1. Similar to the Wa2, the update of Wa1 is(43)Wa1k+1=Wa1k+ΔWa1k,(44)ΔWa1k=ζak⋅−∂Eak∂Wa1k=ζak⋅−∂Eak∂eak∂eak∂J^k∂J^k∂uADPk∂uADPk∂Ah2k∂Ah2k∂Ah1k∂Ah1k∂Wa1k=−ζak⋅eak⋅Wa2Tk⋅∂J^k∂uADPk×12⋅1−Ah2k×Ah2k⋅INaTk.Substituting equation (41) into equation (44), ΔWa1 can be easily obtained.So far, the training process is completed. And the optimal control signaluADP output by the action network will be combined with ωs output by outer loop main controller 1, that is(45)ωc=ωs+uADP,where the angular rate signal ωc∈R3×1 will be input as the reference command of the inner loop controller 2, and the control torque Mc output by controller 2 will operate the vehicle to complete the attitude control task.


## 3.1.1. Main Controller 1

The control objective of the outer loop is to operate the actual attitude angleγ to track γd within the desired accuracy. First, take the tracking error eγ=γd−γ∈R3. The sliding switching surface Sγ∈R3 of the outer loop can be selected as(10)Sγ=Sγ1,Sγ2,Sγ3T=eγ+ργ∫0teγdτ,where ργ=diagργ1,ργ2,ργ3∈R3×3 and ργi>0,i=1,2,3 are the parameters to be designed [30]. Obviously, on the sliding surface Sγ=0, the tracking error eγ can be guaranteed to converge uniformly, that is,(11)Sγ=eγ+ργ∫0teγdτ=0.In order to ensure the asymptotic convergence of the outer loop tracking error to the sliding surface, the virtual control law must be designed. First, take the derivative ofSγ as(12)S˙γ=e˙γ+ργeγ=γ˙d−γ˙+ργeγ=γ˙d−Γω+ργeγ.Take the following Lyapunov function:(13)L1=12SγTSγ>0,and the derivative of L1 is as(14)L˙1=SγTS˙γ.By Lyapunov stability,L˙1<0 has to be guaranteed. Therefore, the sliding mode approach law can be chosen as(15)S˙γ=−τγsignSγ,where designed parameter τγ>0 and signSγ=signSγ1,signSγ2,signSγ3T denotes a sign function.According to equations (12) and (15), there exists(16)γ˙d−Γω+ργeγ=−τγsignSγ.So, the virtual control law of the outer loop can be obtained as follows:(17)ωs=Γ−1γ˙d+ργeγ+τγsignSγ.In order to avoid or reduce the sliding mode chattering caused by the sign function in equation (17), a smooth continuous function can be adopted instead of the sign function. Because the saturation function is one of the most simple and effective ways, the virtual control law is redesigned as follows:(18)ωs=Γ−1γ˙d+ργeγ+τγsatSγξγ,where satSγ/ξγ=satSγ1/ξγ,satSγ2/ξγ,satSγ3/ξγT denotes a saturation function with width ξγ>0 as follows:(19)satSγiξγi=1,2,3=1,Sγi>ξγ,Sγiξγ,Sγi≤ξγ,−1,Sγi<−ξγ.Therefore, according to control law equation (18), the attitude angles can track the commands, and the error eγ uniformly converges. Next, ωs will be provided as the main reference signal to the inner loop.


## 3.1.2. ADP Auxiliary Controller

The idea of ADP is to take advantage of the function estimators to approximate the performance index functions and control strategies that meet the principle of optimality. By designing a critic-action structure, the critic network approximates the performance indexJ (the cost function) and J is defined as the forward accumulation of the utility function U with the discount factor λ [20, 21]:(20)Jk=∑i=k∞λi−kUi,where U is usually defined as a quadratic. It can be seen that the cost function is also a quadratic convex function, with only a local minimum and at the same time a global minimum. The action network obtains the optimal control law u∗ by minimizing J:(21)u∗k=argminukUk+λJk+1.In this paper, only the auxiliary ADP controller is added to the outer loop to compensate for the attitude angle error generated by the main controller 1. ADP outputsuADP (uADP has the same dimension as ωs), and the sum of uADP and ωs inputs as a reference instruction to the inner loop. Obviously, the ADP controller is sensitive to the attitude angle error. It can be imagined that ADP will start to work when a certain error occurs; when the error meets the threshold requirements, the ADP does not need to work, which will balance the loss in accuracy and calculation speed. However, this does not seem to be the focus of this paper. It may be discussed in future research, such as the selection and optimization of the threshold.In Figure2, ADP adopts a network structure based on ADHDP, which includes an action network, a critic network, and attitude model (9a). The input of ADP is the attitude error, and the action network generates the control signal uADP. At the same time, the critic network approximates J. The specific design of each network is given below.Figure 2
Structure of the ADP controller.(1) Critic Network. In Figure 3, the critic network uses a single hidden-layer BP neural network with six input nodes, M hidden nodes, and one output node. The input contains the attitude angle error Δγ and uADP generated by the action network. The output is the estimated J^ of the cost function J. Wc1∈RM×6 is the weight matrix of the input layer to the hidden layer and Wc1jii=1,…,6;j=1,…,M represents the weight of the i-th input node to the j-th hidden node. Wc2∈R1×M is the weight matrix from the hidden-to-output layer, and Wc2j,j=1,…,M represents the connection weight of the j-th hidden node to the output. Ch1∈RM×1 and Ch2∈RM×1 are the input and output vectors of hidden nodes, respectively. The active functions of the hidden layer and the output layer are a bipolar sigmoid function and linear function, respectively. The attitude error is as follows:(22)Δγ=Δα,Δβ,ΔμT=αd−α,βd−β,μd−μT.Figure 3
Structure of the critic network.The input of the critic network isINc∈R6×1 as(23)INc=Δγ;ΔuADP=Δα,Δβ,Δμ,uADP1,uADP2,uADP3T.The training of the critic network consists of two parts, one is the forward calculation, and the other is the error backpropagation of updating network weights. The forward process of stepk is(24)Ch1jk=∑i=16INcik⋅Wc1jik,j=1,2,…,M,Ch2jk=1−e−Ch1jk1+e−Ch1jk,j=1,2,…,M,J^k=∑j=1MCh2jk⋅Wc2jk,j=1,2,…,M.Equation (24) can be rewritten in matrix form as(25)Ch1k=Wc1⋅INc,Ch2k=1−e−Ch1k1+e−Ch1k,J^k=Wc2k⋅Ch2k.Based on the Bellman optimality principle, the critic network approximates the cost function of the system. The actualJk is defined as the cumulative return from the current state to the future:(26)Jk=∑i=k∞λi−kUi,where λ∈0,1 is a discount factor or forgetting factor, indicating the influence of the future state on the current strategy. U is the utility function at each step, which is defined as a quadratic:(27)Uk=ΔγTkΛΔγk,Λ=diagσ,σ,σ,σ>0.The following errorEc can be defined, and the critic network can approximate J by minimizing Ec:(28)Ec=12ec2,ec=J^k−Uk+1−λJ^k+1.Therefore, network weights can be updated through backpropagation ofEc.(2) Updating the Weights Wc2. Using the gradient descent method, let ΔWc2 be the gradient, so(29)Wc2k+1=Wc2k+ΔWc2k,where each component of ΔWc2 is represented as(30)ΔWc2jk=ζck⋅−∂Eck∂Wc2jk,j=1,…,M=−ζck∂Eck∂J^k∂J^k∂Wc2jk,j=1,…,M=−ζck⋅eck⋅Ch2jk,j=1,…,M,where ζck∈0,1 is the learning rate. Equation (30) is combined and rewritten into a matrix form as(31)ΔWc2k=−ζck⋅eck⋅Ch2Tk.(3) Updating the Weights Wc1. Similarly, let ΔWc1 be the gradient, so(32)Wc1k+1=Wc1k+ΔWc1k,ΔWc1jik=ζck⋅−∂Eck∂Wc1jik,i=1,…,6j=1,…,M=−ζck∂Eck∂J^k∂J^k∂Ch2jk∂Ch2jk∂Ch1jk∂Ch1jk∂Wc1jik=−ζck⋅eck⋅Wc2jk⋅121−Ch2j2k⋅INcik.Combine the above formula into a simplified matrix form as follows:(33)ΔWc1k=−12⋅ζck⋅eck⋅Wc2Tk×1−Ch2k×Ch2k⋅INcTk,where the symbol “×” represents the Hadamard product of two matrices, that is, bitwise multiplication; “⋅” represents the ordinary multiplication of matrices. These symbols appearing in the later parts of this paper possess the same meaning.(4) Action Network. As shown in Figure 4, the action network adopts a single hidden-layer BP neural network with three input nodes, N hidden nodes, and three output nodes. The network’s input is INa=Δγ∈R3×1, and output is uADP∈R3×1. Other parameters are defined similarly to the critic network. The active functions of the hidden and output layer are a bipolar sigmoid function and linear function, respectively.Figure 4
Structure of the action network.The training of the action network also includes forward calculation and error backpropagation. Firstly, the forward process is briefly presented as(34)Ah1k=Wa1⋅INa,Ah2k=1−e−Ah1k1+e−Ah1k,uADPk=Wa2k⋅Ah2k.The action network generates an optimal control strategy by minimizing the system cost functionJ. This goal can be achieved by minimizing the defined error Ea:(35)Eak=12ea2k,eak=J^k.(5) Updating the Weights Wa2. With the gradient descent method, the update process of Wa2 is(36)Wa2k+1=Wa2k+ΔWa2k,ΔWa2k=ζak⋅−∂Eak∂Wa2k,where ζak represents the learning rate. The connection weight from the j-th hidden node to the ith output node is denoted as Wa2iji=1,2,3;j=1,…,N, so(37)Wa2ijk+1=Wa2ijk+ΔWa2ijk,i=1,2,3j=1,…,N,(38)ΔWa2ijk=ζak⋅−∂Eak∂eak∂eak∂J^k∂J^k∂Wa2ijk=ζak⋅−eak∂J^k∂uADPik∂uADPik∂Wa2ijk=−ζak⋅eak⋅∂J^k∂uADPik⋅Ah2jk.The middle term∂J^k/∂uADPik in equation (38) indicates that the path of the backpropagated signal passes through the critic network when training the action network [31]. Furthermore, by the output and input of the critic network, ∂J^k/∂uADPik can be obtained:(39)∂J^k∂uADPk=∂J^k∂Ch2k∂Ch2k∂Ch1k∂Ch1k∂uADPk.So,(40)∂J^k∂uADPik=∑j=1M∂J^k∂Ch2jk∂Ch2jk∂Ch1jk∂Ch1jk∂uADPik,i=1,2,3=∑j=1MWc2jk⋅121−Ch2j2k⋅Wc1j,i+3k=12⋅Wc1:,i+3Tk⋅Wc2Tk×1−Ch2k×Ch2k,where Wc1:,i+3 represents the (i + 3)-th column of Wc1. Equation (40) can be rewritten in matrix form:(41)∂J^k∂uADPk=12⋅Wc1uADPTk⋅Wc2Tk×1−Ch2k×Ch2k,where Wc1uADP=Wc1:,4:6 represents columns 4 to 6 of Wc1, that is, the connection weights of the three input nodes corresponding to uADP and all hidden nodes in the critic network. From equations (37)–(41), ΔWa2 can be deduced as(42)ΔWa2k=ζak⋅−∂Eak∂eak∂eak∂J^k∂J^k∂Wa2k=−ζak⋅eak⋅∂J^k∂uADPk⋅Ah2Tk=−12⋅ζak⋅eak⋅Wc1uADPTk⋅Wc2Tk×1−Ch2k×Ch2k⋅Ah2Tk.(6) Updating the Weights Wa1. Similar to the Wa2, the update of Wa1 is(43)Wa1k+1=Wa1k+ΔWa1k,(44)ΔWa1k=ζak⋅−∂Eak∂Wa1k=ζak⋅−∂Eak∂eak∂eak∂J^k∂J^k∂uADPk∂uADPk∂Ah2k∂Ah2k∂Ah1k∂Ah1k∂Wa1k=−ζak⋅eak⋅Wa2Tk⋅∂J^k∂uADPk×12⋅1−Ah2k×Ah2k⋅INaTk.Substituting equation (41) into equation (44), ΔWa1 can be easily obtained.So far, the training process is completed. And the optimal control signaluADP output by the action network will be combined with ωs output by outer loop main controller 1, that is(45)ωc=ωs+uADP,where the angular rate signal ωc∈R3×1 will be input as the reference command of the inner loop controller 2, and the control torque Mc output by controller 2 will operate the vehicle to complete the attitude control task.


## 3.2. Inner Loop Controller

To ensure that the actual angular rateω can stably track the expected reference angular rate ωc, similar to controller 1, the sliding variable is selected for inner loop controller 2 as follows:(46)Sω=Sω1,Sω2,Sω3T=eω+ρω∫0teωdτ,where eω=ωc−ω∈R3×1 and ρω=diagρω1,ρω2,ρω3∈R3×3 with ρωi>0,i=1,2,3. In order to ensure the inner loop tracking error eω asymptotically converges to the sliding surface Sω=0, the actual control law M has to be designed.The derivative ofSω is(47)S˙ω=e˙ω+ρωeω=ω˙c+I−1ΩIω−I−1Mc+ρωeω.Take the following Lyapunov functionL2:(48)L2=12SωTSω>0,L˙2=SωTS˙ω.By Lyapunov stability,L˙2<0 has to be guaranteed. Therefore, the dynamics S˙ω can be chosen as(49)S˙ω=−τωsignSω,where designed parameter τω>0 and signSω=signSω1,signSω2,signSω3T denotes a sign function.According to equations (47) and (49), there exists(50)ω˙c+I−1ΩIω−I−1Mc+ρωeω=−τωsignSω.So, the actual control law of the inner loop can be obtained as follows:(51)Mc=Iω˙c+ΩIω+Iρωeω+τωI⋅signSω.Similarly, a continuous saturation function is chosen to replace the sign function to reduce the chattering. Therefore, the actual control law is rewritten as follows:(52)Mc=Iω˙c+ΩIω+Iρωeω+τωI⋅satSωξω.where satSω/ξω=satSω1/ξω,satSω2/ξω,satSω3/ξωT denotes a saturation function with width ξω>0.Therefore, for actual control law as equation (52), L˙2<0 holds. That is, the actual attitude angular rate ω converges asymptotically to the expected angular rate ωc.


## 4. Implementation Issues

In Section3, the design of ADP auxiliary controller is completed, but the parameter selection and training speed of ADP cannot be ignored in practical application. So, in this section, some issues are discussed about implementation of ADP structure, including parameter selection for networks and skills related to speed up training.


### 4.1. Network Parameters and Their Convergence

It is clear that the critic network with a single hidden layer and randomly initialized weights can approximateJ with arbitrarily small errors, that is, limk⟶∞J^k−Jk=0. Similarly, the action network with randomly initialized weights can minimize the cost function J^ and its output can approximate to the optimal control law uADP∗, that is,(53)uADP∗=argminuADP∗J^.In other words, both the critic network and action network evolve towards the optimal direction to achieve their goals. Furthermore, considering equations (25) and (34), it is because of the adjustment of network weights Wc1, Wc2, Wa1, and Wa2 that the output of the networks reaches the desired optimal value. That is, when the optimal control strategy uADP∗ is obtained, the network weights will also reach the optimal weights as follows [32]:(54)Wc∗=argminWcJ^k−Uk+1−λJ^k+1,Wa∗=argminWaJ^k,where Wc∗ and Wa∗ represent the optimal weights of the critic and action network, respectively.Lemma 1.
In critic and action network, the weightsWc and Wa are finally uniformly stable and approach the optimal weights Wc∗ and Wa∗.Proof.
It is well known that the weights of the input to hidden layer are similar to the weights of the hidden to output layer. In order to facilitate the elaboration, this paper only presents the uniform stability proof aboutWc2 and Wa2, which are the weights of the hidden to output layer. Let the optimal weights corresponding to Wc2 and Wa2 be Wc2∗ and Wa2∗, respectively, and they are bounded. Wc2∗≤κc, Wa2∗≤κa, and κc,κa are positive constants.
Equation (28) can be rewritten as(55)Eck=12ec2k,eck=λJ^k−J^k−1−Uk=λ⋅Wc2k⋅Ch2k−λ⋅Wc2k−1⋅Ch2k−1−Uk.
From equations (29) to (31), the update of Wc2 can be rewritten as follows:(56)Wc2k+1=Wc2k+ΔWc2k=Wc2k+−ζck⋅∂Eck∂eck∂eck∂J^k∂J^kWc2k=Wc2k−λ⋅ζck⋅λ⋅Wc2k⋅Ch2k−λ⋅Wc2k−1⋅Ch2k−1−Uk⋅Ch2Tk.
Similarly, the update ofWa2 is(57)Wa2k+1=Wa2k+ΔWa2k,=Wa2k+ζak⋅−∂Eak∂Wa2k=Wa2k−12⋅ζak⋅eak⋅Wc1uADPTk⋅Wc2Tk×1−Ch2k×Ch2k⋅Ah2Tk=Wa2k−12⋅ζak⋅J^k⋅Θk⋅Wc2Tk⋅Ah2Tk=Wa2k−12⋅ζak⋅Wc2k⋅Ch2k⋅Θk⋅Wc2Tk⋅Ah2Tk,where Θk=Wc1uADPTk×1−Ch2k×Ch2k1−Ch2k×Ch2k1−Ch2k×Ch2kT.
First, the Lyapunov method is adopted to analyse the convergence ofWc2:(58)Vck=1ζcktrW˜c2kW˜c2Tk,where W˜c2k=Wc2k−Wc2∗k is the error between actual and optimal weights. Then, the first-order difference of Vc is expressed as(59)ΔVck=1ζcktrW˜c2k+1W˜c2Tk+1−W˜c2kW˜c2Tk.
According to equation (56), (60) can be obtained:(60)W˜c2k+1=Wc2k+1−Wc2∗=Wc2k−λ⋅ζck⋅λ⋅Wc2k⋅Ch2k−λ⋅Wc2k−1⋅Ch2k−1−Uk⋅Ch2Tk−Wc2∗=W˜c2k−λ⋅ζck⋅λ⋅W˜c2k+Wc2∗⋅Ch2k−λ⋅Wc2k−1⋅Ch2k−1−Uk⋅Ch2Tk=W˜c2k⋅I−λ2⋅ζck⋅Ch2kCh2Tk−λ⋅ζck⋅λ⋅Wc2∗⋅Ch2k−λ⋅Wc2k−1⋅Ch2k−1−Uk⋅Ch2Tk.
In addition, denote the approximation error between actual and optimal output as(61)δck=Wc2k−Wc2∗⋅Ch2k=W˜c2k⋅Ch2k.
Substituting equations (60) and (61) into equation (59), ΔVck can be deduced:(62)ΔVck=−λ2δck2−λ21−λ2⋅ζckCh2k2⋅δck+Wc2∗⋅Ch2k+1λUk−1λWc2k−1⋅Ch2k−12+λ⋅Wc2∗⋅Ch2k+Uk−Wc2k−1⋅Ch2k−12.
Furthermore, applying the Cauchy–Schwarz inequality [33], it can be deduced as(63)ΔVck≤−λ2δck2−λ21−λ2⋅ζckCh2k2δck+Wc2∗⋅Ch2k+1λUk−1λWc2k−1⋅Ch2k−12+2λ⋅Wc2∗⋅Ch2k+Uk−12Wc2k−1⋅Ch2k−1−12Wc2∗⋅Ch2k−12+12δck−12.
Similarly, setVak=1/ψζaktrW˜a2kW˜a2Tk, ψ>0.
Denote the approximation error of the action network between the actual and optimal output asδak=Wa2k−Wa2∗⋅Ah2k=W˜a2k⋅Ah2k. Referring to ΔVc, ΔVa satisfies(64)ΔVak≤1ψ−Θk⋅Wc2Tk2−ζak⋅Θk⋅Wc2Tk2Ah2k2⋅Wc2k⋅Ch2k2+Wc2k⋅Ch2k2δak2+4Wc2∗⋅Ch2k2+4δck2.
Furthermore, setVδk=1/2δck−12, and then(65)ΔVδk=12δck2−δck−12.
From the above derivation, we can finally take the total Lyapunov functionVk as(66)Vk=Vck+Vak+Vδk≤−λ2−12−4ψδck2−λ21−λ2⋅ζckCh2k2⋅δck+Wc2∗⋅Ch2k+1λUk−1λWc2k−1⋅Ch2k−12−1ψΘk⋅Wc2Tk2−ζak⋅Θk⋅Wc2Tk2Ah2k2⋅Wc2k⋅Ch2k2+2λ⋅Wc2∗k⋅Ch2k+Uk−12Wc2k−1⋅Ch2k−1−12Wc2∗⋅Ch2k−12+1ψWc2k⋅Ch2k2δak2+4ψWc2∗k⋅Ch2k2.
Selecting some parameters as equation (67), then equation (68) holds:(67)12<λ<1,ζck<1λ2Ch2k2,ζak<1Ah2k2,ψ>4λ2−1/2,(68)ΔVk≤−λ2−12−4ψδck2−λ21−λ2⋅ζckCh2k2⋅δck+Wc2∗⋅Ch2k+1λUk−1λWc2k−1⋅Ch2k−12−1ψΘk⋅Wc2Tk2−ζak⋅Θk⋅Wc2Tk2Ah2k2⋅Wc2k⋅Ch2k2+D2,where D2 represents(69)D2=2λ⋅Wc2∗k⋅Ch2k+Uk−12Wc2k−1⋅Ch2k−1−12Wc2∗⋅Ch2k−12+1ψWc2k⋅Ch2k2δak2+4ψWc2∗k⋅Ch2k2.
Furthermore, applying the Cauchy–Schwarz inequality, we get(70)D2≤8λ2Wc2∗k⋅Ch2k2+U2k+14Wc2k−1⋅Ch2k−12+14Wc2∗⋅Ch2k−1,+2ψΘk⋅Wc2Tk2⋅Wa2k⋅Ah2k2+Wa2∗⋅Ah2k2+4ψWc2∗k⋅Ch2k2,≤8λ2+4+4ψ⋅Wc2max2⋅Ch2max2+4ψ⋅Wc2max2⋅Θmax2Wa2max2⋅Ah2max2+8Umax2=Dmax2,where the subscript “max” represents the upper bound of the corresponding parameters’ 2-norm, such as Wc2≤Wc2max.
Therefore, for any(71)ζck>Dmaxλ−1/2−4/ψ,ΔVk≤0 holds. This indicates that the actual weights will converge to the optimal weights. In other words, the weight error δcandδa are uniformly bounded. This also results in a stable ADP system and an optimal output.
Furthermore, note that the components ofCh2 and Ah2 are limited to [−1, 1] due to the activation functions of the hidden nodes, that are(72)−1≤Ch2i≤1,i=1,…,M,−1≤Ah2j≤1,j=1,…,N.
So, there exist(73)Ch2k2=∑i=1MCh2ik2≤M,Ah2k2=∑j=1NAh2jk2≤N.
According to equation (67), some networks’ parameters should satisfy(74)12<λ<1,ζck<1λ2M,ζak<1N,ψ>4λ2−1/2.
Equation (74) provides a simple and intuitive guidance to select networks’ structure and learning rate, while maintaining the stability of weights and ADP structure.


### 4.2. Improvement in Implementation

In the previous literature, when it comes to the training of feedforward networks, all weights usually need to be adjusted, so there are serious dependencies between different layers. Moreover, the algorithm based on gradient descent is widely applied to the learning of various feedforward neural networks. However, it is obvious that the learning method based on gradient descent is usually very slow and time-consuming because of improper learning steps, or it is easy to be overtrained and falls into local minima.In order to make the training process as time-saving as possible and better meet the time matching between online training and practical applications, we can consider two ideas: one is based on Igelnik and Pao’s theory [34], that is, for a single hidden-layer forward neural network, if the weights of input to hidden layer are randomly initialized and kept constant, as long as the number of hidden nodes is sufficient, the approximation error of the network can be arbitrarily small. The second is based on the extreme learning machine (ELM) proposed by Huang et al. [35, 36]. For a single hidden layer forward neural network, the weights of the input to hidden layer are initialized randomly and kept constant, and then the hidden nodes are arbitrarily selected. The weights of hidden to output layer are directly determined analytically by the Moore–Penrose inverse, without necessary to derive and calculate partial derivatives layer by layer such as the gradient descent method. The speed of extreme learning methods has been proven to be tens or even thousands of times that of ordinary gradient descent methods, and it can effectively reduce complexity and avoid local minima [37].To facilitate implementation, this paper will adopt the first idea to improve the performance; that is, the weightsWc1 and Wa1 are randomly initialized in a finite interval and kept constant, and only the weights Wc2 and Wa2 are adjusted by the gradient descent algorithm, resulting in effectively avoiding excessive time consumption. As for the thinking based on extreme learning machine, it is only given here without in-depth discussion due to the limited space of this paper and the lack of theoretical guidance in the application of vehicles. We may make further analysis and give more rigorous theories to support the application in practical vehicle control in future research.


## 4.1. Network Parameters and Their Convergence

It is clear that the critic network with a single hidden layer and randomly initialized weights can approximateJ with arbitrarily small errors, that is, limk⟶∞J^k−Jk=0. Similarly, the action network with randomly initialized weights can minimize the cost function J^ and its output can approximate to the optimal control law uADP∗, that is,(53)uADP∗=argminuADP∗J^.In other words, both the critic network and action network evolve towards the optimal direction to achieve their goals. Furthermore, considering equations (25) and (34), it is because of the adjustment of network weights Wc1, Wc2, Wa1, and Wa2 that the output of the networks reaches the desired optimal value. That is, when the optimal control strategy uADP∗ is obtained, the network weights will also reach the optimal weights as follows [32]:(54)Wc∗=argminWcJ^k−Uk+1−λJ^k+1,Wa∗=argminWaJ^k,where Wc∗ and Wa∗ represent the optimal weights of the critic and action network, respectively.Lemma 1.
In critic and action network, the weightsWc and Wa are finally uniformly stable and approach the optimal weights Wc∗ and Wa∗.Proof.
It is well known that the weights of the input to hidden layer are similar to the weights of the hidden to output layer. In order to facilitate the elaboration, this paper only presents the uniform stability proof aboutWc2 and Wa2, which are the weights of the hidden to output layer. Let the optimal weights corresponding to Wc2 and Wa2 be Wc2∗ and Wa2∗, respectively, and they are bounded. Wc2∗≤κc, Wa2∗≤κa, and κc,κa are positive constants.
Equation (28) can be rewritten as(55)Eck=12ec2k,eck=λJ^k−J^k−1−Uk=λ⋅Wc2k⋅Ch2k−λ⋅Wc2k−1⋅Ch2k−1−Uk.
From equations (29) to (31), the update of Wc2 can be rewritten as follows:(56)Wc2k+1=Wc2k+ΔWc2k=Wc2k+−ζck⋅∂Eck∂eck∂eck∂J^k∂J^kWc2k=Wc2k−λ⋅ζck⋅λ⋅Wc2k⋅Ch2k−λ⋅Wc2k−1⋅Ch2k−1−Uk⋅Ch2Tk.
Similarly, the update ofWa2 is(57)Wa2k+1=Wa2k+ΔWa2k,=Wa2k+ζak⋅−∂Eak∂Wa2k=Wa2k−12⋅ζak⋅eak⋅Wc1uADPTk⋅Wc2Tk×1−Ch2k×Ch2k⋅Ah2Tk=Wa2k−12⋅ζak⋅J^k⋅Θk⋅Wc2Tk⋅Ah2Tk=Wa2k−12⋅ζak⋅Wc2k⋅Ch2k⋅Θk⋅Wc2Tk⋅Ah2Tk,where Θk=Wc1uADPTk×1−Ch2k×Ch2k1−Ch2k×Ch2k1−Ch2k×Ch2kT.
First, the Lyapunov method is adopted to analyse the convergence ofWc2:(58)Vck=1ζcktrW˜c2kW˜c2Tk,where W˜c2k=Wc2k−Wc2∗k is the error between actual and optimal weights. Then, the first-order difference of Vc is expressed as(59)ΔVck=1ζcktrW˜c2k+1W˜c2Tk+1−W˜c2kW˜c2Tk.
According to equation (56), (60) can be obtained:(60)W˜c2k+1=Wc2k+1−Wc2∗=Wc2k−λ⋅ζck⋅λ⋅Wc2k⋅Ch2k−λ⋅Wc2k−1⋅Ch2k−1−Uk⋅Ch2Tk−Wc2∗=W˜c2k−λ⋅ζck⋅λ⋅W˜c2k+Wc2∗⋅Ch2k−λ⋅Wc2k−1⋅Ch2k−1−Uk⋅Ch2Tk=W˜c2k⋅I−λ2⋅ζck⋅Ch2kCh2Tk−λ⋅ζck⋅λ⋅Wc2∗⋅Ch2k−λ⋅Wc2k−1⋅Ch2k−1−Uk⋅Ch2Tk.
In addition, denote the approximation error between actual and optimal output as(61)δck=Wc2k−Wc2∗⋅Ch2k=W˜c2k⋅Ch2k.
Substituting equations (60) and (61) into equation (59), ΔVck can be deduced:(62)ΔVck=−λ2δck2−λ21−λ2⋅ζckCh2k2⋅δck+Wc2∗⋅Ch2k+1λUk−1λWc2k−1⋅Ch2k−12+λ⋅Wc2∗⋅Ch2k+Uk−Wc2k−1⋅Ch2k−12.
Furthermore, applying the Cauchy–Schwarz inequality [33], it can be deduced as(63)ΔVck≤−λ2δck2−λ21−λ2⋅ζckCh2k2δck+Wc2∗⋅Ch2k+1λUk−1λWc2k−1⋅Ch2k−12+2λ⋅Wc2∗⋅Ch2k+Uk−12Wc2k−1⋅Ch2k−1−12Wc2∗⋅Ch2k−12+12δck−12.
Similarly, setVak=1/ψζaktrW˜a2kW˜a2Tk, ψ>0.
Denote the approximation error of the action network between the actual and optimal output asδak=Wa2k−Wa2∗⋅Ah2k=W˜a2k⋅Ah2k. Referring to ΔVc, ΔVa satisfies(64)ΔVak≤1ψ−Θk⋅Wc2Tk2−ζak⋅Θk⋅Wc2Tk2Ah2k2⋅Wc2k⋅Ch2k2+Wc2k⋅Ch2k2δak2+4Wc2∗⋅Ch2k2+4δck2.
Furthermore, setVδk=1/2δck−12, and then(65)ΔVδk=12δck2−δck−12.
From the above derivation, we can finally take the total Lyapunov functionVk as(66)Vk=Vck+Vak+Vδk≤−λ2−12−4ψδck2−λ21−λ2⋅ζckCh2k2⋅δck+Wc2∗⋅Ch2k+1λUk−1λWc2k−1⋅Ch2k−12−1ψΘk⋅Wc2Tk2−ζak⋅Θk⋅Wc2Tk2Ah2k2⋅Wc2k⋅Ch2k2+2λ⋅Wc2∗k⋅Ch2k+Uk−12Wc2k−1⋅Ch2k−1−12Wc2∗⋅Ch2k−12+1ψWc2k⋅Ch2k2δak2+4ψWc2∗k⋅Ch2k2.
Selecting some parameters as equation (67), then equation (68) holds:(67)12<λ<1,ζck<1λ2Ch2k2,ζak<1Ah2k2,ψ>4λ2−1/2,(68)ΔVk≤−λ2−12−4ψδck2−λ21−λ2⋅ζckCh2k2⋅δck+Wc2∗⋅Ch2k+1λUk−1λWc2k−1⋅Ch2k−12−1ψΘk⋅Wc2Tk2−ζak⋅Θk⋅Wc2Tk2Ah2k2⋅Wc2k⋅Ch2k2+D2,where D2 represents(69)D2=2λ⋅Wc2∗k⋅Ch2k+Uk−12Wc2k−1⋅Ch2k−1−12Wc2∗⋅Ch2k−12+1ψWc2k⋅Ch2k2δak2+4ψWc2∗k⋅Ch2k2.
Furthermore, applying the Cauchy–Schwarz inequality, we get(70)D2≤8λ2Wc2∗k⋅Ch2k2+U2k+14Wc2k−1⋅Ch2k−12+14Wc2∗⋅Ch2k−1,+2ψΘk⋅Wc2Tk2⋅Wa2k⋅Ah2k2+Wa2∗⋅Ah2k2+4ψWc2∗k⋅Ch2k2,≤8λ2+4+4ψ⋅Wc2max2⋅Ch2max2+4ψ⋅Wc2max2⋅Θmax2Wa2max2⋅Ah2max2+8Umax2=Dmax2,where the subscript “max” represents the upper bound of the corresponding parameters’ 2-norm, such as Wc2≤Wc2max.
Therefore, for any(71)ζck>Dmaxλ−1/2−4/ψ,ΔVk≤0 holds. This indicates that the actual weights will converge to the optimal weights. In other words, the weight error δcandδa are uniformly bounded. This also results in a stable ADP system and an optimal output.
Furthermore, note that the components ofCh2 and Ah2 are limited to [−1, 1] due to the activation functions of the hidden nodes, that are(72)−1≤Ch2i≤1,i=1,…,M,−1≤Ah2j≤1,j=1,…,N.
So, there exist(73)Ch2k2=∑i=1MCh2ik2≤M,Ah2k2=∑j=1NAh2jk2≤N.
According to equation (67), some networks’ parameters should satisfy(74)12<λ<1,ζck<1λ2M,ζak<1N,ψ>4λ2−1/2.
Equation (74) provides a simple and intuitive guidance to select networks’ structure and learning rate, while maintaining the stability of weights and ADP structure.


## 4.2. Improvement in Implementation

In the previous literature, when it comes to the training of feedforward networks, all weights usually need to be adjusted, so there are serious dependencies between different layers. Moreover, the algorithm based on gradient descent is widely applied to the learning of various feedforward neural networks. However, it is obvious that the learning method based on gradient descent is usually very slow and time-consuming because of improper learning steps, or it is easy to be overtrained and falls into local minima.In order to make the training process as time-saving as possible and better meet the time matching between online training and practical applications, we can consider two ideas: one is based on Igelnik and Pao’s theory [34], that is, for a single hidden-layer forward neural network, if the weights of input to hidden layer are randomly initialized and kept constant, as long as the number of hidden nodes is sufficient, the approximation error of the network can be arbitrarily small. The second is based on the extreme learning machine (ELM) proposed by Huang et al. [35, 36]. For a single hidden layer forward neural network, the weights of the input to hidden layer are initialized randomly and kept constant, and then the hidden nodes are arbitrarily selected. The weights of hidden to output layer are directly determined analytically by the Moore–Penrose inverse, without necessary to derive and calculate partial derivatives layer by layer such as the gradient descent method. The speed of extreme learning methods has been proven to be tens or even thousands of times that of ordinary gradient descent methods, and it can effectively reduce complexity and avoid local minima [37].To facilitate implementation, this paper will adopt the first idea to improve the performance; that is, the weightsWc1 and Wa1 are randomly initialized in a finite interval and kept constant, and only the weights Wc2 and Wa2 are adjusted by the gradient descent algorithm, resulting in effectively avoiding excessive time consumption. As for the thinking based on extreme learning machine, it is only given here without in-depth discussion due to the limited space of this paper and the lack of theoretical guidance in the application of vehicles. We may make further analysis and give more rigorous theories to support the application in practical vehicle control in future research.


## 5. Simulations

In this section, the control strategy with ADP derived above is implemented to vehicle attitude control, and the effectiveness of the designed strategy is verified by comparing with the conventional controller without ADP.According to a vehicle model in laboratory, the inertia matrixI is taken as(75)I=135−20−1−201060−1−1−1975.The common parameters are taken as follows:ργ=diag1,1,1, ρw=diag0.5,0.5,0.5; the width ξγ=ξw=2 in the saturation function; τγ=τw=0.2. The number of hidden nodes is M=N=8. According to equation (74), the discount factor takes λ=0.9 with learning rate ζck<0.155, ζak<0.125. Take Λ=diag1,1,1, and all weights are randomly initialized in [−0.2, 0.2].Set the initial flight state of the vehicle asγ0=35,0.5,5Tdeg and w0=0,0,0Trad/s. The desired attitude instruction is γd=37,0,10Tdeg, and the simulation step size is 0.02 s. To verify the performance of the controller, pulsed disturbances d1=d2=10,20,10 will be added at 10 s.Figure5 presents the tracking results of the three attitude angles. As can be seen from these figures, the controller with ADP is more responsive than the controller without ADP. For example, the controller with ADP can accurately track instructions within 100 steps and cause less overshoot, while the controller without ADP requires about 200 steps. When external disturbances are added at 10 s, the controller with ADP also responds more quickly and with less overshoot. Through these, it can be seen that ADP improves the performance of the system.Figure 5
Tracking result of the three attitude angles. (a) The angle of attackα. (b) The sideslip angle β. (c) The bank angle μ.
(a)(b)(c)The controller with ADP shows faster performance and less overshoot, which benefits from the ADP structure’s auxiliary behaviour to the outer loop. Through the training process that meets the expected threshold, the ADP structure generates the auxiliary optimal control signal to compensate for the deficiency of the outer loop main controller 1 in eliminating attitude error. Figures6–11 show the training process of the ADP network. Specifically, Figures 6–9 show the dynamic adjustment of network weights. Figures 10 and 11 are the estimated value of the cost function output by the critic network and the optimal control signal output by the action network. Compared to the previous Figures 5, 6–9 show the rapid adjustment of the network weights at the beginning stage to achieve the purpose of tracking instructions. As the system output gradually keeps up with the instructions, the weights also converge to the optimal weights (W∗ as demonstrated in Section 4) and remain stable. When the external disturbances are added at 10 s, the network weights are adjusted again and tend to other optimal weights. It shows that ADP produces auxiliary output to play a certain role at the beginning and when disturbance appears.Figure 6
Wc1.Figure 7
Wc2.Figure 8
Wa1.Figure 9
Wa2.Figure 10
Estimated cost functionJ.Figure 11
Optimal control signaluADP.According to the thinking and analysis in Section4.2, when implementing this control strategy, it can be considered that randomly initializing the weights of the input to hidden layer (Wc1 and Wa1) and keeping them constant. During the training, only adjusting the weights Wc2 and Wa2 can not only achieve the same optimal control goal but also greatly reduce the time consumption. Figures 12 and 13 show the corresponding weight changes. Simultaneously, Figure 14 shows the comparison of time consumption in 12 group simulations. It can be further concluded that the average time consumption of maintaining the weights of the input-to-hidden layer (Wc1 and Wa1) and only adjusting the weights of the hidden-to-output layer (Wc2 and Wa2) is 31.9% lower than that of adjusting all weights. Although the sample in Figure 14 is limited, combined with the analysis in Section 4.2 and neural network theory, the effectiveness of this idea in reducing time consumption and improving efficiency is significant.Figure 12
Keep (a)Wc1 and update (b) Wc2.
(a)(b)Figure 13
Keep (a)Wa1 and update (b) Wa2.
(a)(b)Figure 14
Comparison of time consumption in 12 group simulations.Furthermore, Figures15–19 show the tracking control results of time-varying attitude commands, using the controller with ADP. The pulsed disturbances d=5,1,1T and d=10,2,2T are introduced at 10 s and 20 s, respectively, as shown by the yellow arrow in the figures. From Figures 15–17, it can be seen that the controller with ADP auxiliary structure can make the actual attitude angles accurately track the commands. Figures 18–19 show the weights of action network and critic network in ADP. The weights of the action network are dynamically adjusted to output the optimal auxiliary control signal uADP in real time, as shown in Figure 20. Figure 21 shows the control torque acting on the vehicle. From these, it can be seen that the controller with ADP auxiliary structure has good dynamic stability performance.Figure 15
Tracking result of the angle of attackα.Figure 16
Tracking result of the sideslip angleβ.Figure 17
Tracking result of the bank angleμ.Figure 18
Wa1 of action network.Figure 19
Wc1 of critic network.Figure 20
Optimal control signaluADP.Figure 21
Control inputsMc.


## 6. Conclusions

Combining the hottest reinforcement learning at present, this paper presents an ADP-based attitude control methodology for reentry vehicles, applying the ADP to the three-channel attitude control. First, a nonlinear model of the three-channel attitude system is established, and it is divided into inner and outer loops according to the principle of time scale separation. Both the inner and outer loops utilize a conventional sliding mode controller as the main controller, and an auxiliary ADP framework is introduced to the outer loop. When facing the vehicle's nonlinearity and sudden disturbances in particular, the main controller is easy to be weak due to its lack of sufficient adaptability. At this time, the auxiliary role of ADP will be fully exerted. Because ADP uses the critic network and action network, ADP structure has good learning ability. It generates the optimal auxiliary signal immediately after learning the tracking error to compensate for the deficiency of the main controller and improves the adaptability and response speed of the entire control system. For implementation, this paper discusses selection strategies of the ADP parameter and some tips for speeding up training. And the stability is proved by the Lyapunov method. Finally, simulation results of step and time-varying commands demonstrate the effectiveness of the designed scheme for the nonlinear attitude system.In the future work, we will focus on some switching or event-triggered strategies for this structure with dual controllers. Imagining that if the ADP auxiliary structure is event-triggered rather than time-triggered, it will greatly reduce consumption of ADP’s time and system resources, to improve efficiency.



---

*Source: 2837916-2020-09-02.xml*