# Adaptive Critic Learning-Based Robust Control of Systems with Uncertain Dynamics

Authors: Jun Zhao; Qingliang Zeng; Bin Guo
Journal: Computational Intelligence and Neuroscience
 (2021)

---

## Abstract

Model uncertainties are usually unavoidable in the control systems, which are caused by imperfect system modeling, disturbances, and nonsmooth dynamics. This paper presents a novel method to address the robust control problem for uncertain systems. The original robust control problem of the uncertain system is first transformed into an optimal control of nominal system via selecting the appropriate cost function. Then, we develop an adaptive critic leaning algorithm to learn online the optimal control solution, where only the critic neural network (NN) is used, and the actor NN widely used in the existing methods is removed. Finally, the feasibility analysis of the control algorithm is given in the paper. Simulation results are given to show the availability of the presented control method.


---

## Body

## 1. Introduction

The basis of intelligent optimization decision-using adaptive dynamic programming (ADP) method is the optimal control design. There are many mature methods for optimal regulation control design of linear systems in the field of control theory and control engineering. However, for general nonlinear systems, Hamilton–Jacobi–Bellman (HJB) equation is yielded. The analytical solution of HJB equation is not easy since it is inherently a partial differential equation. Recently, the optimal control design of systems has attracted extensive attention. Among them, the successive approximation methods [1–3] overcome this difficulty via finding the approximate solution of HJB equation, which is closely related to the ADP method. ADP is a new method based on the idea of intelligent learning, which can provide effective optimal control solution for complex dynamic systems [4, 5]. In the past two decades, ADP has been widely used in solving adaptive optimal control problems of discrete-time and continuous-time systems [6, 7]. Now, data-driven control design has become a research hotspot in the field of control theory and control engineering [8, 9]. The ADP methods can promote the research of data-based decision-making and optimal control and is conducive to the development of artificial intelligence and computational intelligence technology.Most of the existing results of ADP methods are obtained without considering the uncertainty of the controlled plant. However, the actual control system is always affected by model uncertainty, external disturbance, or other changes. We must consider these factors in the controller design to avoid the deterioration of the performance for the closed-loop system and improve the robustness of the controlled system. For robust control design, several alternative methodologies have been suggested in the control community. The work in [10] exploited the relationship between the robust control and the optimal control of nominal system subject to a specific value function. It indicates that one can design a robust control by solving an equivalent optimal control problem alternatively. Similarly, it was shown in [11] that the robust control design may be accomplished by addressing an H control problem. Nevertheless, online solving the derived optimal control equations was not discussed in [10]. Instead, they adopted offline schemes to seek for the solution of the derived optimal control equations. Recently, robust control design using the adaptive critic learning method has gradually become one of the research hotspots in the field of ADP, and many methods have been proposed [12–14]. These results fully show that the ADP method is suitable for robust control design of complex nonlinear systems in uncertain environment. Since many previous ADP literatures do not focus on the robust performance of the controller, the emergence of robust adaptive critic control greatly expands the application scope of the ADP method. Generally, the controller based on robust ADP can not only stabilize the original uncertain system but also make the system optimal without dynamic uncertainty. Thus, adaptive critic learning-based robust control includes the discussion of system stability, convergence, optimality, and robustness. It plays an important role in the field of intelligent learning control of complex systems in uncertain environment.Based on the above facts, we develop an adaptive critic learning algorithm to resolve the robust control problem of uncertain systems. To this end, we construct an equivalence between the robust control problem and the optimal control problem via selecting the appropriate cost function; then, a single critic NN is used to reformulate the cost function. To realize the optimal control solution, we design an adaptive critic leaning algorithm; since it has strong convergence, the actor NN widely used in existing ADP results is removed. Then, the feasibility analysis of the control algorithm is also given in the paper. Simulations are given to indicate the validity of the developed method.The major contributions of this paper include(1).
To address the robust control problem, we transform the robust control problem of uncertain systems into an optimal control problem of the nominal system. It provides a new approach to address the robust control problem.(2).
Different to [13], the uncertainty in the input matrix is considered in this paper, and then, the proposed control method is used in robotic systems. This helps to apply the proposed control algorithm to the practical industrial robotic systems in the future.(3).
A novel designed adaptation algorithm driven by the NN weights’ errors is used to online learn the critic NN weights. Different to [15], the convergence of the estimated NN weights to the true values can be retained.This paper is organized as follows. In Section2, we introduce the robust control problem and transform the robust control problem into an optimal control problem. In Section 3, a single critic NN is used to reformulate the optimal cost function, and then, an adaptive critic learning method is proposed to address the derived optimal control problem. Section 4 gives some simulation results to illustrate the effectiveness of the proposed method. Some conclusions are stated in Section 5.


## 2. Preliminaries and Problem Formulation

A continuous-time (CT) uncertain system can be written as(1)x˙t=fx+gxu+bxu+gxdx,where x∈ℝn and u∈ℝm are the system state and the control action, respectively. fx∈ℝn with f0=0 and gx∈ℝn×m are the nonlinear functions. bx and dx are the uncertainties. The purpose of this paper is designing a controller to make system (1) asymptotically stable under the uncertainties bx and dx. To this end, we give following assumptions.Assumption 1.
bx≥0 is the uncertainty in the input matrix. The uncertainty dx is bounded, i.e., there exists a nonnegative function dmaxx such that dx≤dmaxx.
To design a robust controller for the linear system, a linear matrix inequality (LMI) is proposed [16], while for nonlinear system (1), it is not easy. Inspired by [10, 12], an equivalence is built between the robust control problem of the uncertain system and the optimal control of the nominal system via selecting the appropriate cost function. Thus, we define the nominal system of the uncertain system (1) as(2)x˙t=fx+gxu.
For system (2), a control action u should be found to minimize the following cost function [17]:(3)Vx=∫t∞dmax2x+xTMx+uTNuds,where M∈ℝn×n and N∈ℝm×m are the positive definite weight matrices. Hence, based on the optimal principle, we can obtain the Lyapunov function of the cost function (3) as(4)0=VxTfx+gxu+dmax2x+xTMx+uTNu,where Vx=∂V/∂x is the derivative of V with respect to x.
Therefore, we can get the optimal cost function as(5)V∗x=minVx,and its corresponding HJB equation can be given as(6)0=minHx,u∗,V∗.
By solving (6), we have the optimal control action as(7)u∗=−12N−1gT∂V∗x∂x.
Then, we will give the lemma to explain the robust control problem of system (1) which can be transformed into an optimal control problem of system (2) via constructing cost function (3).Lemma 1 (see [11, 18]).
Assume that the solution can be solved via optimal control problem of system (2) with cost function (3) and dmax2x≥dTxNdx, and this solution can make uncertain system (1) asymptotically stable, which means that the optimal control solution is the solution of the robust control problem for system (1).Proof.
BecauseV∗x>0 for x≠0 and V∗0=0 for x=0 given in (5), then we can consider V∗x is a Lyapunov function; based on (6) and (7), we have(8)V˙∗x=Vx∗Tx˙=Vx∗Tfx+gxu∗+bxu∗+gxdx=−dmax2x−xTMx−u∗TNu∗+Vx∗Tgxbxu∗+Vx∗Tgxdx=−dmax2x−xTMx−u∗TNu∗−2u∗TNdx−2u∗TNbxu∗≤−dmax2x−xTMx−u∗TNu∗−2u∗TNdx=−dmax2x−xTMx−u∗TNu∗+dTxNdx−dTxNdx−2u∗TNdx=−dmax2x+dTxNdx−xTMx−u∗+dxTNu∗+dx≤−dmax2x+dTxNdx−xTMx.
According to the condition given in Lemma1, i.e., dmax2x≥dTxNdx, we obtain(9)V˙∗x≤−xTMx,i.e., V˙∗x<0,x≠0, and V˙∗0=0, and the uncertain system (1) is asymptotically stable for any uncertainties bx and dx. According to the above facts, the optimal solution u∗ is the robust control solution of the uncertain system (1). This completes the proof.
From Lemma1, we have that if we select N=Im, where Im is the identity matrix with m×m, then dmax2x≥dTxNdx=dTxdx=dx2 holds.Remark 1.
Lemma1 shows that the robust control problem of the original uncertain system can be equivalent to the optimal control problem of the nominal system, and then, the solution of the robust control problem can be obtained indirectly by solving the optimal control problem. Therefore, this equivalence relationship can be used to develop a new robust control design method and solve it by using ADP method, as described in the following section.Remark 2.
It is well-known thatH∞ control belongs to robust control. Although many H∞ control design techniques have been proposed, it should be noted that, as explained in Section 8.5 in [18], the H∞ control differs in the optimal method proposed in this paper. In the optimal control method, we start from the uncertainty bounds and then design the controller according to these bounds. Hence, if the controller exists, we can say the uncertain system is robustly stable.


## 3. Solving the Robust Control Problem via Adaptive Critic Learning

To obtain the optimal control solution (7), the unknown cost function (5) should be resolved. However, it is quite difficult to address the cost function (5) directly; then, a critic NN in this section will be proposed to approximate the cost function (5); this allows to develop an adaptive learning method to update online the NN weights, where the convergence of NN weights can be retained. Because its strong convergence, the actor NN widely used in the ADP schemes is removed. The proposed control system structure is given in Figure 1.Figure 1
Schematic of the proposed control system.This section will propose an adaptive critic learning method to obtain the solution of the derived optimal control problem. To this end, a critic NN is trained to estimate the cost functionV∗x, where the cost function V∗x is considered as continuous; hence, we have the following NN [13],(10)V∗x=WTσx+εvx,where W∈ℝl is the ideal critic NN weight, σx∈ℝl is the regressor vector, l is the number of neurons, and εvx is the approximate error of NN.Then, we have the partial derivative of cost function as(11)Vx∗x=∇σxTW+∇εvx,where ∇σx=∂σx/∂x∈ℝl×n is the regressor matrix and ∇εvx=∂εvx/∂x∈ℝn is the NN error.Without loss of generality, the following assumption is given in [13].Assumption 2.
The NN weightW, the regressor vector σ, the regressor matrix ∇σx, and the approximate errors εvx and ∇εvx are all bounded, i.e., W≤WM,σ≤σN,∇σ≤σM, and ∇εv≤σε for positive constants WM,σN,σM, and σε.
In fact, the ideal NN weightW is unknown; hence, we have that the practice W^ can be online updated; then, the actual cost function can be written as(12)V^x=W^Tσx.
Hence, the practical estimated optimal solution can be solved as(13)V^xx=∇σxTW^.
According to (10) and (11), we have the ideal optimal control (7) as(14)u∗=−12N−1gT∇σxTW+∇εvx,and its practical optimal control is given as(15)u^=−12N−1gT∇σxTW^.
The problem next to be solved is solving the unknown weightW^, which can guarantee the stability of the controlled system and realize the convergence to the ideal value W. Most existing ADP methods can only get the uniform ultimate boundedness (UUB) of the approximated NN weight rather than the convergence. In this paper, a novel adaptive critic learning method is introduced to guarantee the convergence of W^ to W. This strong convergence property is conducive to avoiding the use of actor NN, and the optimal control approximated via critic NN can converge to its ideal optimal solution.
Substituting (11) into (4), we can rewrite the HJB equation as(16)0=WT∇σxfx+gxu+dmax2x+xTMx+uTNu+εHJB,where εHJB=∇εvxTfx+gxu is the residual error determined by the approximation error ∇εvx.
For developing an adaptive critic learning law to estimate the critic NN weightW, the known terms in (16) can be defined as(17)Ξ=∇σxfx+gxu,Θ=dmax2x+xTMx+uTNu.
Then, the HJB equation (16) with (17) can be given as(18)Θ=−WTΞ−εHJB.
According to (18), only the NN weight W is unknown in this parameterized formulation. Hence, it can be estimated by using a recently proposed learning algorithm [19, 20], which is driven by the derived estimation error.
To this end, the filtered regressor matricesP∈ℝl×l and Q∈ℝl can be denoted as [19, 20](19)P˙=−ℓP+ΞΞT,P0=0,Q˙=−ℓQ+ΞΘ,Θ0=0,where ℓ>0 is a positive parameter. Hence, its solution can be derived as(20)P=∫0te−ℓt−sΞsΞTsds,Q=∫0te−ℓt−sΞsΘsds,which can be online calculated based on the system state x.
WithP and Q in (20), an auxiliary vector M1∈ℝl can be defined as(21)M1=−PW^+Q.
From (18) and (20), we have Q=−PW+v with v=−∫0te−ℓt−sεHJBsΞsds. A bounded variable, e.g., v≤εv, for a positive constant, εv>0. Then, we can obtain from (19)–(21) that(22)M1=−PW˜+v,with W˜=W−W^ being the NN weight estimation error.
The estimation error used in the adaptive learning algorithm can help to guarantee the convergence of the estimate, as shown in [13]. Hence, we can design the following adaptive law to online calculate W^ as(23)W^˙=−ΓM1,with Γ>0 being the adaptive learning gain.Remark 3.
The adaptive law (23) is driven by the estimation error W˜. The purpose of this new learning algorithm is to guarantee the convergence of estimate W^ to unknown weight W. Therefore, the learning algorithm given in this paper is different from those used in the existing ADP methods, e.g., [3, 21], which employ the gradient-based methods [22] to guarantee the boundedness of W^ only.
To illustrate the convergence of the proposed learning algorithm, the positive definiteness of the matrixP defined in (20) will be introduced:Lemma 2.
When the regressorΞ in (18) fulfills the persistent excitation (PE) condition, the matrix P defined in (20) is positive.
The convergence of the proposed learning algorithm can be summarized as follows.Theorem 1.
For the adopted critic NN with adaptive law (23), if the regressor vector Ξ in (18) satisfies the PE condition, the critic NN weight error W˜ exponentially converges to a small bounded set around zero.Proof.
For Lemma2, the matrix P is positive definite when the regressor Ξ satisfies the PE condition, i.e., the minimum eigenvalue λminP>δ>0. Hence, a Lyapunov function can be chosen as V1=1/2W˜TΓ−1W˜, and its derivative V˙1 along with (23) can be derived as(24)V˙1=W˜TΓ−1W˜˙=−W˜TPW˜+W˜Tv,which further implies(25)V˙1=W˜TΓ−1W˜+W˜Tv≤−W˜δW˜−εv.
Thus, we have that the estimation errors of NN weightW˜ will converge to a compact set Ω: W˜|‖W˜≤εv/δ, in which the size of this set depends on the approximation error εv and the excitation level δ, i.e., for an arbitrarily small NN approximation error (according to the NN approximation property, this error can be arbitrarily small for sufficient NN nodes, i.e., ∇εvx⟶0 with l⟶∞). Therefore, W^ can converge to W. In the ideal case, i.e., εHJB=0 and v=0, then we know the estimation errors of weights W˜ converge to zero exponentially.For system (2) with practical optimal control (15) and adaptive law (23), if the regressor Ξ satisfies the PE condition, the error W˜ converges to a small set around zero. Moreover, the actual optimal control u in (15) converges to a region around its optimal solution u∗ in (14), i.e., u−u∗≤εu. Hence, the original robust control problem is resolved.


## 4. Simulation

### 4.1. Numerical Simulation

Consider an uncertain system as(26)x˙=fx+gxu+bxu+gxdx,where fx=−x1+x2−0.5x1−0.5x21−cos2x2+22, gx=0cos2x1+2, x=x1,x2T∈ℝ2 is the system state, u∈ℝ is the control input, and the term kx=r2x22 with r2∈0,1 and dx=r1x1sinx2 with r1∈−1,1 denote the uncertainties.Because the uncertain termsbx=r2x22≥0 and dx=r1x1sin2x2 are bounded by dx=p1x1sin2x2≤x1=dmaxx, then we can obtain the optimal control problem as(27)x˙=fx+gxu,with the cost function as(28)Vx=∫t∞dmax2x+xTMx+uTNuds.As given in [18], the optimal cost function is written as(29)V∗x=x12+2x22.Then, we can obtain its optimal solution as(30)u∗=−12N−1gTVx∗=−cos2x1+22x2.A critic NN will be used to approximate the cost functionV; thus, the activation function σx is defined as(31)σx=x12,x1x2,x22T.To realize the simulation, we set the learning parametersΓ=100andℓ=10, the initial system state is given as x0=1,−0.2T, the initial weight W^0=0, and the weight matrices are set as M=IandN=I.Figure2 shows the estimated value of the evaluation NN weights. It can be seen from Figure 2 that the estimated NN weight converges to a certain value. This result verifies the convergence of Theorem 1 and the effectiveness of the proposed learning algorithm, which indicates that estimated critic NN weight W^ converges to its true value, i.e., W=1,0,2. To better display the performance of the proposed learning method, the error between the ideal cost function V∗ and practical coat function V^ is given in Figure 3, where we can obtain the fairly satisfactory approximation performance. In fact, the simulation results in Figures 2 and 3 can be also found in [13]; different from [13], this paper considers the uncertainties involved in control input. Figure 4 shows the change of the state of the controlled system under the derived optimal control, which shows that the closed-loop system is asymptotically stable. The corresponding control input is shown in Figure 5, bounded and smooth.Figure 2
Convergence of the critic NN weightsW^ [13].Figure 3
Critic NN approximation error of optimal cost function [13].Figure 4
Profile of system statex.Figure 5
The system input for proposed control.


### 4.2. Application to Robotic Systems

This section will develop a simulation based on a 2-DOF robot [18, 23]. To realize the simulation, the robotic systems model can be defined as(32)M¯qq¨+Cq,q˙+Fq˙+Gq=τ,where q is the joint variables, τ is the generalized forces, M¯q denotes the inertia matrix, Cq,q˙ is the centripetal vector, Fq is the friction vector, and Gq defines the gravity vector. In this section, we denote W¯q,q˙=Cq,q˙+Fq˙+Gq. There are uncertainties in M¯q and W¯q,q˙ due to the unknown load on the manipulator and unmodeled frictions.The inertia matrix can be derived as(33)M¯q=M¯11M¯12M¯21M¯22,where M¯11=J1+J2+m1r12+m2l12+m2r22+2m2l1r2cosq2+mLl12+mLl22+2mLl1l2cosq2, M¯12=J2+m2r22+m2l1r2cosq2+mLl22+mLl1l2cosq2, and M¯22=J2+m2r22+mLl22.The centripetal vector is(34)Cq,q˙=C1C2,where C1=m2l1r2+mLl1l22q˙1−q˙2q˙2sinq2 and C2=m2l1r2+mLl1l2q˙12sinq2.The friction vector and gravity matrix are(35)Fq˙=b1q˙1b2q˙2,Gq=G1G2,where G1=m1gr1+m2gr1+mLgr1sinq1+m2gr2+mLgl2sinq1+q2 and G2=m2gr2+mLgl2sinq1+q2.Some model parameters are given asm1=9.387/kg,m2=1.729/kg,l1=l2=0.250/m, b1=b2=0.14, r1=0.053/m, r2=0.170/m, J1=0.1190/kg.m2,J2=0.0652/kg.m2, mL=0,3kg, and g=9.8m/s. With above system dynamics, the state equation of the system can be given as [18](36)x˙=fx+gxu+bxu+gxdx,where x=x1,x2,x3,x4T=q1,q2,q3,q4T, fx=0x2,gx=01, bx=M¯x1−1M¯0x1−I≥0, M¯0 is the value of M¯ when the load is 0. dx=M¯q−1W¯0q,q˙−W¯q,q˙, and W¯0 is the value of W¯ when the load 0.In this simulation, we set the initial weightW^0=0; when the load mL=3kg, we choose the leaning parameters ℓ=21 and Γ=0.1 and weight matrices M=I and N=I. The initial states are q10=15°, q20=15°, q˙10=0, and q˙20=0.Figure6 shows the estimated critic NN weights. It can be seen from Figure 6 that the estimated NN weight converges to certain value. This result verifies the convergence of Theorem 1 and the effectiveness of the proposed learning algorithm. Figure 7 shows the change of the controlled system state under the derived optimal control when the load condition is set mL=3kg, which shows that the closed-loop system is asymptotically stable. The corresponding control input is shown in Figure 8. Although it jitters at first, it tends to be smooth when it stabilizes.Figure 6
Convergence of the critic NN weightsW^.Figure 7
Profile of system statex.Figure 8
The system input for proposed control.From above simulation results, we have that the proposed learning method and control technique are effective.


## 4.1. Numerical Simulation

Consider an uncertain system as(26)x˙=fx+gxu+bxu+gxdx,where fx=−x1+x2−0.5x1−0.5x21−cos2x2+22, gx=0cos2x1+2, x=x1,x2T∈ℝ2 is the system state, u∈ℝ is the control input, and the term kx=r2x22 with r2∈0,1 and dx=r1x1sinx2 with r1∈−1,1 denote the uncertainties.Because the uncertain termsbx=r2x22≥0 and dx=r1x1sin2x2 are bounded by dx=p1x1sin2x2≤x1=dmaxx, then we can obtain the optimal control problem as(27)x˙=fx+gxu,with the cost function as(28)Vx=∫t∞dmax2x+xTMx+uTNuds.As given in [18], the optimal cost function is written as(29)V∗x=x12+2x22.Then, we can obtain its optimal solution as(30)u∗=−12N−1gTVx∗=−cos2x1+22x2.A critic NN will be used to approximate the cost functionV; thus, the activation function σx is defined as(31)σx=x12,x1x2,x22T.To realize the simulation, we set the learning parametersΓ=100andℓ=10, the initial system state is given as x0=1,−0.2T, the initial weight W^0=0, and the weight matrices are set as M=IandN=I.Figure2 shows the estimated value of the evaluation NN weights. It can be seen from Figure 2 that the estimated NN weight converges to a certain value. This result verifies the convergence of Theorem 1 and the effectiveness of the proposed learning algorithm, which indicates that estimated critic NN weight W^ converges to its true value, i.e., W=1,0,2. To better display the performance of the proposed learning method, the error between the ideal cost function V∗ and practical coat function V^ is given in Figure 3, where we can obtain the fairly satisfactory approximation performance. In fact, the simulation results in Figures 2 and 3 can be also found in [13]; different from [13], this paper considers the uncertainties involved in control input. Figure 4 shows the change of the state of the controlled system under the derived optimal control, which shows that the closed-loop system is asymptotically stable. The corresponding control input is shown in Figure 5, bounded and smooth.Figure 2
Convergence of the critic NN weightsW^ [13].Figure 3
Critic NN approximation error of optimal cost function [13].Figure 4
Profile of system statex.Figure 5
The system input for proposed control.


## 4.2. Application to Robotic Systems

This section will develop a simulation based on a 2-DOF robot [18, 23]. To realize the simulation, the robotic systems model can be defined as(32)M¯qq¨+Cq,q˙+Fq˙+Gq=τ,where q is the joint variables, τ is the generalized forces, M¯q denotes the inertia matrix, Cq,q˙ is the centripetal vector, Fq is the friction vector, and Gq defines the gravity vector. In this section, we denote W¯q,q˙=Cq,q˙+Fq˙+Gq. There are uncertainties in M¯q and W¯q,q˙ due to the unknown load on the manipulator and unmodeled frictions.The inertia matrix can be derived as(33)M¯q=M¯11M¯12M¯21M¯22,where M¯11=J1+J2+m1r12+m2l12+m2r22+2m2l1r2cosq2+mLl12+mLl22+2mLl1l2cosq2, M¯12=J2+m2r22+m2l1r2cosq2+mLl22+mLl1l2cosq2, and M¯22=J2+m2r22+mLl22.The centripetal vector is(34)Cq,q˙=C1C2,where C1=m2l1r2+mLl1l22q˙1−q˙2q˙2sinq2 and C2=m2l1r2+mLl1l2q˙12sinq2.The friction vector and gravity matrix are(35)Fq˙=b1q˙1b2q˙2,Gq=G1G2,where G1=m1gr1+m2gr1+mLgr1sinq1+m2gr2+mLgl2sinq1+q2 and G2=m2gr2+mLgl2sinq1+q2.Some model parameters are given asm1=9.387/kg,m2=1.729/kg,l1=l2=0.250/m, b1=b2=0.14, r1=0.053/m, r2=0.170/m, J1=0.1190/kg.m2,J2=0.0652/kg.m2, mL=0,3kg, and g=9.8m/s. With above system dynamics, the state equation of the system can be given as [18](36)x˙=fx+gxu+bxu+gxdx,where x=x1,x2,x3,x4T=q1,q2,q3,q4T, fx=0x2,gx=01, bx=M¯x1−1M¯0x1−I≥0, M¯0 is the value of M¯ when the load is 0. dx=M¯q−1W¯0q,q˙−W¯q,q˙, and W¯0 is the value of W¯ when the load 0.In this simulation, we set the initial weightW^0=0; when the load mL=3kg, we choose the leaning parameters ℓ=21 and Γ=0.1 and weight matrices M=I and N=I. The initial states are q10=15°, q20=15°, q˙10=0, and q˙20=0.Figure6 shows the estimated critic NN weights. It can be seen from Figure 6 that the estimated NN weight converges to certain value. This result verifies the convergence of Theorem 1 and the effectiveness of the proposed learning algorithm. Figure 7 shows the change of the controlled system state under the derived optimal control when the load condition is set mL=3kg, which shows that the closed-loop system is asymptotically stable. The corresponding control input is shown in Figure 8. Although it jitters at first, it tends to be smooth when it stabilizes.Figure 6
Convergence of the critic NN weightsW^.Figure 7
Profile of system statex.Figure 8
The system input for proposed control.From above simulation results, we have that the proposed learning method and control technique are effective.


## 5. Conclusion

The purpose of this paper is to address the robust control problem of the uncertain systems via developing an adaptive critic learning method. To this end, the robust control problem of the uncertain systems is transformed into an optimal control problem of the nominal systems via selecting the cost function. Then, a single NN is used to reformulate the cost function, where the unknown cost function can be represented as a known term; then, an adaptive critic learning method based on the adaptive parameter estimation technique is presented to obtain the optimal cost function such that the optimal control problem can be solved. Simulations are given to show the effectiveness of the proposed leaning algorithm and control method. Future work will focus on the robust tracking control with uncertain systems.



---

*Source: 2952115-2021-11-16.xml*