# Robust Trajectory Tracking of Uncertain Systems via Adaptive Critic Learning

Authors: Ziliang Zhao; Qinglin Zhu; Bin Guo
Journal: Complexity
 (2022)

---

## Abstract

This study develops an adaptive dynamic programming (ADP) scheme for uncertain systems to achieve the robust trajectory tracking. In this framework, the augmented state is first established via combining the tracking error and reference trajectory, where the robust tracking control problem can be resolved using the regulation control strategy. Then, the robust control problem of uncertain system can be represented as an optimal control problem of nominal system, which provides a new pathway to address the robust control problem. To realize the optimal control, the derived Hamilton–Jacobi–Bellman equation (HJBE) is solved by training a critic neural network (CNN). Finally, two innovative critic learning techniques are suggested to calculate the unknown NN weights, where the convergence of NN weights can be guaranteed. Simulations are carried out to demonstrate the effectiveness of the proposed method.


---

## Body

## 1. Introduction

Considering the various and complex working conditions, some systems cannot avoid suffering from the uncertainties and nonlinearities, which brings challenge for higher control performance of these systems [1, 2]. Designing a robust controller for uncertain systems that can accommodate such uncertainties has always been an important yet challenging problem. To retain the robustness property, various robust control methods have been developed against model uncertainties [3–8]. Conventional robust control methods, however, were studied mainly via offline means. Moreover, the majority of existing robust control designs were devoted to addressing regulation issues [9], while the tracking problems of dynamic systems are even more difficult to be tackled. Although some satisfactory results have been yielded, the developed optimal control is implemented offline [10, 11].Adaptive dynamic programming (ADP) [12] is a new subject formed by the development of artificial intelligence and control. One of the well-known merits of the ADP method is that it can yield an approximation solution [13, 14] due to the employed critic neural network (CNN). Hence, it has been widely tailored and applied to solve the unknown cost function for optimal control. However, it is worth mentioning that the classical ADP framework requires an extra actor NN [12] to estimate the optimal control action, which results in a more complex ADP structure. To further reduce the computational burden, an innovative identifier-critic-based ADP structure was reported in [15], where only one CNN is used, and a new adaptive law is introduced to retain the convergence of CNN weights. Owing to its learning ability, recent work has also been carried out to solve specific robust control problems [16, 17]. Nevertheless, most existing ADP methods aim at the optimal or robust control regulation problem only.In fact, the trajectory tracking performance of uncertain systems is an important index to realize the stable and reliable work such as robotic systems. The tracking control of the robotic systems is mainly reflected in the tracking reference signal of each joint, so as to realize the end tracking to the desired trajectory. Therefore, it is necessary to design an appropriate controller for the uncertain nonlinear system to ensure the optimal tracking performance. Recently, some ADP-based tracking control results have been studied. In [18], an optimal tracking controller designed by the ADP was incorporated into steady-state control, by which the steady-state tracking response can be guaranteed, but the suboptimal control problem was obtained. In [19], an optimal tracking control scheme with an iterative learning method is introduced. Nevertheless, the aforementioned results are devoted to realizing the optimal tracking control without considering the uncertain system dynamics.Inspired by the above discussions, we will develop a novel ADP-based online adaptive critic method for robust trajectory tracking of systems with uncertain dynamics. To realize this purpose, the augmented state is established in terms of the tracking error and reference trajectory simultaneously, such that we can design the tracking controller from the perspective of regulation control problem. Then, the robust control problem is represented as an optimal control problem of the nominal systems with an appropriate index function; thus, a Hamilton–Jacobi–Bellman equation (HJBE) can be derived. Furthermore, two critic learning laws are presented to online calculate the unknown CNN weights; thus, the solution of HJBE can be resolved and the convergence of the CNN weights can be retained simultaneously. Finally, simulation results are provided to exemplify efficiency of the proposed method.The main contributions of this study are summarized as follows:(1)
Different from the classical robust control synthesis, the robust trajectory tracking problem is represented as an equivalent optimal regulation problem. Then, we can obtain the online solution using the critic learning algorithms based on ADP scheme.(2)
Based on the ADP framework, a simple CNN is used to reconstruct the cost function on the premise of ensuring convergence, by which the actor NN applied in the existing ADP structure is eliminated.(3)
A new adaptive critic learning law driven via CNN weights error is extracted to online training the CNN weights, where the convergence of the updated CNN weights can be guaranteed.The rest of the study is arranged as follows. In Section2, the system dynamics and problem formulation are described. Section 3 gives the robust tracking control design using the ADP scheme. Simulation results are provided in Section 4. Section 5 summarizes the main content of this study.


## 2. Problem Description

Consider the following system with uncertain dynamics:(1)x˙=fx+gxu+kxu+gxdx,where x∈ℝn denotes the state variables, u∈ℝn is the control input, fx and gx are the differentiable functions with f0=0, and kx and dx are the uncertain terms with k0=0 and d0=0. In this study, we presume the uncertainties kx≥0 and with fmax0=0 [10, 11].The objective of this study is achieving the trajectory tracking control of uncertain system (1); hence, the tracking error ex between the system output x and the reference trajectory xd∈ℝn is designed as(2)ex≜x−xd,where the reference trajectory xd is Lipschitz continuous.According to system (1) and reference trajectory xd, the time derivative of ex is calculated as(3)e˙x=x˙−x˙d=fx+gxu+kxu+gxdx−x˙d.Furthermore, define augmented stateY=exT,xdTT∈ℝ2n, and then, an augmented system can be derived along with (3):(4)Y.=fx−x˙dx˙d+gx0u+gx0kxu,+gx0dx,=AY+ℬYu+ℬYKYu+ℬYFY,where(5)AY=fx−x˙dx˙d,ℬY=gx0,KY=kx,FY=dx.Then, we have new uncertaintiesKY and FY as(6)KY≜kx≥0,FY≜fx≤fmaxx≜λdY.According to the above development, one can find that the original tracking control problem can be transformed into a regulation problem of system (4) [20]. Different from the existed result [20], there exists the uncertain terms KY and FY in (4), making a robust control problem. Inspired by [10, 21], we will further exploit the robust trajectory tracking control action via the optimal equation.Now, the nominal plant of system (4) can be defined:(7)Y.=AY+ℬYu.Then, the following cost function (8) should be minimized via an optimal control u∗ [20, 22, 23]:(8)VYt=∫t∞e−γs−tℱY+YTsQTYs+uTsRusds,where γ〉0 is the discount factor; this can guarantee the boundedness of the cost function even if the reference trajectory and the embedded feedforward control action do not converge to zero. QT=diagQ,0n×n, where Q∈ℝn×n and R∈ℝm×m denote the appropriate symmetric matrices.The cost function, defined byV∗, is the optimal value of V denoted in (8), i.e.,(9)V∗Yt=minuVYt.Using Bellman’s optimality principle toV∗Y in (9), we know the derivative VY∗Y of V∗Y satisfies(10)0=minuHY,u,VY=HY,u∗,VY∗,with(11)HY,u∗,VY∗=ℱY+YTQTY+u∗TRu∗+VY∗TAY+ℬYu∗−γV∗Y.Applying the optimal control theory, the control solution of the optimization problem is(12)u∗Y=−12R−1ℬTYVY∗Y.To implement the tracking control, we choose the utility functionℱY∈ as(13)ℱY=14VY∗ℬYTVY∗ℬY+λd2Y.Remark 1.
As discussed in [22], this study constructs an equivalence between the robust trajectory tracking problem and an optimal control problem of system (4) with the cost function (9), which gives a novel idea to resolve the robust trajectory tracking problem.


## 3. Solving Robust Tracking Control via ADP

This section will find an optimal solution forVY∗Y in (12). Within the ADP framework, a CNN is first constructed to update the optimal cost function; then, a CNN self-learning is proposed to learn robust tracking control solution. To improve the convergence of the CNN weights and transient performance, an adaptive learning scheme based on the estimated CNN weight error is developed to approximate the CNN weights; thus, the robust tracking control policy can be obtained.


### 3.1. CNN Design

Applying the universal approximation property, the estimated cost function is constructed via an CNN given in (14), which can be considered as a continuous function:(14)V∗Y=WTσ¯Y+εvY,where W∈ℝl is the ideal CNN weight, σ¯Y∈ℝl denotes the activation function, l is the number of hidden neurons, and εvY is the construction error. Its derivative along Y gives(15)VY∗Y=∇σ¯YTW+∇εvY,with ∇σ¯Y=∂σ¯Y/∂Y∈ℝl×2n and ∇εvY=∂εvY/∂Y∈ℝ2n.Generally, the optimal CNN weightW is to resolve; thus, the approximated W^ involved in cost function is expressed via(16)V^Y=W^Tσ¯Y,with W^ being the approximated CNN weight. Its derivative is(17)V^YY=∇σ¯YTW^.Based on (14) and (15), we can rewrite (12) as(18)u∗Y=−12R−1ℬTY∇σ¯YTW+∇εvY.The practical optimal controller becomes(19)uY=−12R−1ℬTY∇σ¯YTW^.From (19), one can claim that the tracking performance (i.e., x⟶xd) heavily relies on the estimation performance of W^. Nevertheless, most of the traditional learning algorithms based on ADP cannot make the CNN weights converge to the true values. Therefore, an actor NN needs to be introduced to stabilize the system. To overcome this issue, we will present a novel self-learning algorithm to online learn the CNN weights without using the actor NN.


### 3.2. Self-Learning Robust Tracking Control

This section will develop a self-learning algorithm to online learn the CNN weights, so as to obtain the robust tracking control. To this end, based on Section2, we have the estimated Hamiltonian function as(20)HY,u,W^=ℱY+YTQTY+uTRu+W^T∇σ¯YY.−γW^Tσ¯Y=ec.To realize the self-learning for CNN weights, we define the objective functionEc=1/2ecTec. Then, based on the standard steepest descent algorithm, the critic learning law can be designed as(21)W^˙=−δ∂Ec¯∂W^,with δ〉0 being the learning gain.From (20), we rewrite Hamiltonian function as(22)HY,u,W=ℱY+YTQTY+uTRu+WT∇σ¯YY.−γWTσ¯Y=eci,where eci=−∇εvYTY.+γεvY is the residual error due to the neural network approximation.DenoteW˜=W−W^; then, recalling (20) and (22), we have eci−ec=W˜T∇σ¯Y−γσ¯; let Θ=∇σ¯Y−γσ¯; then, we have eci−ec=W˜Θ with Θ≤ΘM, Θ≥0. Thus, the weight estimation error dynamic can be given as(23)W˜˙=−W^˙=δeci−W˜TΘΘ.Then, we include some results as follows.Theorem 1.
Consider system (7) with the learning law (21); the CNN weights’ error is uniform ultimate boundedness (UUB).Proof.
One chooses a Lyapunov function asJ1=1/δtrW˜TW˜; then, we can derive its time derivative:(24)J˙1=2δtrW˜TW˜˙=2δtrW˜Tδeci−W˜TΘΘ.
Applying some mathematical operation on (24), we have(25)J˙1≤−2−δW˜TΘ2+1δeci2.
Consider the Cauchy–Schwarz inequality and assumptionΘ≤ΘM; we can claim that J˙1〈0 for 0〈δ〈2 and(26)W˜eci2δ2−δΘM2.
From the Lyapunov theory, we can obtain the CNN weights error is UUB.
In this case, using the same Lyapunov method, we can getu−u∗≤κ with κ〉0. This UUB stability can refer to [24].


### 3.3. Adaptive Learning Robust Tracking Control

To realize the fast convergence of CNN weights, this section will develop a novel adaptive learning algorithm. To this end, we first substitute (15) into (10) as(27)ℱY+YTQTY+uTRu+WT−γσ¯+∇σ¯AY+ℬYu+εHJB=0,with εHJB=∇εvTAY+ℬYu−γεv being the HJBE approximate error.To approximate the CNN weightW with the novel adaptive critic learning algorithm, we first introduce two variables to redefine (27), such that(28)X=∇σ¯YAY+ℬYu,G=ℱY+YTQTY+uTRu..Then, based on (28), we rewrite (27) as(29)G=−WTX−εHJB.According to (29), the unknown CNN weights W are involved. Thus, the adaptive critic learning law can be designed to online approximate the CNN weights W. For this purpose, two auxiliary matrices U∈ℝl×l and W∈ℝl are denoted as(30)U.=−ℓU+XXT,U0=0W.=−ℓW+XG,W0=0,with ℓ〉0 being the positive constant. Thus, the solution of U and W can be calculated based on (30) as(31)U=∫0te−ℓt−sXsXTsds,W=∫0te−ℓt−sXsGsds.Furthermore, another auxiliary vectorM∈ℝl is defined along with U and W as(32)M=UW^+W⋅Based on (29) and (31), one can derive W=−UW+v with v=−∫0te−ℓt−sεHJBsXsds a bounded variable, i.e., v≤ε1v for a positive constant ε1v〉0. Then, the auxiliary vector M∈ℝl from (30)–(32) can be rewritten as(33)M=−UW˜+v,where W˜=W−W^ is the estimation error of NN weights.Therefore, an adaptive critic learning law can be designed via (33) as(34)W^˙=−TM,where T〉0 is the learning gain.It is noted that the critic learning law (34) contains the weights W˜, which can drive the estimate W^ toward to its true values, i.e., W⟶W^. This helps to achieve x tracking xd.Before proving the convergence property, the property of matrixX is examined:Lemma 1 (see [25, 26]).
If the variableX given in (28) satisfies the persistent excitation (PE) condition, then the matrix U presented in (31) is positive define.We can include the advantage of the designed learning technique as follows.Theorem 2.
For (29) with critic learning law (34), the weights error W˜ will exponentially converge to a compact set around zero provided that the vector G is persistently excited.Proof.
According to Lemma1, we have the minimal eigenvalue of U as λminU〉ϱ〉0. One chooses a Lyapunov function as V1=1/2W˜TT−1W˜; then, its time derivative can be calculated as(35)V.1=W˜TTW˜˙=−W˜TUW˜+W˜Tv≤−W˜ϱW˜−ε1v.
From (35), we can conclude that the estimation error W˜ converge to Ω:W˜|‖W˜|≤ε1v/ϱ, and the magnitude is decided by the construction error εv and the excitation level ϱ. Ideally, εHJB=0; we have that the convergence of estimated error W˜ can be guaranteed, i.e., W˜=0. This completes the proof.
Consider system (7) with practical optimal control (19) and critic learning law (34); the error W˜ and augmented system state Y are uniformly ultimately bounded (UUB) provided that the vector G satisfies the PE condition. Then, the practical optimal control u in (19) is convergent to its optimal solution u∗ given in (18), i.e., u−u∗≤εu.


### 3.4. Analysis for Robust Trajectory Tracking Performance

This section will show the tracking performance of controlled system.Theorem 3.
Considering the augmented plant (7) and cost function (8), the optimal control action (19) can guarantee the tracking error is UUB.Proof.
Based on (10) and (12), we have(36)V∗TYAY=−ℱY−YTQTY−u∗TRu∗−VY∗TℬYu∗+γV∗Y=−ℱY−YTQTY+u∗TRu∗+γV∗Y.
Using (36), we apply the optimal control (19) to the uncertain system (4). Because we can consider the cost function V∗Y as a Lyapunov function and the fact KY〉0, then its time derivative can be derived as(37)V.∗Y=VYTY.=−ℱY−YTQTY+u∗TRu∗+γV∗Y−2u∗TRu−2u∗TRKYu+VY∗ℬYFY≤−ℱY−YTQTY+u∗TRu∗+γV∗Y−2u∗TRu+VY∗ℬYFY.
Then, we have(38)V.∗Y≤−YTQTY+u∗TRu∗+γV∗Y−2u∗TRu−12VY∗ℬY−FYT12VY∗ℬY−FY−λd2+FYTFY.
Consider the factYTQTY=exTQex, and u∗TRu∗−2u∗TRu≤u−u∗TRu−u∗. We denote λmin⋅ and λmax⋅ as the minimal and maximal eigenvalues of a matrix, respectively. From the fact FY≤λd2Y and the optimal cost function being bounded by Bv〉0, we can finally derive (38):(39)V.∗Y≤exTQex+u−u∗TRu−u∗+γV∗≤−λminQex2+λmaxRΛ2+γBv.
To obtainV.Y〈0, the tracking error ex should satisfy the following condition ex:ex≤λmaxRΛ2+γBv/λminQ=Bex with Bex〉0 being the constant. Moreover, we denote Λ=εvorκ. Then, we have that applying the optimal action (19) to the uncertain system (4), the tracking error is UUB. This completes the proof.


## 3.1. CNN Design

Applying the universal approximation property, the estimated cost function is constructed via an CNN given in (14), which can be considered as a continuous function:(14)V∗Y=WTσ¯Y+εvY,where W∈ℝl is the ideal CNN weight, σ¯Y∈ℝl denotes the activation function, l is the number of hidden neurons, and εvY is the construction error. Its derivative along Y gives(15)VY∗Y=∇σ¯YTW+∇εvY,with ∇σ¯Y=∂σ¯Y/∂Y∈ℝl×2n and ∇εvY=∂εvY/∂Y∈ℝ2n.Generally, the optimal CNN weightW is to resolve; thus, the approximated W^ involved in cost function is expressed via(16)V^Y=W^Tσ¯Y,with W^ being the approximated CNN weight. Its derivative is(17)V^YY=∇σ¯YTW^.Based on (14) and (15), we can rewrite (12) as(18)u∗Y=−12R−1ℬTY∇σ¯YTW+∇εvY.The practical optimal controller becomes(19)uY=−12R−1ℬTY∇σ¯YTW^.From (19), one can claim that the tracking performance (i.e., x⟶xd) heavily relies on the estimation performance of W^. Nevertheless, most of the traditional learning algorithms based on ADP cannot make the CNN weights converge to the true values. Therefore, an actor NN needs to be introduced to stabilize the system. To overcome this issue, we will present a novel self-learning algorithm to online learn the CNN weights without using the actor NN.


## 3.2. Self-Learning Robust Tracking Control

This section will develop a self-learning algorithm to online learn the CNN weights, so as to obtain the robust tracking control. To this end, based on Section2, we have the estimated Hamiltonian function as(20)HY,u,W^=ℱY+YTQTY+uTRu+W^T∇σ¯YY.−γW^Tσ¯Y=ec.To realize the self-learning for CNN weights, we define the objective functionEc=1/2ecTec. Then, based on the standard steepest descent algorithm, the critic learning law can be designed as(21)W^˙=−δ∂Ec¯∂W^,with δ〉0 being the learning gain.From (20), we rewrite Hamiltonian function as(22)HY,u,W=ℱY+YTQTY+uTRu+WT∇σ¯YY.−γWTσ¯Y=eci,where eci=−∇εvYTY.+γεvY is the residual error due to the neural network approximation.DenoteW˜=W−W^; then, recalling (20) and (22), we have eci−ec=W˜T∇σ¯Y−γσ¯; let Θ=∇σ¯Y−γσ¯; then, we have eci−ec=W˜Θ with Θ≤ΘM, Θ≥0. Thus, the weight estimation error dynamic can be given as(23)W˜˙=−W^˙=δeci−W˜TΘΘ.Then, we include some results as follows.Theorem 1.
Consider system (7) with the learning law (21); the CNN weights’ error is uniform ultimate boundedness (UUB).Proof.
One chooses a Lyapunov function asJ1=1/δtrW˜TW˜; then, we can derive its time derivative:(24)J˙1=2δtrW˜TW˜˙=2δtrW˜Tδeci−W˜TΘΘ.
Applying some mathematical operation on (24), we have(25)J˙1≤−2−δW˜TΘ2+1δeci2.
Consider the Cauchy–Schwarz inequality and assumptionΘ≤ΘM; we can claim that J˙1〈0 for 0〈δ〈2 and(26)W˜eci2δ2−δΘM2.
From the Lyapunov theory, we can obtain the CNN weights error is UUB.
In this case, using the same Lyapunov method, we can getu−u∗≤κ with κ〉0. This UUB stability can refer to [24].


## 3.3. Adaptive Learning Robust Tracking Control

To realize the fast convergence of CNN weights, this section will develop a novel adaptive learning algorithm. To this end, we first substitute (15) into (10) as(27)ℱY+YTQTY+uTRu+WT−γσ¯+∇σ¯AY+ℬYu+εHJB=0,with εHJB=∇εvTAY+ℬYu−γεv being the HJBE approximate error.To approximate the CNN weightW with the novel adaptive critic learning algorithm, we first introduce two variables to redefine (27), such that(28)X=∇σ¯YAY+ℬYu,G=ℱY+YTQTY+uTRu..Then, based on (28), we rewrite (27) as(29)G=−WTX−εHJB.According to (29), the unknown CNN weights W are involved. Thus, the adaptive critic learning law can be designed to online approximate the CNN weights W. For this purpose, two auxiliary matrices U∈ℝl×l and W∈ℝl are denoted as(30)U.=−ℓU+XXT,U0=0W.=−ℓW+XG,W0=0,with ℓ〉0 being the positive constant. Thus, the solution of U and W can be calculated based on (30) as(31)U=∫0te−ℓt−sXsXTsds,W=∫0te−ℓt−sXsGsds.Furthermore, another auxiliary vectorM∈ℝl is defined along with U and W as(32)M=UW^+W⋅Based on (29) and (31), one can derive W=−UW+v with v=−∫0te−ℓt−sεHJBsXsds a bounded variable, i.e., v≤ε1v for a positive constant ε1v〉0. Then, the auxiliary vector M∈ℝl from (30)–(32) can be rewritten as(33)M=−UW˜+v,where W˜=W−W^ is the estimation error of NN weights.Therefore, an adaptive critic learning law can be designed via (33) as(34)W^˙=−TM,where T〉0 is the learning gain.It is noted that the critic learning law (34) contains the weights W˜, which can drive the estimate W^ toward to its true values, i.e., W⟶W^. This helps to achieve x tracking xd.Before proving the convergence property, the property of matrixX is examined:Lemma 1 (see [25, 26]).
If the variableX given in (28) satisfies the persistent excitation (PE) condition, then the matrix U presented in (31) is positive define.We can include the advantage of the designed learning technique as follows.Theorem 2.
For (29) with critic learning law (34), the weights error W˜ will exponentially converge to a compact set around zero provided that the vector G is persistently excited.Proof.
According to Lemma1, we have the minimal eigenvalue of U as λminU〉ϱ〉0. One chooses a Lyapunov function as V1=1/2W˜TT−1W˜; then, its time derivative can be calculated as(35)V.1=W˜TTW˜˙=−W˜TUW˜+W˜Tv≤−W˜ϱW˜−ε1v.
From (35), we can conclude that the estimation error W˜ converge to Ω:W˜|‖W˜|≤ε1v/ϱ, and the magnitude is decided by the construction error εv and the excitation level ϱ. Ideally, εHJB=0; we have that the convergence of estimated error W˜ can be guaranteed, i.e., W˜=0. This completes the proof.
Consider system (7) with practical optimal control (19) and critic learning law (34); the error W˜ and augmented system state Y are uniformly ultimately bounded (UUB) provided that the vector G satisfies the PE condition. Then, the practical optimal control u in (19) is convergent to its optimal solution u∗ given in (18), i.e., u−u∗≤εu.


## 3.4. Analysis for Robust Trajectory Tracking Performance

This section will show the tracking performance of controlled system.Theorem 3.
Considering the augmented plant (7) and cost function (8), the optimal control action (19) can guarantee the tracking error is UUB.Proof.
Based on (10) and (12), we have(36)V∗TYAY=−ℱY−YTQTY−u∗TRu∗−VY∗TℬYu∗+γV∗Y=−ℱY−YTQTY+u∗TRu∗+γV∗Y.
Using (36), we apply the optimal control (19) to the uncertain system (4). Because we can consider the cost function V∗Y as a Lyapunov function and the fact KY〉0, then its time derivative can be derived as(37)V.∗Y=VYTY.=−ℱY−YTQTY+u∗TRu∗+γV∗Y−2u∗TRu−2u∗TRKYu+VY∗ℬYFY≤−ℱY−YTQTY+u∗TRu∗+γV∗Y−2u∗TRu+VY∗ℬYFY.
Then, we have(38)V.∗Y≤−YTQTY+u∗TRu∗+γV∗Y−2u∗TRu−12VY∗ℬY−FYT12VY∗ℬY−FY−λd2+FYTFY.
Consider the factYTQTY=exTQex, and u∗TRu∗−2u∗TRu≤u−u∗TRu−u∗. We denote λmin⋅ and λmax⋅ as the minimal and maximal eigenvalues of a matrix, respectively. From the fact FY≤λd2Y and the optimal cost function being bounded by Bv〉0, we can finally derive (38):(39)V.∗Y≤exTQex+u−u∗TRu−u∗+γV∗≤−λminQex2+λmaxRΛ2+γBv.
To obtainV.Y〈0, the tracking error ex should satisfy the following condition ex:ex≤λmaxRΛ2+γBv/λminQ=Bex with Bex〉0 being the constant. Moreover, we denote Λ=εvorκ. Then, we have that applying the optimal action (19) to the uncertain system (4), the tracking error is UUB. This completes the proof.


## 4. Simulations’ Verification

This section will first present numerical simulations to show the effectiveness of proposed method and then apply proposed methods to the 2-DOF robotic system to illustrate the applicability.


### 4.1. Numerical Simulations

In this section, we will provide numerical simulation results to illustrate the effectiveness of the developed method in terms of an uncertain system [27]:(40)x˙=x2−0.5x1+x2+0.5x12x2+011+ku+01d,where x=x1,x2T∈ℝ2 is the system state, u∈ℝ denotes the system input, and the terms kx=p1x2 with p1=0,1 and dx=p2x1sinx2+p3 with p2=−1,1 and p3=−1,1 are the uncertain parameters.The system initial state is set asx0=−0.5,0.5T and the desired trajectory is chosen as xd1=0.5sint and xd2=0.5cost. Then, from the fact ex=x−xd and xd=xd1,xd2T, we can construct the augmented system as (4). Hence, we can obtain(41)KY≜kx≥0FY≜dx≤fmaxx≜λdY=x1.To obtain the optimal control solution, we set matricesQT=diag8I2,02×2, R=1. Moreover, the discounted factor in cost function (8) is set as λ=0.05.The other parameters in this simulation are selected asΓ=18.5 and ℓ=15.8; the parameters in uncertain terms are selected as p1=1, p2=0.5, and p3=1.To obtain the optimal control action, we apply a single NN to reformulate HJBE (27), i.e., estimate V∗; its convergence of weights can be guaranteed using the learning law (34). In the process of simulation, we set the activation function of CNN as σ¯=Y12,Y1Y2,Y1Y3,Y1Y4,Y22,Y2Y3,Y2Y4,Y32,Y3Y4,Y42T and the initial values for the CNN weights are chosen as W^0=0. After the online learning of t=25s, the CNN weights are convergence to its certain values, which are displayed in Figure 1. Then, the online updated CNN weights are applied to robust trajectory tracking implementation. Figure 2 gives the tracking performance of uncertain system (44). To better show the effectiveness of the developed method, the tracking error between practical system output x and desired trajectory xd is provided in Figure 3, which can converge to zero, and the proposed tracking control action is also displayed in Figure 3, which is bounded and smooth.Figure 1
Profiles of CNN weights with (34).Figure 2
Tracking performance for uncertain system (44) with (34).Figure 3
Tracking error and proposed control action with (34).To show the effectiveness of the proposed single critic NN-based ADP, a critic-actor NN-based ADP method [18] is proposed to use for comparison. The critic-actor weights can be found in Figure 4, and the corresponding control performances are given in Figure 5. We have that the proposed single critic NN-based ADP has the advantages of fast convergence speed and high control accuracy.Figure 4
Profiles of critic-actor NN weights [18].Figure 5
Tracking error and control actionu for nonlinear systems [18].To further show the robustness of the suggested control, we will give other two cases: (1)p1=1, p2=1, and p3=1 and (2) p1=0.5, p2=1, and p3=0.5. Then, the tracking performance and control action can be yielded in Figures 6–9, where we have that even under different uncertainties, the system can effectively track the reference trajectory; i.e., the robustness of the designed controller can be guaranteed.Figure 6
Tracking performance for uncertain system (44) with (34) (case 1).Figure 7
Tracking error and proposed control action with (34) (case 1).Figure 8
Tracking performance for uncertain system (44) with (34) (case 2).Figure 9
Tracking error and proposed control action with (34) (case 2).Finally, some simulations for (21) are given in Figures 10 and 11. From these results, we have that the convergence of CNN weights in Figure 10 is more than that in Figure 1 and used adaptive learning 34; this lead to a poor tracking performance, as shown in Figure 11.Figure 10
Profiles of CNN weights with (21).Figure 11
Tracking error and proposed control action with (21).


### 4.2. Application to Robotic Systems

In this section, comparative simulations are tested out via a 2-DOF robotic system, to illustrate the feasibility of the presented adaptive robust tracking control method. We first give the dynamics of the robotic system as [28](42)ℋqq¨+C'q,q˙+Fq˙+Gq=τ,where q∈ℝn denotes the join variables, τ∈ℝn is the generalized forces, ℋq∈ℝn×n is the inertia matrix, C0q,q˙∈ℝn represents the Coriolis/centripetal vector, Gq∈ℝn is the gravity vector, and Fq˙∈ℝn is the friction vector. This study denotes Nq,q˙=C0q,q˙+Fq˙+Gq. ℋq and C0q,q˙ are uncertain because the unknown load and unmodeled frictions. The inertia matrix is(43)ℋq=ℋ11ℋ12ℋ21ℋ22,where ℋ11=J1+J2+m1r12+m2l12+m2r22+2m2l1r2cosq2+mLl12+mLl22+2mLl1l2cosq2. ℋ12=J2+m2r22+m2l1r2cosq2+mLl22+mLl1l2cosq2, and ℋ22=J2+m2r22+mLl22.The centripetal vector is(44)C'q,q˙=C'1C'2.where C01=m2l1r2+mLl1l22q˙1−q˙2q˙2sinq2 and C02=m2l1r2+mLl1l2q˙12sinq2.The friction vector and gravity matrix denote(45)Fq˙=b1q˙1b2q˙2;Gq=G1G2,where G1=m1gr1+m2gr1+mLgr1sinq1+m2gr2+mLgl2sinq1+q2 and G2=m2gr2+mLgl2sinq1+q2.Based on the above facts, we can define the state equation of system (42) as(46)x=fx+gxu+kxu+gxdx,where x=x1,x2,x3,x4T=q1,q2,q˙1,q˙2T, fx=x2,0T, gx=0,IT, kx=ℋx1−1ℋ0x1−I≥0, and dx=ℋ−1qN0q,q˙−Nq,q˙.To realize experimental verification, the dynamic parameters of this 2-DOF robot are given in Table1. The upper bound of uncertainty dx is given:(47)dx=ℋ−1x1N0x1,x2−Nx1,x2≤ℋx1−1N0x1,x2−Nx1,x2≤ℋminx1−1n¯maxx1,x2=fmaxx.Table 1
Dynamic parameters of 2-DOF robot.
SymbolMeaningValuesJ1The moment of inertia of link 10.0856J2The moment of inertia of link 20.0707b1q˙1The friction at joint 10.014q˙1b2q˙2The friction at joint 20.035q˙2m1Mass of joint 14.467m2Mass of joint 21.694l1Length 10.250l2Length 20.250r1The center of mass0.079r2The center of mass0.085mLMass of load[0, 3]gGravitational acceleration9.8To compete the tracking control, the sinusoidal signalsq1d=0.5sin5t, q2d=0.55cos5t, q˙1d=q2d, and q˙2d=−5q1d are chosen as the desired trajectories. The regressor vector of CNN is designed as σ¯=Y12,Y22,Y32,Y42,Y52,Y62,Y72,Y82T. Because the uncertainties in ℋq and C0q,q˙ are existed due to issues such as the different load to be picked, then we consider the following cases.Case 1.
In this case, we will test the tracking response under the loadmL=0.5kg. The control parameters are selected as λ=0.05, ℓ=0.011, Γ=0.019, Q=1, and R=1.
Figure12 shows the learning results of the online updated weights W^ using the proposed method (34). Then, based on the solved results, the profiles of the tracking performance are given in Figures 13 and 14, in which one can find that both the proposed control method (34) can make both q1, q2 and q˙1, q˙2 track the given command perfectly. This can be further found in Figure 15, where the tracking errors are given. Figure 16 shows the control action, which is bounded.
Since the robotic system uncertainty mostly comes from the uncertainty of end load, then we set a Case2; the maximum end load is given in this case.Figure 12
Profiles of CNN weights.Figure 13
Tracking performance forq1 and q2.Figure 14
Tracking performance forq˙1 and q˙2.Figure 15
Tracking errors ofq1 and q2.Figure 16
The required control input forq1 and q2.Case 2.
In this case, we will test the tracking response under the loadmL=3kg, i.e., maximum end load. The control parameters are given as: λ=0.05, ℓ=0.011, Γ=0.019, Q=1, and R=1.
Based on the learning results in Figure12, the tracking response in this case is given in Figures 17 and 18, and the corresponding tracking errors are given Figure 19. From Figures 17–19, we have that even if the end load of the robot becomes larger, axis 1 and axis 2 of the robotic system can track accurately on the basis of previous learning.
Finally, we give some results for proposed learning (21), as shown Figures 20 and 21, in which we have that although the accurate tracking of the robot can finally be realized, compared with the learning law (34), the learning time of neural network weight is too long, which will reduce the transient of tracking.
The above simulation results show the effectiveness of the proposed control method and the feasibility of the learning algorithm.Figure 17
Tracking performance forq1 and q2.Figure 18
Tracking performance forq˙ and q˙d.Figure 19
Tracking errors ofq1 and q2.Figure 20
Profiles of CNN weights with (21).Figure 21
Tracking errors ofq1 and q2.


## 4.1. Numerical Simulations

In this section, we will provide numerical simulation results to illustrate the effectiveness of the developed method in terms of an uncertain system [27]:(40)x˙=x2−0.5x1+x2+0.5x12x2+011+ku+01d,where x=x1,x2T∈ℝ2 is the system state, u∈ℝ denotes the system input, and the terms kx=p1x2 with p1=0,1 and dx=p2x1sinx2+p3 with p2=−1,1 and p3=−1,1 are the uncertain parameters.The system initial state is set asx0=−0.5,0.5T and the desired trajectory is chosen as xd1=0.5sint and xd2=0.5cost. Then, from the fact ex=x−xd and xd=xd1,xd2T, we can construct the augmented system as (4). Hence, we can obtain(41)KY≜kx≥0FY≜dx≤fmaxx≜λdY=x1.To obtain the optimal control solution, we set matricesQT=diag8I2,02×2, R=1. Moreover, the discounted factor in cost function (8) is set as λ=0.05.The other parameters in this simulation are selected asΓ=18.5 and ℓ=15.8; the parameters in uncertain terms are selected as p1=1, p2=0.5, and p3=1.To obtain the optimal control action, we apply a single NN to reformulate HJBE (27), i.e., estimate V∗; its convergence of weights can be guaranteed using the learning law (34). In the process of simulation, we set the activation function of CNN as σ¯=Y12,Y1Y2,Y1Y3,Y1Y4,Y22,Y2Y3,Y2Y4,Y32,Y3Y4,Y42T and the initial values for the CNN weights are chosen as W^0=0. After the online learning of t=25s, the CNN weights are convergence to its certain values, which are displayed in Figure 1. Then, the online updated CNN weights are applied to robust trajectory tracking implementation. Figure 2 gives the tracking performance of uncertain system (44). To better show the effectiveness of the developed method, the tracking error between practical system output x and desired trajectory xd is provided in Figure 3, which can converge to zero, and the proposed tracking control action is also displayed in Figure 3, which is bounded and smooth.Figure 1
Profiles of CNN weights with (34).Figure 2
Tracking performance for uncertain system (44) with (34).Figure 3
Tracking error and proposed control action with (34).To show the effectiveness of the proposed single critic NN-based ADP, a critic-actor NN-based ADP method [18] is proposed to use for comparison. The critic-actor weights can be found in Figure 4, and the corresponding control performances are given in Figure 5. We have that the proposed single critic NN-based ADP has the advantages of fast convergence speed and high control accuracy.Figure 4
Profiles of critic-actor NN weights [18].Figure 5
Tracking error and control actionu for nonlinear systems [18].To further show the robustness of the suggested control, we will give other two cases: (1)p1=1, p2=1, and p3=1 and (2) p1=0.5, p2=1, and p3=0.5. Then, the tracking performance and control action can be yielded in Figures 6–9, where we have that even under different uncertainties, the system can effectively track the reference trajectory; i.e., the robustness of the designed controller can be guaranteed.Figure 6
Tracking performance for uncertain system (44) with (34) (case 1).Figure 7
Tracking error and proposed control action with (34) (case 1).Figure 8
Tracking performance for uncertain system (44) with (34) (case 2).Figure 9
Tracking error and proposed control action with (34) (case 2).Finally, some simulations for (21) are given in Figures 10 and 11. From these results, we have that the convergence of CNN weights in Figure 10 is more than that in Figure 1 and used adaptive learning 34; this lead to a poor tracking performance, as shown in Figure 11.Figure 10
Profiles of CNN weights with (21).Figure 11
Tracking error and proposed control action with (21).


## 4.2. Application to Robotic Systems

In this section, comparative simulations are tested out via a 2-DOF robotic system, to illustrate the feasibility of the presented adaptive robust tracking control method. We first give the dynamics of the robotic system as [28](42)ℋqq¨+C'q,q˙+Fq˙+Gq=τ,where q∈ℝn denotes the join variables, τ∈ℝn is the generalized forces, ℋq∈ℝn×n is the inertia matrix, C0q,q˙∈ℝn represents the Coriolis/centripetal vector, Gq∈ℝn is the gravity vector, and Fq˙∈ℝn is the friction vector. This study denotes Nq,q˙=C0q,q˙+Fq˙+Gq. ℋq and C0q,q˙ are uncertain because the unknown load and unmodeled frictions. The inertia matrix is(43)ℋq=ℋ11ℋ12ℋ21ℋ22,where ℋ11=J1+J2+m1r12+m2l12+m2r22+2m2l1r2cosq2+mLl12+mLl22+2mLl1l2cosq2. ℋ12=J2+m2r22+m2l1r2cosq2+mLl22+mLl1l2cosq2, and ℋ22=J2+m2r22+mLl22.The centripetal vector is(44)C'q,q˙=C'1C'2.where C01=m2l1r2+mLl1l22q˙1−q˙2q˙2sinq2 and C02=m2l1r2+mLl1l2q˙12sinq2.The friction vector and gravity matrix denote(45)Fq˙=b1q˙1b2q˙2;Gq=G1G2,where G1=m1gr1+m2gr1+mLgr1sinq1+m2gr2+mLgl2sinq1+q2 and G2=m2gr2+mLgl2sinq1+q2.Based on the above facts, we can define the state equation of system (42) as(46)x=fx+gxu+kxu+gxdx,where x=x1,x2,x3,x4T=q1,q2,q˙1,q˙2T, fx=x2,0T, gx=0,IT, kx=ℋx1−1ℋ0x1−I≥0, and dx=ℋ−1qN0q,q˙−Nq,q˙.To realize experimental verification, the dynamic parameters of this 2-DOF robot are given in Table1. The upper bound of uncertainty dx is given:(47)dx=ℋ−1x1N0x1,x2−Nx1,x2≤ℋx1−1N0x1,x2−Nx1,x2≤ℋminx1−1n¯maxx1,x2=fmaxx.Table 1
Dynamic parameters of 2-DOF robot.
SymbolMeaningValuesJ1The moment of inertia of link 10.0856J2The moment of inertia of link 20.0707b1q˙1The friction at joint 10.014q˙1b2q˙2The friction at joint 20.035q˙2m1Mass of joint 14.467m2Mass of joint 21.694l1Length 10.250l2Length 20.250r1The center of mass0.079r2The center of mass0.085mLMass of load[0, 3]gGravitational acceleration9.8To compete the tracking control, the sinusoidal signalsq1d=0.5sin5t, q2d=0.55cos5t, q˙1d=q2d, and q˙2d=−5q1d are chosen as the desired trajectories. The regressor vector of CNN is designed as σ¯=Y12,Y22,Y32,Y42,Y52,Y62,Y72,Y82T. Because the uncertainties in ℋq and C0q,q˙ are existed due to issues such as the different load to be picked, then we consider the following cases.Case 1.
In this case, we will test the tracking response under the loadmL=0.5kg. The control parameters are selected as λ=0.05, ℓ=0.011, Γ=0.019, Q=1, and R=1.
Figure12 shows the learning results of the online updated weights W^ using the proposed method (34). Then, based on the solved results, the profiles of the tracking performance are given in Figures 13 and 14, in which one can find that both the proposed control method (34) can make both q1, q2 and q˙1, q˙2 track the given command perfectly. This can be further found in Figure 15, where the tracking errors are given. Figure 16 shows the control action, which is bounded.
Since the robotic system uncertainty mostly comes from the uncertainty of end load, then we set a Case2; the maximum end load is given in this case.Figure 12
Profiles of CNN weights.Figure 13
Tracking performance forq1 and q2.Figure 14
Tracking performance forq˙1 and q˙2.Figure 15
Tracking errors ofq1 and q2.Figure 16
The required control input forq1 and q2.Case 2.
In this case, we will test the tracking response under the loadmL=3kg, i.e., maximum end load. The control parameters are given as: λ=0.05, ℓ=0.011, Γ=0.019, Q=1, and R=1.
Based on the learning results in Figure12, the tracking response in this case is given in Figures 17 and 18, and the corresponding tracking errors are given Figure 19. From Figures 17–19, we have that even if the end load of the robot becomes larger, axis 1 and axis 2 of the robotic system can track accurately on the basis of previous learning.
Finally, we give some results for proposed learning (21), as shown Figures 20 and 21, in which we have that although the accurate tracking of the robot can finally be realized, compared with the learning law (34), the learning time of neural network weight is too long, which will reduce the transient of tracking.
The above simulation results show the effectiveness of the proposed control method and the feasibility of the learning algorithm.Figure 17
Tracking performance forq1 and q2.Figure 18
Tracking performance forq˙ and q˙d.Figure 19
Tracking errors ofq1 and q2.Figure 20
Profiles of CNN weights with (21).Figure 21
Tracking errors ofq1 and q2.


## 5. Conclusion

This study aims at resolving the robust trajectory tracking of uncertain systems, which adopts the ideas of ADP presented for optimal control. The basic concept is to represent the robust control problem as an equivalent optimal control problem for the nominal system, where the augmented state is considered. Thus, we can resolve this robust tracking control problem using an optimal control method. To obtain the optimal control solution, two adaptive critic learning techniques are developed via ADP scheme, where the parameter estimation method is adopted. The closed-loop system stability and robust tracking performance are rigorously proved; simulation results are given to exemplify the feasibility of the developed method. In our future work, we will extend the proposed idea to address the robust tracking control problem with several unmatched dynamics cases, which allows to carry out practical experimental validations based on existing test rigs in our lab.



---

*Source: 8701272-2022-05-24.xml*