# Online Optimal Control of Robotic Systems with Single Critic NN-Based Reinforcement Learning

Authors: Xiaoyi Long; Zheng He; Zhongyuan Wang
Journal: Complexity
 (2021)

---

## Abstract

This paper suggests an online solution for the optimal tracking control of robotic systems based on a single critic neural network (NN)-based reinforcement learning (RL) method. To this end, we rewrite the robotic system model as a state-space form, which will facilitate the realization of optimal tracking control synthesis. To maintain the tracking response, a steady-state control is designed, and then an adaptive optimal tracking control is used to ensure that the tracking error can achieve convergence in an optimal sense. To solve the obtained optimal control via the framework of adaptive dynamic programming (ADP), the command trajectory to be tracked and the modified tracking Hamilton-Jacobi-Bellman (HJB) are all formulated. An online RL algorithm is the developed to address the HJB equation using a critic NN with online learning algorithm. Simulation results are given to verify the effectiveness of the proposed method.


---

## Body

## 1. Introduction

In the control field and practical applications, reinforcement learning (RL) [1, 2] and adaptive dynamic programming (ADP) [3, 4] play a critical role to address the optimal control problems. The purpose of optimal control is to design a stabilizing control law by minimizing a predefined performance function. A lot of work focusing on the regulation problem for optimal control using the RL/ADP algorithms has been reported [5, 6] in the past years. The objective is to solve an optimal control that can maximize or minimize the system output energy and control actions, where the associated optimal control equations can be numerically solved via neural networks (NNs). From the perspective of both the theoretical study and practical application, these results pave a new way to solve the optimal control problems. Relevant surveys about the recent developments on the RL and ADP can be referred to [7, 8].RL was first developed in the intelligent control field, which was used to address the discrete-time Markov decision problems. Then, it has been extended to solve the continuous-time (CT) systems. With respect to optimal control designs, Abu-Khalaf et al. [9] suggested a policy iteration (PI) for the optimal regulation of CT nonlinear systems with actuator saturation. To overcome the problem of using the time derivatives of CT dynamics, Lewis et al. [10] developed an integral RL (IRL) technique for systems with partially known dynamics, where the full system information is avoided. In [11], the authors have employed an actor-critic structure and developed a synchronous PI algorithm for CT systems. In this framework, both the optimal cost function and control policy are estimated using NNs, whose weights are updated online simultaneously. For completely unknown system dynamics, the results in [12] showed that a model-free PI approach can be developed for CT linear systems, which can online calculate the optimal solutions using the input/output measurements. This principle was subsequently extended to nonlinear systems in [13, 14]. Another intelligent learning method used in the RL, experience replay (ER), was recently incorporated into the synthesis of ADP for optimal control in [15], where the past system observations are utilized together with the current information to enhance the convergence speed of the online learning.Moreover, most existing results on the ADP-based optimal control designs focus on the optimal regulation problems only. However, in the practical application, optimal tracking control problem (OTCP) is more widely used than the optimal regulation problem [16–18], in particular for robotic applications. However, the OTCP is more challenging to address than the regulation problem, since its solution is usually composed of a feedforward action to guarantee the perfect tracking and a feedback action to stabilize the closed-loop system dynamics [19]. For linear systems, the solution of OTCP is calculated by addressing the Riccati equations [20], while for nonlinear systems, the existing solution for the OTCP can be derived with a feedforward term by employing the dynamics inversion and a feedback term by calculating a complex HJB equation [16, 21]. However, it is well known that deriving the solution of OTCP is typically intractable, especially for the online tracking control. Hence, only few results have been reported to address the OTCP in the literature, in particular for robotic systems.According to the above facts, we propose a new RL algorithm to realize the optimal tracking control of robotic systems. To this end, the system model is rewritten as a state-space form, which will contribute to the realization of optimal tracking control. Then, a steady-state control is designed, and then an adaptive optimal tracking control is used to retain that the tracking error converges in an optimal manner. To derive this optimal control, the command trajectory to be tracked and the modified tracking Hamilton–Jacobi–Bellman (HJB) can be formulated. Finally, an online RL method is used to address the derived HJB equation using a single critic NN approximation. Numerical simulations are also given to show the validity of the proposed approach. The contributions can be summarized as follows:(1)
To achieve the optimal tracking control, the robotic system model is transformed into a canonical form, which will contribute to the realization of optimal tracking control.(2)
A critic NN is applied to reconstruct the cost function with guaranteed convergence, such that the actor NN used in the existing ADP structures is avoided and the computational costs can be reduced.(3)
A RL algorithm is proposed to obtain the solution of the derived HJB equation, which can guarantee the convergence of critic NN weights to ensure the optimal tracking error convergence.The paper is structured as follows: in Section2, the system model is transformed into a canonical form, and a tracking performance function is constructed. In Section 3, an adaptive steady-state control is designed and an optimal control is developed with RL to make the tracking error dynamics convergent. For this purpose, a single critic NN is applied to estimate the solution of the HJB equation and update the optimal control action. Section 4 gives some simulation results to show the validity of the developed control and learning techniques. Conclusions are summarized in Section 5.Notations:ℜ denotes the real number set. ℜn is the n-dimensional real vector. ℜn×mis the real matrices. ⋅ denotes the Euclidean norm of a vector in ℜn or a matrix in ℜn×m. I is the identity matrix, and 0×0 means the zero matrix. λmax and λmin are the maximal and minimum eigenvalues of a matrix, respectively. diaga1,a2,a3,…,an is a diagonal matrix with component a1,…,an.⋅x=∂⋅/∂x defines the partial differential operation.


## 2. Preliminaries and Problem Statement

In this paper, we consider the general rigid body dynamics for a nonlinear robotic manipulator. On the basis of the Lagrangian formulation, the manipulator dynamics can be formulated as [22, 23](1)Mqq¨+Cq,q˙q˙+Gq=τ,where q=qt∈ℜn denotes the generalized coordinates representing the joint position and q˙,q¨ denote the derivatives of joint position (e.g., velocity and acceleration) with respect to time t. Let n denote the number of degrees of freedom and Mq denote a positive definite n×n inertia matrix which is invertible, Cq,q˙∈ℜn are the Coriolis/centripetal dynamics, and Gq represents the gravitational dynamics.For the brevity of notation, we setx1=q,x2=q˙,andu=τ; then,(2)x˙1=x2,x˙2=−M−1Cx2−M−1G+M−1u.Hence, we write system (1) as a state-space form as(3)x˙t=fx+gut,wherext=qq˙T∈ℜn is the system state, fx=x2−M−1Cx2+GT is the known system dynamics, which is a continuous function with f0=0, g=0M−1Tis the control gain matrix, and ut∈ℜm denotes the control torque.Assumption 1.
The system dynamicsfx with f0=0 are Lipschitz. Hence, system (3) is stable, i.e., a continuous control ut∈Ω can be found to stabilize the system, where Ω is an admissible set.
This paper aims to find an optimal controlut to ensure that the system state xt tracks a desired trajectory xdt by minimizing the following cost function:(4)Vut,et=∫t∞eTtQet+uTtRutdt,where Q and R are positive definite symmetric matrices [2]. ΨΩ is the set of admissible policies [9], eTtQet+uTtRut is a positive utility function, and et is the tracking error defined as(5)et=xt−xdt,where xdt∈ℜn is the reference trajectory. In this paper, the reference trajectory xdt and its derivative x˙dt are assumed to be continuous and bounded.Remark 1.
There have been several methods developed to solve the tracking control problem of robotic systems, e.g., [24, 25]. However, most existing robotic controllers are not designed in the optimal manner, i.e., the required control actions may be large. Differing from these results, a novel RL algorithm is proposed in the following sections to design an optimal control for robotic systems to achieve trajectory tracking and reduce the required control energy.Remark 2.
For linear OTCP scheme, the derivation of optimal solution is shown in [18], i.e., u=−R−1B0TSx+R−1B0Tvss with B0 being the input matrix, and S can be solved by addressing a Riccati equation and vss is the feedforward action. Different from linear systems, the tracking control problem for nonlinear systems is not trivial since we cannot obtain a uniform formulation as the linear cases. This fact stimulated the current study.


## 3. Online Dynamic Tracking Control

To realize the optimal tracking control design, we decompose the control inputu into two parts [18, 21] as(6)u=us+ue,where us and ueare the steady-state control and optimal control, which are applied to make the steady-state trajectory tracking and stabilize the tracking error dynamics optimally, respectively. Figure 1 shows the proposed control system structure.Figure 1
Schematic of the proposed control system.


### 3.1. Steady-State Tracking Control

Sinceus can be adopted to ensure that the control error converges to zero in the steady-state, then we have from (3) that(7)us=g+x˙d−fx−Ke,where K>0 is the feedback gain set by the desingers and g+=gTg−1gT is defined as the generalized inverse of g.Then, based on (3) and (7), we have the tracking error e as(8)e˙=fx+gus+ue−x˙d=−Ke+gue.From equations (7) and (8), we know that the tracking control of system (3) can be considered the regulation problem of (8). Hence, an optimal control ue will be designed to stabilize tracking error dynamics (8) in an optimal manner.


### 3.2. Approximate Optimal Tracking Control

The controllerue can be used to make (8) converge in an optimal sense. For this purpose, we can rewrite the infinite horizon cost function (4) as follows:(9)Vet=∫t∞eTtQet+ueTtRuetdt.Then, an admissible control policyue should be found so that cost function (9) of system (8) can be minimized. To this end, the Lyapunov equation of (7) is given by(10)0=VeT−Ke+gue+eTQe+ueTRue,with Ve=∂V/∂e being the partial differential.Then, the optimal cost functionV∗e is given as(11)V∗e=minue∫t∞eTtQet+ue∗TtRue∗tdt,and the derived HJB equation is shown as(12)0=minueHe,ue∗,V∗.The optimal controlue can be derived by solving ∂He,ue∗,V∗/∂ue∗=0 from (10) as(13)ue∗=−12R−1gT∂V∗e∂e.The problem to be finally addressed is to solve HJB equation (12) to obtain the optimal cost function V∗e required in control (13).


#### 3.2.1. Online Reinforcement Learning Algorithm

To calculate the above optimal control, we can recall the policy iteration (PI) method. Inspired by [1, 26], a policy iteration algorithm can be given as follows:(1)
Select a small positive constantκ. Let i=0 and V0=0, then set an initial admissible control policy ue0.(2)
Solve the nonlinear Lyapunov equation using the control policyuei(14)0=eTQe+ueTiRuei+∇Vi+1eT−Ke+guei,withVi+10=0.(3)
Improve the control policy by(15)uei+1=−12R−1gT∇Vi+1.(4)
IfVi+1e−Vie≤κ, stop the iteration and take the approximate optimal control; else, let i=i+1and go back to Step 2.The above PI scheme can guarantee the convergence to the optimal cost function and control action, i.e.,Vi⟶V∗ and uei⟶u∗ as i⟶∞. The convergence proof of the PI algorithm was detailed in [9].


#### 3.2.2. Neural Network Approximation

The above policy iteration method is runoffline. To implement online optimal control, we will introduce an online learning method in this section.From HJB equation (12), it is generally difficult to derive its solution. As shown in [27, 28], we will use a critic NN to estimate the ideal cost function V∗e. In this paper, the cost function can be considered as smooth, then a critic NN [26, 28] is applied to approximate V∗e as(16)V∗e=WTσe+εe,where W∈ℜl is the ideal NN weight, σe∈ℜl is the activation function, l is the number of neurons, and εe is the approximation error. Then, we have its derivative with respect to e as(17)∇Ve=∇σeTW+∇εe,where ∇σe=∂σe/∂e∈ℜl×n and ∇εe=∂εe/∂e∈ℜnare the partial derivatives. Then, based on (17), equation (10) is represented as(18)0=eTQe+ue∗TRue∗+WT∇σe+∇εeTe˙.Assumption 2.
(see [9, 26]). Consider the critic NN weight W with the regressor ∇σe, and then the error εe and its gradient ∇εe are all bounded. Moreover, we have εe⟶0 and ∇εe⟶0 as l⟶∞.
According to ideal cost function (16), the actual cost function can be given as(19)V^e=W^Tσe,which can be used to estimate the practical cost function. For critic NN (19), we can select σe such that V^e>0 for e≠0 and V^e=0 for e=0. Then, we have(20)∇V^e=∇σeTW^,where ∇V^e=∂V^e/∂e.
Then, the approximated Hamiltonian function can be derived as(21)0≜Hx,u,W^=eTQe+ueTRue+W^T∇σee˙=ec.
For training the critic NN to obtain the control action, it is expected to estimateW^ to minimize the objective function E=1/2ecTec. Hence, the gradient descent algorithm can be used to update the critic NN weights W^ by(22)W^.=−Γ∂E∂W^=−Γec∂ec∂W^,where Γ>0 is the learning gain.
Based on (18), the Hamiltonian function is(23)0=eTQe+ue∗TRue∗+WT∇σee˙=ec2,where ec2=−∇εeTe˙ is the residual error.
Defineϑ=∇σee˙ and W˜=W−W^ as the estimation error of the critic NN weights, and a positive constant ϑM with ϑ≤ϑM. Then, from (21) and (23), we have ec2−ec=W˜Tϑ. Hence, the estimation error dynamics are given by(24)W˜.=−W^.=Γec2−W˜Tϑϑ.
The persistent excitation (PE) condition is required to retain the critic NN weights convergence and the conditionϑ≥ϑm with a positive constant ϑm. This condition can be satisfied in this paper since we consider the tracking control problem, and thus, the probing noise used in many existing ADP literature studies may be not necessary.
When implementing the online optimal control algorithm with critic NN (16), we have from (13) and (17) the optimal control as(25)ue∗=−12R−1gT∇σeTW+∇εe.
Then, the approximated control action with critic NN (19) is formulated as(26)ue∗=−12R−1gT∇σeTW^.
Equation (26) implies that with the updated critic NN weights W^, the approximated control action can be calculated directly. Consequently, the widely used actor-critic structure can be simplified, and only the critic NN is adopted in this paper to reduce the computational cost.
Next, the stability of the proposed algorithm is given.Lemma 1.
For error system (8) with control (26) and learning law (22), then the estimation error dynamics (24) are uniformly ultimately bounded (UUB).Proof.
We select the Lyapunov function candidate asJt=trW˜TΓ−1W˜. Then, the time derivative of the Lyapunov function along the trajectory of error dynamics (24) is(27)J˙t=2trW˜TΓ−1W˜.=2trW˜Tec2−W˜Tϑϑ.
After some mathematical manipulations, we have(28)J˙t≤−2−ΓW˜Tϑ2+1Γec22.
Considering the Cauchy–Schwarz inequality and noticing the assumptionϑ≤ϑM, we can conclude that J˙t<0 as long as 0<Γ<2 and(29)W˜>ec22Γ2−ΓϑM2.
According to the Lyapunov theory, we obtain that the estimation error is UUB.


## 3.1. Steady-State Tracking Control

Sinceus can be adopted to ensure that the control error converges to zero in the steady-state, then we have from (3) that(7)us=g+x˙d−fx−Ke,where K>0 is the feedback gain set by the desingers and g+=gTg−1gT is defined as the generalized inverse of g.Then, based on (3) and (7), we have the tracking error e as(8)e˙=fx+gus+ue−x˙d=−Ke+gue.From equations (7) and (8), we know that the tracking control of system (3) can be considered the regulation problem of (8). Hence, an optimal control ue will be designed to stabilize tracking error dynamics (8) in an optimal manner.


## 3.2. Approximate Optimal Tracking Control

The controllerue can be used to make (8) converge in an optimal sense. For this purpose, we can rewrite the infinite horizon cost function (4) as follows:(9)Vet=∫t∞eTtQet+ueTtRuetdt.Then, an admissible control policyue should be found so that cost function (9) of system (8) can be minimized. To this end, the Lyapunov equation of (7) is given by(10)0=VeT−Ke+gue+eTQe+ueTRue,with Ve=∂V/∂e being the partial differential.Then, the optimal cost functionV∗e is given as(11)V∗e=minue∫t∞eTtQet+ue∗TtRue∗tdt,and the derived HJB equation is shown as(12)0=minueHe,ue∗,V∗.The optimal controlue can be derived by solving ∂He,ue∗,V∗/∂ue∗=0 from (10) as(13)ue∗=−12R−1gT∂V∗e∂e.The problem to be finally addressed is to solve HJB equation (12) to obtain the optimal cost function V∗e required in control (13).


### 3.2.1. Online Reinforcement Learning Algorithm

To calculate the above optimal control, we can recall the policy iteration (PI) method. Inspired by [1, 26], a policy iteration algorithm can be given as follows:(1)
Select a small positive constantκ. Let i=0 and V0=0, then set an initial admissible control policy ue0.(2)
Solve the nonlinear Lyapunov equation using the control policyuei(14)0=eTQe+ueTiRuei+∇Vi+1eT−Ke+guei,withVi+10=0.(3)
Improve the control policy by(15)uei+1=−12R−1gT∇Vi+1.(4)
IfVi+1e−Vie≤κ, stop the iteration and take the approximate optimal control; else, let i=i+1and go back to Step 2.The above PI scheme can guarantee the convergence to the optimal cost function and control action, i.e.,Vi⟶V∗ and uei⟶u∗ as i⟶∞. The convergence proof of the PI algorithm was detailed in [9].


### 3.2.2. Neural Network Approximation

The above policy iteration method is runoffline. To implement online optimal control, we will introduce an online learning method in this section.From HJB equation (12), it is generally difficult to derive its solution. As shown in [27, 28], we will use a critic NN to estimate the ideal cost function V∗e. In this paper, the cost function can be considered as smooth, then a critic NN [26, 28] is applied to approximate V∗e as(16)V∗e=WTσe+εe,where W∈ℜl is the ideal NN weight, σe∈ℜl is the activation function, l is the number of neurons, and εe is the approximation error. Then, we have its derivative with respect to e as(17)∇Ve=∇σeTW+∇εe,where ∇σe=∂σe/∂e∈ℜl×n and ∇εe=∂εe/∂e∈ℜnare the partial derivatives. Then, based on (17), equation (10) is represented as(18)0=eTQe+ue∗TRue∗+WT∇σe+∇εeTe˙.Assumption 2.
(see [9, 26]). Consider the critic NN weight W with the regressor ∇σe, and then the error εe and its gradient ∇εe are all bounded. Moreover, we have εe⟶0 and ∇εe⟶0 as l⟶∞.
According to ideal cost function (16), the actual cost function can be given as(19)V^e=W^Tσe,which can be used to estimate the practical cost function. For critic NN (19), we can select σe such that V^e>0 for e≠0 and V^e=0 for e=0. Then, we have(20)∇V^e=∇σeTW^,where ∇V^e=∂V^e/∂e.
Then, the approximated Hamiltonian function can be derived as(21)0≜Hx,u,W^=eTQe+ueTRue+W^T∇σee˙=ec.
For training the critic NN to obtain the control action, it is expected to estimateW^ to minimize the objective function E=1/2ecTec. Hence, the gradient descent algorithm can be used to update the critic NN weights W^ by(22)W^.=−Γ∂E∂W^=−Γec∂ec∂W^,where Γ>0 is the learning gain.
Based on (18), the Hamiltonian function is(23)0=eTQe+ue∗TRue∗+WT∇σee˙=ec2,where ec2=−∇εeTe˙ is the residual error.
Defineϑ=∇σee˙ and W˜=W−W^ as the estimation error of the critic NN weights, and a positive constant ϑM with ϑ≤ϑM. Then, from (21) and (23), we have ec2−ec=W˜Tϑ. Hence, the estimation error dynamics are given by(24)W˜.=−W^.=Γec2−W˜Tϑϑ.
The persistent excitation (PE) condition is required to retain the critic NN weights convergence and the conditionϑ≥ϑm with a positive constant ϑm. This condition can be satisfied in this paper since we consider the tracking control problem, and thus, the probing noise used in many existing ADP literature studies may be not necessary.
When implementing the online optimal control algorithm with critic NN (16), we have from (13) and (17) the optimal control as(25)ue∗=−12R−1gT∇σeTW+∇εe.
Then, the approximated control action with critic NN (19) is formulated as(26)ue∗=−12R−1gT∇σeTW^.
Equation (26) implies that with the updated critic NN weights W^, the approximated control action can be calculated directly. Consequently, the widely used actor-critic structure can be simplified, and only the critic NN is adopted in this paper to reduce the computational cost.
Next, the stability of the proposed algorithm is given.Lemma 1.
For error system (8) with control (26) and learning law (22), then the estimation error dynamics (24) are uniformly ultimately bounded (UUB).Proof.
We select the Lyapunov function candidate asJt=trW˜TΓ−1W˜. Then, the time derivative of the Lyapunov function along the trajectory of error dynamics (24) is(27)J˙t=2trW˜TΓ−1W˜.=2trW˜Tec2−W˜Tϑϑ.
After some mathematical manipulations, we have(28)J˙t≤−2−ΓW˜Tϑ2+1Γec22.
Considering the Cauchy–Schwarz inequality and noticing the assumptionϑ≤ϑM, we can conclude that J˙t<0 as long as 0<Γ<2 and(29)W˜>ec22Γ2−ΓϑM2.
According to the Lyapunov theory, we obtain that the estimation error is UUB.


## 3.2.1. Online Reinforcement Learning Algorithm

To calculate the above optimal control, we can recall the policy iteration (PI) method. Inspired by [1, 26], a policy iteration algorithm can be given as follows:(1)
Select a small positive constantκ. Let i=0 and V0=0, then set an initial admissible control policy ue0.(2)
Solve the nonlinear Lyapunov equation using the control policyuei(14)0=eTQe+ueTiRuei+∇Vi+1eT−Ke+guei,withVi+10=0.(3)
Improve the control policy by(15)uei+1=−12R−1gT∇Vi+1.(4)
IfVi+1e−Vie≤κ, stop the iteration and take the approximate optimal control; else, let i=i+1and go back to Step 2.The above PI scheme can guarantee the convergence to the optimal cost function and control action, i.e.,Vi⟶V∗ and uei⟶u∗ as i⟶∞. The convergence proof of the PI algorithm was detailed in [9].


## 3.2.2. Neural Network Approximation

The above policy iteration method is runoffline. To implement online optimal control, we will introduce an online learning method in this section.From HJB equation (12), it is generally difficult to derive its solution. As shown in [27, 28], we will use a critic NN to estimate the ideal cost function V∗e. In this paper, the cost function can be considered as smooth, then a critic NN [26, 28] is applied to approximate V∗e as(16)V∗e=WTσe+εe,where W∈ℜl is the ideal NN weight, σe∈ℜl is the activation function, l is the number of neurons, and εe is the approximation error. Then, we have its derivative with respect to e as(17)∇Ve=∇σeTW+∇εe,where ∇σe=∂σe/∂e∈ℜl×n and ∇εe=∂εe/∂e∈ℜnare the partial derivatives. Then, based on (17), equation (10) is represented as(18)0=eTQe+ue∗TRue∗+WT∇σe+∇εeTe˙.Assumption 2.
(see [9, 26]). Consider the critic NN weight W with the regressor ∇σe, and then the error εe and its gradient ∇εe are all bounded. Moreover, we have εe⟶0 and ∇εe⟶0 as l⟶∞.
According to ideal cost function (16), the actual cost function can be given as(19)V^e=W^Tσe,which can be used to estimate the practical cost function. For critic NN (19), we can select σe such that V^e>0 for e≠0 and V^e=0 for e=0. Then, we have(20)∇V^e=∇σeTW^,where ∇V^e=∂V^e/∂e.
Then, the approximated Hamiltonian function can be derived as(21)0≜Hx,u,W^=eTQe+ueTRue+W^T∇σee˙=ec.
For training the critic NN to obtain the control action, it is expected to estimateW^ to minimize the objective function E=1/2ecTec. Hence, the gradient descent algorithm can be used to update the critic NN weights W^ by(22)W^.=−Γ∂E∂W^=−Γec∂ec∂W^,where Γ>0 is the learning gain.
Based on (18), the Hamiltonian function is(23)0=eTQe+ue∗TRue∗+WT∇σee˙=ec2,where ec2=−∇εeTe˙ is the residual error.
Defineϑ=∇σee˙ and W˜=W−W^ as the estimation error of the critic NN weights, and a positive constant ϑM with ϑ≤ϑM. Then, from (21) and (23), we have ec2−ec=W˜Tϑ. Hence, the estimation error dynamics are given by(24)W˜.=−W^.=Γec2−W˜Tϑϑ.
The persistent excitation (PE) condition is required to retain the critic NN weights convergence and the conditionϑ≥ϑm with a positive constant ϑm. This condition can be satisfied in this paper since we consider the tracking control problem, and thus, the probing noise used in many existing ADP literature studies may be not necessary.
When implementing the online optimal control algorithm with critic NN (16), we have from (13) and (17) the optimal control as(25)ue∗=−12R−1gT∇σeTW+∇εe.
Then, the approximated control action with critic NN (19) is formulated as(26)ue∗=−12R−1gT∇σeTW^.
Equation (26) implies that with the updated critic NN weights W^, the approximated control action can be calculated directly. Consequently, the widely used actor-critic structure can be simplified, and only the critic NN is adopted in this paper to reduce the computational cost.
Next, the stability of the proposed algorithm is given.Lemma 1.
For error system (8) with control (26) and learning law (22), then the estimation error dynamics (24) are uniformly ultimately bounded (UUB).Proof.
We select the Lyapunov function candidate asJt=trW˜TΓ−1W˜. Then, the time derivative of the Lyapunov function along the trajectory of error dynamics (24) is(27)J˙t=2trW˜TΓ−1W˜.=2trW˜Tec2−W˜Tϑϑ.
After some mathematical manipulations, we have(28)J˙t≤−2−ΓW˜Tϑ2+1Γec22.
Considering the Cauchy–Schwarz inequality and noticing the assumptionϑ≤ϑM, we can conclude that J˙t<0 as long as 0<Γ<2 and(29)W˜>ec22Γ2−ΓϑM2.
According to the Lyapunov theory, we obtain that the estimation error is UUB.


## 4. Simulation

To demonstrate the validity of the developed method, a numerical simulation based on a SCARA robot plant is given. Consider the dynamics of a two degree-of-freedom SCARA robot system as(30)x˙=x˙1x˙2=x2−M−1Cx2−M−1G+M−1u,where x1=q1q2 and x2=q˙1q˙2 are the SCARA robot’s joint position and velocity vectors. The inverse of inertia matrix M−1, Coriolis dynamics C, and gravity dynamics G is shown as(31)M−1=b−b−c−b−ca+b+2cab−c2,C=−2dq˙2−dq˙2dq˙10, andG=00,with a=0.613,b=0.1173,c=0.1584cosq2,andd=0.1584sinq2, and the detail modelling process can be found in [29].Then, system (30) can be formulated as(32)x˙t=fx+gut,where fx=x2−M−1Cx2+GT denote the drift dynamics and g=0M−1T denotes the control gain. To complete the optimal tracking control, equation (7) for system (30) can be given as ud=g+x˙d−fx−Ke with K=0.8. The initial critic NN weights are selected as W^=1252−31−1−75−6T, and the initial system states are x1=−2,6T and x2=−2.4,1T. In the learning process, we set the learning gain as Γ=0.02. The activation function of the critic NN is chosen as σ=e12,e1e2,e1e3,e1e4,e22,e2e3,e2e4,e32,e3e4,e42, and the weighting matrices Q and R are selected as identity matrices as [29]. The desired trajectory xd are given as xd=q1d,q2d,dq1d,dq2d, where q1d=2sint, q2d=2cost, dq1d=q2d, and dq2d=−q1d. During the implementation of the policy iteration algorithm, we take the sinusoidal signals as the reference and thus the persistence excitation condition has been fulfilled. In this case, the probing noise introduced into the system can be avoided.With these parameter configurations, numerical simulations are conducted and simulation results are given in Figures2–5. The online profile of the approximated critic NN weights W^ with proposed adaptive law (22) are displayed in Figure 2, which converge to the vector −0.4847,1.6323,−0.4100,6.0223,−3.8430,−1.3071,1.9629,−2.2160,−3.7896,2.9225Tafter a short transient stage. Clearly, we can find that the weights are convergent when performing the proposed online learning. With these online updated critic NN weights, the estimated control can converge to the ideal solution. As a consequence, the system tracking performances are given in Figures 3 and 4, which indicate that the system states can track the desired trajectories with the proposed optimal control. To better show the motion tracking results, the tracking errors are given in Figure 5, which converge to zero with very smooth profiles. Moreover, the control input is given as Figure 6, which is also bounded. It can be found from above simulations that the proposed optimal control can realize perfect tracking control. Specifically, the proposed RL algorithm can retain convergent response for the critic NN.Figure 2
Convergence of the critic NN weights.Figure 3
Position tracking of joint 1 and joint 2 of SCARA.Figure 4
Velocity tracking of joint 1 and joint 2 of SCARA.Figure 5
Tracking error convergence.Figure 6
Control action (u).


## 5. Conclusion

The optimal tracking control design for robotic systems using the RL algorithm is presented in this paper. The system model is first transformed into a canonical form, which can facilitate the realization of optimal tracking control design. To maintain the tracking response, a steady-state control is designed, and then an optimal tracking control is used to ensure the tracking error dynamics to be optimal. The online RL algorithm is dedicated to solving the HJB equation using a critic NN. Numerical simulations are given to show the effectiveness of the proposed technique. We will study the nonlinear optimal tracking problem with fully unknown system dynamics in the future work.



---

*Source: 8839391-2021-02-09.xml*