# Adaptive Inverse Optimal Control for Rehabilitation Robot Systems Using Actor-Critic Algorithm

Authors: Fancheng Meng; Yaping Dai
Journal: Mathematical Problems in Engineering
 (2014)

---

## Abstract

The higher goal of rehabilitation robot is to aid a person to achieve a desired functional task (e.g., tracking trajectory) based on assisted-as-needed principle. To this goal, a new adaptive inverse optimal hybrid control (AHC) combining inverse optimal control and actor-critic learning is proposed. Specifically, an uncertain nonlinear rehabilitation robot model is firstly developed that includes human motor behavior dynamics. Then, based on this model, an open-loop error system is formed; thereafter, an inverse optimal control input is designed to minimize the cost functional and a NN-based actor-critic feedforward signal is responsible for the nonlinear dynamic part contaminated by uncertainties. Finally, the AHC controller is proven (through a Lyapunov-based stability analysis) to yield a global uniformly ultimately bounded stability result, and  the resulting cost functional is meaningful. Simulation and experiment on rehabilitation robot demonstrate the effectiveness of the proposed control scheme.


---

## Body

## 1. Introduction

Rehabilitation robots are becoming increasingly common in upper extremity rehabilitation [1], because they are not only able to provide intensive rehabilitation consistently for a longer duration, but also able to offer object assessment for patient’s motor performance. Relevant studies show they contribute significantly to motor recovery [2–4]. However, most of the existing rehabilitation robots (RRs) so far have still suffered from a number of challenges. Remarkably, an autonomous rehabilitation robot has yet to be developed due to the lack of an advanced and intelligent controller. To achieve this requirement, various control strategies have been proposed, such as impedance control [5], admittance control [6], and PID control [7]. One common aspect of the majority of these RRs control algorithms applied to robot-aided therapies is to allow a variable deviation from a predefined trajectory rather than imposing a rigid training pattern [8]. Relevant studies demonstrate these RRs controllers are effective to some extent [9, 10]. However, their control parameters in essence are predefined by human. From the practical viewpoint, the fixed and predefined control parameters might often lead to either an overdetermined or an underdetermined control performance. Furthermore, the predefined control pattern does not suit each person. Thus, how to provide controllable, suitable assistance specific to a particular patient is a worth studying problem.One possible solution to this problem is adaptive functional training. The basic idea of the adaptive functional training is to allow a modification of controller parameters or task difficulty levels according to the patient’s motor performance, rather than imposing a rigid predefined training pattern. Typical examples include adaptive admittance control [11], adaptive impedance control [12, 13], adaptive task-oriented therapy [13–15], and adaptive mixed reality rehabilitation training [16, 17], as well as adaptive slide model control [18]. Relevant works show these approaches can adjust their controller parameters or task difficulty levels according to the patients’ motor performance or diagnosis and can provide a better control performance. However, their resulting impedance parameters (e.g., mass, damping, and stiffness) are still obtained in a heuristically way and cannot be easily extended to other applications. Recently, the combinations of neural network- (NN-) based intelligent control and adaptive functional training have been proposed to RRs, such as NN-based adaptive sliding mode control [19], neural-fuzzy robust adaptive control [20], and neural-fuzzy control [21]. Relevant studies show these NN-based intelligent RRs controllers can provide a better performance than the above-mentioned traditional adaptive control. Unfortunately, their results are limited to uniformly ultimately bounded (UUB) stability because of the inevitable NN reconstruction error [22]. Crucially, most of these existing intelligent RRs controllers are in essence adaptive but not optimal for rehabilitation training. However, in practical application, the optimal RRs controller can benefit the patient’s rehabilitation training more than the traditional adaptive RRs controllers due to the trade-off between control torques and tracking accuracy [23].Reinforcement learning (RL) has the potential to overcome the above-mentioned problems [24–28]. Actor-critic- (AC-) based controller has, for example, been used for controlling a three-finger gripper [29]; also, this method can be used to find suitable compliance for different contact tasks [30]. Recently, the AC-based controller design problems have been extensively investigated for functional electrical stimulation (FES) [31] and myoelectric devices [32]. Thomas et al. [31] showed the feasibility of using the AC for FES control of upper extremities. Similarly, Pilarski et al. [32] proposed an AC-based reinforcement learning method for optimizing the control of myoelectric devices. Despite such encouraging results, the AC-based control approach so far found relatively limited application in real-time robotic rehabilitation system, because rehabilitation actions are often continuous in robotic rehabilitation process, whereas most of the existing AC-based controllers concern discrete actions. Unfortunately, the discrete AC controllers cannot ensure effectiveness in continuous-time systems. Moreover, during rehabilitation training process, since the exogenous reference inputs for many online applications are event-driven and not known a priori, it is often impossible to monitor online whether a signal is persistence of excitation (PE) and whereas PE condition is vital for AC learning [33]. In this case, the AC-based controller is not able to ensure convergence in a finite time. Recently, a continuous adaptive critic control [34] solves this problem, where a NN-based critic is used to approximate a long-term cost function and a NN-based actor approximates the optimal control. Simulation results demonstrate the feasibility of the proposed AC-based approach. However, a limitation of the method is that it requires the knowledge of the Hamilton-Jacobi-Bellman (HJB) equation. In practice, solving the HJB equation is quite a difficult and time-consuming task in model-free online way [35].To overcome the above-mentioned difficulties and to develop an AC-based controller for the highly nonlinear RRs, a new adaptive inverse optimal hybrid control (AHC) combining inverse optimal control and AC learning is proposed in this paper, where the inverse optimal controller guarantees the RRs’ optimality by minimizing a meaningful cost function; thus, the effort of solving the HJB equation is totally avoided; the AC-based learning method helps the robot select the meaningful cost function despite being uncertain in RRs. In addition, understanding the human motor behavior is essential to assisted-as-needed rehabilitation training. Therefore, an uncertain nonlinear rehabilitation robot model is firstly developed that includes human motor behavior dynamics. Then, based on this model, an open-loop error system is formed; thereafter, an inverse optimal feedback control input is designed to be responsible for nonlinear static part and minimize the cost functional, and a NN-based AC feedforward control is responsible for the nonlinear dynamic part contaminated by uncertainties. Finally, Lyapunov-based stability analysis proved the stability of the proposed AHC and determined a respective meaningful cost functional. Simulation and experiments on a 2-degree-of-freedom rehabilitation robot indicate the feasibility and effectiveness of the proposed AHC.Based on the previous discussion, we highlight the contributions of this paper as follows.(1)
An uncertain nonlinear rehabilitation robot model is developed that includes individual motor behavior dynamics such that the rehabilitation robot can better capture enough information on the real individual (human) motor behavior.(2)
A data-driven NN is employed to cope with the problem of unknown rehabilitation robotic dynamics such that neither the rehabilitation robot structure nor the physical parameters are required in the control design to be developed.(3)
AHC is adopted such that the robot can achieve some function task (e.g., tracking trajectory) in an optimal way.The rest of the paper is organized as follows. In Section2, a specific rehabilitation robot system under study is described and control objective is discussed. In Section 3, the details of the proposed AHC control are presented, followed by the rigorous analysis. In Section 4, the validity of the proposed method is verified by simulation and experimental studies. Concluding remarks are given in Section 5.


## 2. Problem Statement

### 2.1. System Description

In this paper, we investigate a typical robot-assisted system, which includes a human limb and a rehabilitation robot with an end-effector, as shown in Figure1. Without loss of generality, similar to [36], the following assumption is given.Figure 1
The system under study.Assumption 1.
The handle (C) of the rehabilitation robot is tightly grasped by the patient and there is no relative motion between the patient’s hand and the handle (i.e., end-effector). Based on this assumption, the patient’s hand is deemed as “a part” of the rehabilitation robot.The rehabilitation robot in task space can be modeled as(1)MRx¨+CRx˙+KR(x)+d=fH+u,
where x,x˙,x¨∈Rn are the position, velocity, and acceleration of the end-effector, respectively. MR∈Rn×n is the positive inertia matrix of the robot at the end-effector, CR∈Rn×n denotes the centripetal and Coriolis force at the end-effector, KR∈Rn is stiffness matrices, and d∈Rn is unknown bounded disturbance. fH∈Rn is the user’s force. u∈Rn is the input control vector. In general, in many cases, fH in (1) is often only considered as a measurable load. However, the patient’s motion behavior is typically a time-varying trajectory, which cannot be represented by only force states; thus, this pure load-based control is limited by its poor robustness in practice [37].To solve this problem, it is necessary to integrate the patient’s motor behavior dynamics into system (1). The user’s motor dynamics is defined as [38]
(2)fH*=fH+MHx¨+Fdx˙+KHx+d1,
where MH, Fd, and KH are the mass, damper, and spring matrices of the arm end-effector, respectively, and fH is the user hand force, fH* is the planned force in the patient’s CNS, and d1 is the unknown bounded disturbance (e.g., muscle spasticity). Note that, by using (2), the rehabilitation robot is able to capture the dynamic characteristics of motion behavior.Substituting (2) into (1), the total rehabilitation robotic dynamics can thus be rewritten as
(3)Mx¨+Cx˙+G(x)+f+d=u,
where M, C, and G denote the combined patient-robot positive inertia matrix, centripetal and Coriolis force, and stiffness matrices at the handle, respectively. And they satisfy the following equation:
(4)M=MR+MH,C=CR+Fd,G=KR+KH,f=fH*-d1.Remark 2.
An uncertain nonlinear rehabilitation robot model (3) is developed that includes human motor behavior dynamics (2). The objective of the models developed is not to replicate human behavior but only to capture enough information to compensate the drawbacks inherent to the most prior rehabilitation robotic dynamics [39, 40]. By recognizing the patient’s motor behavior, RRs can adjust its robotic assistance according to the patient’s performance on the task, based on assisted-as-needed principle [39].Without loss of generality, the following properties are given.Property 1.
The inertia matrixM(x) is symmetric, positive definite and satisfies the following inequality:
(5)m_∥ξ∥2≤ξTM(x)ξ≤m¯∥ξ∥2,∀ξ∈Rn,
where m_ is a known positive constant, m¯(x) is a known positive function, and ∥·∥ denotes the standard Euclidean norm.Property 2.
The following skew-symmetric relationship is satisfied:(6)ξT(M(x)-2C(x,x˙))ξ=0,∀ξ∈Rn.Property 3.
The nonlinear disturbance term and its first and second time derivatives are bounded by known constants.Property 4.
The desired trajectoryxd(t) is assumed to be designed such that x˙d(t), x¨d(t) exist and are bounded.Property 5 (see [41]).
A continuous nonlinear functionf(x) can be approximated by NN as follows:
(7)f(x)=WTφ(θ)+ε(θ),
where W∈R represents weight vector and φ(·)∈R denotes the NN activation function with ε(θ) being approximation error satisfying ∥ε(θ)∥≤εm, where εm is the approximation error bound. The NN activation function and the weight vector are upper bounded by a positive constant such that ∥φ(θ)∥≤φM and ∥W∥≤WM. Note that Property 5 is easily obtained, because any C1-function can be expressed as (7).Remark 3.
The above-mentioned properties are the fundamental in this paper, and their validity will be shown later.


### 2.2. Control Object

The object of rehabilitation robot is to enable a patient to achieve some function task (e.g., trajectory tracking) based on assisted-as-needed principle. Towards this goal, the current control objective is to find a control input which enables a patient to track a desired trajectory, denoted byxd(t), despite uncertainties in the dynamic model, while also minimizing a given performance index that includes a penalty on the tracking error and the control effort. To quantify the objective, a position tracking error e1(t)∈R is defined as
(8)e=xd(t)-x(t).Further, to facilitate the subsequent closed-loop error system development and stability analysis, a filtered tracking error is defined as(9)r=e˙+λe,
where λ denotes positive constants, which behaves like the control gain in PID controller.


## 2.1. System Description

In this paper, we investigate a typical robot-assisted system, which includes a human limb and a rehabilitation robot with an end-effector, as shown in Figure1. Without loss of generality, similar to [36], the following assumption is given.Figure 1
The system under study.Assumption 1.
The handle (C) of the rehabilitation robot is tightly grasped by the patient and there is no relative motion between the patient’s hand and the handle (i.e., end-effector). Based on this assumption, the patient’s hand is deemed as “a part” of the rehabilitation robot.The rehabilitation robot in task space can be modeled as(1)MRx¨+CRx˙+KR(x)+d=fH+u,
where x,x˙,x¨∈Rn are the position, velocity, and acceleration of the end-effector, respectively. MR∈Rn×n is the positive inertia matrix of the robot at the end-effector, CR∈Rn×n denotes the centripetal and Coriolis force at the end-effector, KR∈Rn is stiffness matrices, and d∈Rn is unknown bounded disturbance. fH∈Rn is the user’s force. u∈Rn is the input control vector. In general, in many cases, fH in (1) is often only considered as a measurable load. However, the patient’s motion behavior is typically a time-varying trajectory, which cannot be represented by only force states; thus, this pure load-based control is limited by its poor robustness in practice [37].To solve this problem, it is necessary to integrate the patient’s motor behavior dynamics into system (1). The user’s motor dynamics is defined as [38]
(2)fH*=fH+MHx¨+Fdx˙+KHx+d1,
where MH, Fd, and KH are the mass, damper, and spring matrices of the arm end-effector, respectively, and fH is the user hand force, fH* is the planned force in the patient’s CNS, and d1 is the unknown bounded disturbance (e.g., muscle spasticity). Note that, by using (2), the rehabilitation robot is able to capture the dynamic characteristics of motion behavior.Substituting (2) into (1), the total rehabilitation robotic dynamics can thus be rewritten as
(3)Mx¨+Cx˙+G(x)+f+d=u,
where M, C, and G denote the combined patient-robot positive inertia matrix, centripetal and Coriolis force, and stiffness matrices at the handle, respectively. And they satisfy the following equation:
(4)M=MR+MH,C=CR+Fd,G=KR+KH,f=fH*-d1.Remark 2.
An uncertain nonlinear rehabilitation robot model (3) is developed that includes human motor behavior dynamics (2). The objective of the models developed is not to replicate human behavior but only to capture enough information to compensate the drawbacks inherent to the most prior rehabilitation robotic dynamics [39, 40]. By recognizing the patient’s motor behavior, RRs can adjust its robotic assistance according to the patient’s performance on the task, based on assisted-as-needed principle [39].Without loss of generality, the following properties are given.Property 1.
The inertia matrixM(x) is symmetric, positive definite and satisfies the following inequality:
(5)m_∥ξ∥2≤ξTM(x)ξ≤m¯∥ξ∥2,∀ξ∈Rn,
where m_ is a known positive constant, m¯(x) is a known positive function, and ∥·∥ denotes the standard Euclidean norm.Property 2.
The following skew-symmetric relationship is satisfied:(6)ξT(M(x)-2C(x,x˙))ξ=0,∀ξ∈Rn.Property 3.
The nonlinear disturbance term and its first and second time derivatives are bounded by known constants.Property 4.
The desired trajectoryxd(t) is assumed to be designed such that x˙d(t), x¨d(t) exist and are bounded.Property 5 (see [41]).
A continuous nonlinear functionf(x) can be approximated by NN as follows:
(7)f(x)=WTφ(θ)+ε(θ),
where W∈R represents weight vector and φ(·)∈R denotes the NN activation function with ε(θ) being approximation error satisfying ∥ε(θ)∥≤εm, where εm is the approximation error bound. The NN activation function and the weight vector are upper bounded by a positive constant such that ∥φ(θ)∥≤φM and ∥W∥≤WM. Note that Property 5 is easily obtained, because any C1-function can be expressed as (7).Remark 3.
The above-mentioned properties are the fundamental in this paper, and their validity will be shown later.


## 2.2. Control Object

The object of rehabilitation robot is to enable a patient to achieve some function task (e.g., trajectory tracking) based on assisted-as-needed principle. Towards this goal, the current control objective is to find a control input which enables a patient to track a desired trajectory, denoted byxd(t), despite uncertainties in the dynamic model, while also minimizing a given performance index that includes a penalty on the tracking error and the control effort. To quantify the objective, a position tracking error e1(t)∈R is defined as
(8)e=xd(t)-x(t).Further, to facilitate the subsequent closed-loop error system development and stability analysis, a filtered tracking error is defined as(9)r=e˙+λe,
where λ denotes positive constants, which behaves like the control gain in PID controller.


## 3. Adaptive Inverse Optimal Hybrid Control Design

In this section, a new AC-based adaptive inverse optimal hybrid control (i.e., AHC) for rehabilitation robot system was given. First, a state-space model is developed based on the tracking errors in (8) and (9). Then, based on this model, direct feedforward terms (i.e., identifier NN and an action NN) with an inverse optimal PD feedback controller are given, and, subsequently, a critic NN is proposed; finally, Lyapunov-based stability and optimality analysis are performed to show that the AHC can yield an asymptotic optimal tracking.


### 3.1. Actor NN-Based Control

Using the errors in (8) and tracking errors in (9) and the system dynamics (3), an open-loop error for system (1) can be obtained:
(10)Mr˙=Mx¨d+Cx˙+λMe˙+G+f+d-u.To facilitate the subsequent control design, (10) is rewritten as
(11)Mr˙=-Cr+f1+f2+f3-u,
where some auxiliary terms are defined as follows:
(12)f1=f,f2=Mx¨d+Cx˙d+G+d,f3=λMe˙+λe.Note thatf1 and f2 depend on the position and desired trajectory and contain the same system parameter (seen from target impedance); instead, f3 is the error function. Motivation for this segregation is to facilitate the subsequent control design and stability analysis. Nevertheless, since all the terms contain unknown parameters, the controller u given in (10) cannot be implemented. But if the proposed control input can identify and/or cancel these effects, then the optimal control law can be realized. To identify and cancel these effects, two typical NN-based approximators for both f1 and f2 are defined:
(13)f1=WIφ(y)+εI(y),f2=WaTφ(y)+εa(y),
where Wa∈R(n+1)×m and WI∈R(n+1)×m are the ideal weight matrices, φ(·) denotes the activation function of NN, εI(y), εa(y) are bounded reconstruction errors, and y is the input vector to NN, which is defined as
(14)y=(x,x˙,xd,x˙d).The estimates off1 and f2 are designed as
(15)f^1=W^ITφ(y),f^2=W^aTφ(y),
where W^a∈R(n+1)×m, W^I∈R(n+1)×m are the estimates of the ideal weights. Based on the open-loop error system in (10), the control input is defined as
(16)u=-u01+W^ITφ(y)+W^aTφ(y),
where u01 is the control law to be developed, which is later shown to minimize a meaningful cost, and the estimates for the NN weights in (16) are generated online by the following adaptive update law:
(17)W^˙I=ΥIwφ(y)r+ΥIwφ(y)M-1RW^cTφ′(z)V^cT,(18)W^˙a=Υawφ(y)r+Υawφ(y)M-1RW^cTφ′(z)V^cT,
where Υaw∈R(n+1)×(n+1) and ΥIw∈R(n+1)×(n+1) are constant, positive definite and symmetric gain matrices, z, W^cT, V^cT, and R are the input and weight estimates of the critic NN and reinforcement signal, respectively. Note that the weight update law in (17)-(18) consists of a gradient-based term and a reinforcement signal term R; this makes the subsequent critic NN affect the behavior of the action and/or identifier NN.As described earlier, AC learning’s convergence cannot be ensured in a finite time due to the lack of PE. That is, weight parameter convergence is not guaranteed. Thus, to overcome this difficulty and ensure weight parameter convergence in (17)-(18), a virtual control input with the known noise (i.e., PE) [42, 43] is injected to our control design (16), and the virtual control input is defined:
(19)uv=-kvr+ω,
where kv∈R denotes a positive constant adjustable control gain. ω is any given nonzero measurable signal which is exactly known a priori and bounded by ∥ω∥≤ωM.Thus, the control input with a PE input can be defined as(20)u=-u01+W^ITφ(y)+W^aTφ(y)-uv.To facilitate the analysis, an auxiliary function is given:(21)uo=u01+uv.Furthermore, substituting (20) and (21) into (11), the closed-loop error for system (1) can hence be obtained:
(22)Mr˙=-Cr+W~aTφ(y)+W~ITφ(y)+f3+ε+uo,
where the weight mismatch errors are defined as
(23)W~aT=WaT-W^aT,W~IT=WIT-W^IT.For simplicity, (22) is expressed as
(24)Mr˙=SA+SI+uo=S+uo,
where the auxiliary terms SA and SI are defined as
(25)SA=W~aTφ(y)+f3+ε,SI=W~ITφ(y)-Cr.According to Properties1 and 4 and using the mean value theorem, the following upper bound can be developed:
(26)∥SI∥≤δ1+δ2∥X∥,∥SA∥≤δ3+δ4∥X∥,
where X is defined as
(27)X=[e,r]T.SinceSA and SI contain the same parameters, to facilitate the subsequent stability, the term S in (24) can be defined as
(28)∥S∥≤c1+c2∥X∥≤Φ(∥X∥)X,
where c1=δ1+δ3 and c2=δ2+δ4 are known constants. Based on (10) and the subsequent stability analysis, an inverse optimal PD feedback control input is designed as
(29)u01=-K-1r=-(k1+k2)r.Remark 4.
W
^
a
T
φ
(
y
) and W^ITφ(y) in (16) belong to the actor NN in practice. The goal of the actor NN is to generate appropriate control signals and approximate the uncertainties in the system; instead, u01 in (29) is used to stabilize the system. In contrast to typical controllers, for example, NN + PD control [44] and AC control [45], the proposed AHC can not only ensure system stability but also improve control performance in flexible gains. That is, combining the inverse optimal policy with the AC direct optimal algorithm, the rehabilitation robot can successfully achieve the desired trajectory tracking and minimize a cost functional, despite being uncertain and nonlinear in the dynamic system.


### 3.2. Critic NN

In this section, a critic NN is employed to approximate the long-term cost function. Similar to [46, 47], the reinforcement signal R(t)∈R is assumed to be defined as
(30)R=W^cTφ(z)+ρ,
where z=V^cTr, W^c∈R(nc+1)×1, and V^c∈Rm×nc are the estimates of the ideal weight matrices, φ(z)∈Rnc+1 denotes the activation function, r is the input to the critic NN, and ρ is an adaptive term (auxiliary critic signal) with known noise (the noise is used to ensure PE condition), to aid the subsequent stability; its time derivative is written as
(31)ρ˙=W^cTφ′(z)M-1(K-1+kv)z,
where M and K are as defined before. Based on the subsequent stability analysis, the weight update law for the critic NN can be easily obtained:
(32)W^˙c=-Υcwφ(z)R-ΥcwW^c,V^˙c=-ΥcvrRW^cTφ′(z)-ΥcvV^c,
where Υcw and Υcv∈R are positive and known constants. After taking the time derivative of (30)
(33)R˙=(W^cTφT(z))′+ρ˙.To facilitate the subsequent stability analysis, the expression of (33) is rewritten as
(34)R˙=W^˙cTφ(z)+W^cTφ′(z)V^˙cTr+W^cTφ′(z)V^cTr˙+W^cTφ′(z)M-1(K-1+kv)z.Some auxiliary terms are given:(35)NRc=W^cTφ′(z)z,NRk=M-1CR.Using Properties1–5, and the mean value theorem, the following upper bound can be developed:
(36)∥NRc∥≤ς1r,∥NRk∥≤η1R.Remark 5.
The critic network monitors the state of the rehabilitation robot system and affects the behavior of the actor (i.e., action NN and identifier NN). In addition, since the termK-1r in (29) can change the time derivative of R in (34), thus, increasing the term K-1 can speed up convergence of the critic. It is concluded that our AHC has faster convergence than the AC in [45]. Furthermore, provided the critic NN converges correctly and r can be adjusted arbitrarily by the actor NN, the proposed AHC will provide global (universal) stable and optimal control [48].


### 3.3. Stability Analysis

The stability of the proposed AHC given in (20) and (29) can be examined through Theorem 10. First, some assumptions are given.Assumption 6.
The ideal weight values forWI, Wa, Wc, and Vc are assumed to exist and be bounded by known positive values; that is,
(37)∥WI∥≤WImax⁡,∥Wa∥≤Wamax⁡,∥Wc∥≤Wcmax⁡,∥Vc∥≤Wcmax⁡.Assumption 7.
The weight estimate values and mismatch errors for the ideal weightsWI, Wa, Wc, and Vc are assumed to exist and be bounded by known positive values.Assumption 8.
If Property1 is satisfied, there exists a positive, invertible function Ψ such that
(38)12|M˙|≤Ψ(∥X∥).Remark 9.
Using Property1, Assumption 8 is easily obtained. The above-mentioned assumptions are the basis of Theorem 10.Theorem 10.
For the rehabilitation robot system (1)–(3), let the proposed identification scheme in (15) along with the weight update law in (17) be used to identify the human motor behavior dynamics in (2), and let the action-critic controller given in (15) and (30) along with the weight update laws for the action and critic NN given in (18) and (32) and the inverse optimal control u01 in (29), respectively, ensure that all system signals are bounded under closed-loop operation and that the position tracking error is regulated in the sense that
(39)|e(t)|≤ζ0exp⁡(-ζ1(t))+ζ2,
where ζ0,ζ1,ζ2∈R denote positive constants in Ω≜{Y⊂R2|∥Y∥≤Φ-1(2τ1k1)}. Provided the control gains k1,k2 introduced in (29) and λ in (9) are selected according to the following sufficient condition:
(40)k1>c1,λ>12,min⁡(k2-12,λ-12)>c2kv, Ψ, η, ς∈R are chosen to satisfy the following sufficient condition:
(41)kv>Ψ>0,ς>0,η>0.Proof.
Consider the following positive definite Lyapunov function defined as(42)L(t)=12e2+12Mr2+12R2+P1+P2+Q,
where P1, P2, and Q are defined as, respectively,
(43)P1=12tr⁡{W~ITΥIw-1W~I},P2=12tr⁡{W~aTΥaw-1W~a},Q=12tr⁡{W^cTW^c}+12tr⁡{V^cTV^c}.
Using Property1 and Assumptions 6–8, L(t) can be bounded:
(44)γ1∥Y∥2≤L≤γ2∥Y∥2+γ3,
where γ1,γ2,γ3∈R are known positive constants and Y(t)∈R is defined as
(45)Y≜[e,r,R]T.
Taking the time derivative of (42) yields
(46)L˙=ee˙+12M˙r2+rMr˙+RR˙+P1˙+P2˙+Q˙.
Based on the previous analysis, using (17)-(18), (32), and (29), the expression of (46) can be bounded:
(47)L˙≤(12-λ)e2+(12M˙+12)r2-Υcw∥W^c∥2-Υcw|R|2∥φ(z)∥2+2|R|Υcw∥W^cφ(z)∥-Υcv∥W^cTφ′(z)∥2∥r∥2|R|2-rTCr+rε+rd+rω+2Υcv∥W^cTφ′(z)∥∥r∥|R|∥V^c∥-Υcv∥V^c∥2-rTK-1r-rTkvr-RW^cTφ′(z)M-1Cz.
Using (24)-(25) and (8)-(9) and canceling similar terms, the expression in (47) can hence be bounded as
(48)L˙≤(12-λ)e2+(12M˙+12)r2+rS+ru-NR,
where the auxiliary term NR is defined:
(49)NR=W^cTφ′(z)M-1CzR.
Using inequality (36) and Young’s inequality, the auxiliary function NR can be rewritten as
(50)∥NR∥≤ςr2+ηR2,
where ς=ς12, η=η12. Furthermore, using (25)-(26), (8)-(9), (19), and (38) and canceling similar terms, the expression in (48) can hence be bounded as
(51)L˙≤-(λ-12)e2+(k2-12)r2-Ψ(∥X∥)r2-NR-(k1r2-δ1|r|-δ3|r|)+(δ2+δ4)∥X∥2+kvr2.
Based on the above-mentioned analysis and using (26)–(28), the expression in (51) can be rewritten as
(52)L˙≤-(λ-12)e2+(k2-12)r2-(k1r-c12k1)2+c124k1-ηR2-ςr2+c2∥X∥2-(kv-Ψ)r2.
Provided the conditions given in inequality (41) are satisfied, the expression of (52) is hence simplified as
(53)L˙≤-[min⁡(λ-12,k2-12)-c2]∥X∥2-ηR2+c124k1.
And the following equation is assumed to be satisfied:(54)τ1=min⁡[(min⁡(λ-12,k2-12)-c2),η].
Thus, the expression of (53) can be rewritten as
(55)L˙≤-min⁡[(min⁡(λ-12,k2-12)-c2),η]·∥Y∥2+c124k1=-τ1·∥Y∥2+c124k1.
To facilitate the subsequent analysis, using inequality (44), the expression of (55) is further simplified as
(56)L˙≤-τ1γ2L+ε,
where ε∈R is a positive constant, which is defined as
(57)ε=c124k1+τ1γ2γ3.
Furthermore, using inequalities (44) and (56) and assuming that ττ1=γ2 is satisfied, thus, the above-mentioned inequality (56) can be rewritten as
(58)L(t)≤L(0)e-t/τ+ετ[1-e-t/τ],∀∥Y∥∈Ω,
where Ω is defined as before. Provided the sufficient conditions given in (40)-(41) are satisfied, the expressions in (42) and (58) can be used to prove the control input and all the closed-loop signals are bounded in Ω:
(59)Ω≜{Y⊂R2∣∥Y∥≤Φ-1(2τ1k1)}.
According to (56)-(57) and (59), it is observed that larger value of k1 will expand the size of the domain Ω and reduce the residual error. According to inequality (58), the result in (39) can be easily obtained.


### 3.4. Optimality Analysis

Motivated by the works in [49], the ability of the controller u01 to minimize a meaningful cost can be examined through the following theorem.Theorem 11.
The feedback law is given by(60)u01*=ΗK-1r.
With the scalar gain constant selected asΗ>2 and the adaptive weight update law given in (17)-(18), (32), minimizes the meaningful cost functional
(61)J=lim⁡t→∞{c124k12ΗL(t)(t)+∫0t(M+u01TΞu01)-2Ηc124k1dσ},
where a positive function M(Y,t) is defined as
(62)M=-2Η(ee˙+12M˙r2+rMr˙+Θ-c124k1)+Η2rTK-1r,
where Θ=P1˙+P2˙+Q˙+RR˙.Proof.
The cost functionJ in (61) is considered to be meaningful if it is a positive control function. That is, J in (61) is a positive function if M>0. To examine the sign of M(Y,t), utilizing (46) and inequality (55), the following inequality is hence obtained:
(63)ee˙+12M˙r2+rMr˙+Θ-c124k1≤-τ1∥Y∥2.
Multiplying both sides by-2Η and then adding Η(Η-2)rTK-1r, inequality (63) can be rewritten as
(64)-2Η(ee˙+12M˙r2+rMr˙+Θ-c124k1)+Η(Η-2)·rTK-1r≥2Ητ1∥Y∥2+Η(Η-2)rTK-1r.
Based on inequality (63), inequality (64) can be simplified as
(65)M≥2Ητ1∥Y∥2+Η(Η-2)rTK-1r.
According to inequality (65), if Η≥2, then M≥0 is satisfied. Therefore J is a meaningful cost.
To show thatu01* can minimize J(·), an auxiliary signal μ is defined as
(66)μ=u01+ΗK-1r.
Substitutingμ in (66) into (61)
(67)J=lim⁡t→∞∫0t(M+(μ-ΗK-1r)TΞ(μ-ΗK-1r))-2Ηc124k1dσ+2ΗL(t).
SubstitutingM in (62) into (67) and then utilizing L˙ in (46) and lim⁡t→∞r=0, (67) can be rewritten as
(68)J=lim⁡t→∞[2ΗL(t)-2Η∫0tL˙(t)dσ+∫0tμTΞμdσ]=lim⁡t→∞[2ΗL(t)-2ΗL(t)|0t+∫0tμTΞμdσ]=lim⁡t→∞[2ΗL(0)+∫0tμTΞμdσ].
According to the result in (68), μ can minimize J(·) and stabilize the system. Therefore μ is optimal control law. Moreover, if the action NN and the identifier NN can precisely approximate the uncertainties and nonlinearities in system (1) or their approximation performance satisfies the proposed condition by [50], then a global optimal control law for rehabilitation robot system is obtained.Remark 12.
Differing from direct optimal control [33–35], the proposed AHC is hybrid (indirect and direct) optimal control without having to solve an HJB equation but rather minimizing a meaningful cost. Thus computation complex is alleviated.


## 3.1. Actor NN-Based Control

Using the errors in (8) and tracking errors in (9) and the system dynamics (3), an open-loop error for system (1) can be obtained:
(10)Mr˙=Mx¨d+Cx˙+λMe˙+G+f+d-u.To facilitate the subsequent control design, (10) is rewritten as
(11)Mr˙=-Cr+f1+f2+f3-u,
where some auxiliary terms are defined as follows:
(12)f1=f,f2=Mx¨d+Cx˙d+G+d,f3=λMe˙+λe.Note thatf1 and f2 depend on the position and desired trajectory and contain the same system parameter (seen from target impedance); instead, f3 is the error function. Motivation for this segregation is to facilitate the subsequent control design and stability analysis. Nevertheless, since all the terms contain unknown parameters, the controller u given in (10) cannot be implemented. But if the proposed control input can identify and/or cancel these effects, then the optimal control law can be realized. To identify and cancel these effects, two typical NN-based approximators for both f1 and f2 are defined:
(13)f1=WIφ(y)+εI(y),f2=WaTφ(y)+εa(y),
where Wa∈R(n+1)×m and WI∈R(n+1)×m are the ideal weight matrices, φ(·) denotes the activation function of NN, εI(y), εa(y) are bounded reconstruction errors, and y is the input vector to NN, which is defined as
(14)y=(x,x˙,xd,x˙d).The estimates off1 and f2 are designed as
(15)f^1=W^ITφ(y),f^2=W^aTφ(y),
where W^a∈R(n+1)×m, W^I∈R(n+1)×m are the estimates of the ideal weights. Based on the open-loop error system in (10), the control input is defined as
(16)u=-u01+W^ITφ(y)+W^aTφ(y),
where u01 is the control law to be developed, which is later shown to minimize a meaningful cost, and the estimates for the NN weights in (16) are generated online by the following adaptive update law:
(17)W^˙I=ΥIwφ(y)r+ΥIwφ(y)M-1RW^cTφ′(z)V^cT,(18)W^˙a=Υawφ(y)r+Υawφ(y)M-1RW^cTφ′(z)V^cT,
where Υaw∈R(n+1)×(n+1) and ΥIw∈R(n+1)×(n+1) are constant, positive definite and symmetric gain matrices, z, W^cT, V^cT, and R are the input and weight estimates of the critic NN and reinforcement signal, respectively. Note that the weight update law in (17)-(18) consists of a gradient-based term and a reinforcement signal term R; this makes the subsequent critic NN affect the behavior of the action and/or identifier NN.As described earlier, AC learning’s convergence cannot be ensured in a finite time due to the lack of PE. That is, weight parameter convergence is not guaranteed. Thus, to overcome this difficulty and ensure weight parameter convergence in (17)-(18), a virtual control input with the known noise (i.e., PE) [42, 43] is injected to our control design (16), and the virtual control input is defined:
(19)uv=-kvr+ω,
where kv∈R denotes a positive constant adjustable control gain. ω is any given nonzero measurable signal which is exactly known a priori and bounded by ∥ω∥≤ωM.Thus, the control input with a PE input can be defined as(20)u=-u01+W^ITφ(y)+W^aTφ(y)-uv.To facilitate the analysis, an auxiliary function is given:(21)uo=u01+uv.Furthermore, substituting (20) and (21) into (11), the closed-loop error for system (1) can hence be obtained:
(22)Mr˙=-Cr+W~aTφ(y)+W~ITφ(y)+f3+ε+uo,
where the weight mismatch errors are defined as
(23)W~aT=WaT-W^aT,W~IT=WIT-W^IT.For simplicity, (22) is expressed as
(24)Mr˙=SA+SI+uo=S+uo,
where the auxiliary terms SA and SI are defined as
(25)SA=W~aTφ(y)+f3+ε,SI=W~ITφ(y)-Cr.According to Properties1 and 4 and using the mean value theorem, the following upper bound can be developed:
(26)∥SI∥≤δ1+δ2∥X∥,∥SA∥≤δ3+δ4∥X∥,
where X is defined as
(27)X=[e,r]T.SinceSA and SI contain the same parameters, to facilitate the subsequent stability, the term S in (24) can be defined as
(28)∥S∥≤c1+c2∥X∥≤Φ(∥X∥)X,
where c1=δ1+δ3 and c2=δ2+δ4 are known constants. Based on (10) and the subsequent stability analysis, an inverse optimal PD feedback control input is designed as
(29)u01=-K-1r=-(k1+k2)r.Remark 4.
W
^
a
T
φ
(
y
) and W^ITφ(y) in (16) belong to the actor NN in practice. The goal of the actor NN is to generate appropriate control signals and approximate the uncertainties in the system; instead, u01 in (29) is used to stabilize the system. In contrast to typical controllers, for example, NN + PD control [44] and AC control [45], the proposed AHC can not only ensure system stability but also improve control performance in flexible gains. That is, combining the inverse optimal policy with the AC direct optimal algorithm, the rehabilitation robot can successfully achieve the desired trajectory tracking and minimize a cost functional, despite being uncertain and nonlinear in the dynamic system.


## 3.2. Critic NN

In this section, a critic NN is employed to approximate the long-term cost function. Similar to [46, 47], the reinforcement signal R(t)∈R is assumed to be defined as
(30)R=W^cTφ(z)+ρ,
where z=V^cTr, W^c∈R(nc+1)×1, and V^c∈Rm×nc are the estimates of the ideal weight matrices, φ(z)∈Rnc+1 denotes the activation function, r is the input to the critic NN, and ρ is an adaptive term (auxiliary critic signal) with known noise (the noise is used to ensure PE condition), to aid the subsequent stability; its time derivative is written as
(31)ρ˙=W^cTφ′(z)M-1(K-1+kv)z,
where M and K are as defined before. Based on the subsequent stability analysis, the weight update law for the critic NN can be easily obtained:
(32)W^˙c=-Υcwφ(z)R-ΥcwW^c,V^˙c=-ΥcvrRW^cTφ′(z)-ΥcvV^c,
where Υcw and Υcv∈R are positive and known constants. After taking the time derivative of (30)
(33)R˙=(W^cTφT(z))′+ρ˙.To facilitate the subsequent stability analysis, the expression of (33) is rewritten as
(34)R˙=W^˙cTφ(z)+W^cTφ′(z)V^˙cTr+W^cTφ′(z)V^cTr˙+W^cTφ′(z)M-1(K-1+kv)z.Some auxiliary terms are given:(35)NRc=W^cTφ′(z)z,NRk=M-1CR.Using Properties1–5, and the mean value theorem, the following upper bound can be developed:
(36)∥NRc∥≤ς1r,∥NRk∥≤η1R.Remark 5.
The critic network monitors the state of the rehabilitation robot system and affects the behavior of the actor (i.e., action NN and identifier NN). In addition, since the termK-1r in (29) can change the time derivative of R in (34), thus, increasing the term K-1 can speed up convergence of the critic. It is concluded that our AHC has faster convergence than the AC in [45]. Furthermore, provided the critic NN converges correctly and r can be adjusted arbitrarily by the actor NN, the proposed AHC will provide global (universal) stable and optimal control [48].


## 3.3. Stability Analysis

The stability of the proposed AHC given in (20) and (29) can be examined through Theorem 10. First, some assumptions are given.Assumption 6.
The ideal weight values forWI, Wa, Wc, and Vc are assumed to exist and be bounded by known positive values; that is,
(37)∥WI∥≤WImax⁡,∥Wa∥≤Wamax⁡,∥Wc∥≤Wcmax⁡,∥Vc∥≤Wcmax⁡.Assumption 7.
The weight estimate values and mismatch errors for the ideal weightsWI, Wa, Wc, and Vc are assumed to exist and be bounded by known positive values.Assumption 8.
If Property1 is satisfied, there exists a positive, invertible function Ψ such that
(38)12|M˙|≤Ψ(∥X∥).Remark 9.
Using Property1, Assumption 8 is easily obtained. The above-mentioned assumptions are the basis of Theorem 10.Theorem 10.
For the rehabilitation robot system (1)–(3), let the proposed identification scheme in (15) along with the weight update law in (17) be used to identify the human motor behavior dynamics in (2), and let the action-critic controller given in (15) and (30) along with the weight update laws for the action and critic NN given in (18) and (32) and the inverse optimal control u01 in (29), respectively, ensure that all system signals are bounded under closed-loop operation and that the position tracking error is regulated in the sense that
(39)|e(t)|≤ζ0exp⁡(-ζ1(t))+ζ2,
where ζ0,ζ1,ζ2∈R denote positive constants in Ω≜{Y⊂R2|∥Y∥≤Φ-1(2τ1k1)}. Provided the control gains k1,k2 introduced in (29) and λ in (9) are selected according to the following sufficient condition:
(40)k1>c1,λ>12,min⁡(k2-12,λ-12)>c2kv, Ψ, η, ς∈R are chosen to satisfy the following sufficient condition:
(41)kv>Ψ>0,ς>0,η>0.Proof.
Consider the following positive definite Lyapunov function defined as(42)L(t)=12e2+12Mr2+12R2+P1+P2+Q,
where P1, P2, and Q are defined as, respectively,
(43)P1=12tr⁡{W~ITΥIw-1W~I},P2=12tr⁡{W~aTΥaw-1W~a},Q=12tr⁡{W^cTW^c}+12tr⁡{V^cTV^c}.
Using Property1 and Assumptions 6–8, L(t) can be bounded:
(44)γ1∥Y∥2≤L≤γ2∥Y∥2+γ3,
where γ1,γ2,γ3∈R are known positive constants and Y(t)∈R is defined as
(45)Y≜[e,r,R]T.
Taking the time derivative of (42) yields
(46)L˙=ee˙+12M˙r2+rMr˙+RR˙+P1˙+P2˙+Q˙.
Based on the previous analysis, using (17)-(18), (32), and (29), the expression of (46) can be bounded:
(47)L˙≤(12-λ)e2+(12M˙+12)r2-Υcw∥W^c∥2-Υcw|R|2∥φ(z)∥2+2|R|Υcw∥W^cφ(z)∥-Υcv∥W^cTφ′(z)∥2∥r∥2|R|2-rTCr+rε+rd+rω+2Υcv∥W^cTφ′(z)∥∥r∥|R|∥V^c∥-Υcv∥V^c∥2-rTK-1r-rTkvr-RW^cTφ′(z)M-1Cz.
Using (24)-(25) and (8)-(9) and canceling similar terms, the expression in (47) can hence be bounded as
(48)L˙≤(12-λ)e2+(12M˙+12)r2+rS+ru-NR,
where the auxiliary term NR is defined:
(49)NR=W^cTφ′(z)M-1CzR.
Using inequality (36) and Young’s inequality, the auxiliary function NR can be rewritten as
(50)∥NR∥≤ςr2+ηR2,
where ς=ς12, η=η12. Furthermore, using (25)-(26), (8)-(9), (19), and (38) and canceling similar terms, the expression in (48) can hence be bounded as
(51)L˙≤-(λ-12)e2+(k2-12)r2-Ψ(∥X∥)r2-NR-(k1r2-δ1|r|-δ3|r|)+(δ2+δ4)∥X∥2+kvr2.
Based on the above-mentioned analysis and using (26)–(28), the expression in (51) can be rewritten as
(52)L˙≤-(λ-12)e2+(k2-12)r2-(k1r-c12k1)2+c124k1-ηR2-ςr2+c2∥X∥2-(kv-Ψ)r2.
Provided the conditions given in inequality (41) are satisfied, the expression of (52) is hence simplified as
(53)L˙≤-[min⁡(λ-12,k2-12)-c2]∥X∥2-ηR2+c124k1.
And the following equation is assumed to be satisfied:(54)τ1=min⁡[(min⁡(λ-12,k2-12)-c2),η].
Thus, the expression of (53) can be rewritten as
(55)L˙≤-min⁡[(min⁡(λ-12,k2-12)-c2),η]·∥Y∥2+c124k1=-τ1·∥Y∥2+c124k1.
To facilitate the subsequent analysis, using inequality (44), the expression of (55) is further simplified as
(56)L˙≤-τ1γ2L+ε,
where ε∈R is a positive constant, which is defined as
(57)ε=c124k1+τ1γ2γ3.
Furthermore, using inequalities (44) and (56) and assuming that ττ1=γ2 is satisfied, thus, the above-mentioned inequality (56) can be rewritten as
(58)L(t)≤L(0)e-t/τ+ετ[1-e-t/τ],∀∥Y∥∈Ω,
where Ω is defined as before. Provided the sufficient conditions given in (40)-(41) are satisfied, the expressions in (42) and (58) can be used to prove the control input and all the closed-loop signals are bounded in Ω:
(59)Ω≜{Y⊂R2∣∥Y∥≤Φ-1(2τ1k1)}.
According to (56)-(57) and (59), it is observed that larger value of k1 will expand the size of the domain Ω and reduce the residual error. According to inequality (58), the result in (39) can be easily obtained.


## 3.4. Optimality Analysis

Motivated by the works in [49], the ability of the controller u01 to minimize a meaningful cost can be examined through the following theorem.Theorem 11.
The feedback law is given by(60)u01*=ΗK-1r.
With the scalar gain constant selected asΗ>2 and the adaptive weight update law given in (17)-(18), (32), minimizes the meaningful cost functional
(61)J=lim⁡t→∞{c124k12ΗL(t)(t)+∫0t(M+u01TΞu01)-2Ηc124k1dσ},
where a positive function M(Y,t) is defined as
(62)M=-2Η(ee˙+12M˙r2+rMr˙+Θ-c124k1)+Η2rTK-1r,
where Θ=P1˙+P2˙+Q˙+RR˙.Proof.
The cost functionJ in (61) is considered to be meaningful if it is a positive control function. That is, J in (61) is a positive function if M>0. To examine the sign of M(Y,t), utilizing (46) and inequality (55), the following inequality is hence obtained:
(63)ee˙+12M˙r2+rMr˙+Θ-c124k1≤-τ1∥Y∥2.
Multiplying both sides by-2Η and then adding Η(Η-2)rTK-1r, inequality (63) can be rewritten as
(64)-2Η(ee˙+12M˙r2+rMr˙+Θ-c124k1)+Η(Η-2)·rTK-1r≥2Ητ1∥Y∥2+Η(Η-2)rTK-1r.
Based on inequality (63), inequality (64) can be simplified as
(65)M≥2Ητ1∥Y∥2+Η(Η-2)rTK-1r.
According to inequality (65), if Η≥2, then M≥0 is satisfied. Therefore J is a meaningful cost.
To show thatu01* can minimize J(·), an auxiliary signal μ is defined as
(66)μ=u01+ΗK-1r.
Substitutingμ in (66) into (61)
(67)J=lim⁡t→∞∫0t(M+(μ-ΗK-1r)TΞ(μ-ΗK-1r))-2Ηc124k1dσ+2ΗL(t).
SubstitutingM in (62) into (67) and then utilizing L˙ in (46) and lim⁡t→∞r=0, (67) can be rewritten as
(68)J=lim⁡t→∞[2ΗL(t)-2Η∫0tL˙(t)dσ+∫0tμTΞμdσ]=lim⁡t→∞[2ΗL(t)-2ΗL(t)|0t+∫0tμTΞμdσ]=lim⁡t→∞[2ΗL(0)+∫0tμTΞμdσ].
According to the result in (68), μ can minimize J(·) and stabilize the system. Therefore μ is optimal control law. Moreover, if the action NN and the identifier NN can precisely approximate the uncertainties and nonlinearities in system (1) or their approximation performance satisfies the proposed condition by [50], then a global optimal control law for rehabilitation robot system is obtained.Remark 12.
Differing from direct optimal control [33–35], the proposed AHC is hybrid (indirect and direct) optimal control without having to solve an HJB equation but rather minimizing a meaningful cost. Thus computation complex is alleviated.


## 4. Simulations and Experiments

In this section, the proposed AHC method is examined through simulation and experiments. The objective of the simulation is to examine the performance of the controller in (20); instead, the objective of the experiment is to test the controller’s validity. Next we present the simulation and experimental procedure, relevant simulation, and experimental results with some volunteers.


### 4.1. Simulation

To illustrate the effectiveness and robustness of the proposed control algorithm on robotic rehabilitation system (3), a comparison among the adaptive proportional-derivative (PD) control, the NN-based adaptive control (NN + PD), the AC-based adaptive robust control (NN + AC), and the proposed AC-based inverse optimal control (NN + AHC) is made. Relevant simulation results are shown in Figures 3–6; summary results between NN + AC and NN + AHC are shown in Table 1.Table 1
Summarized simulation results.
Simulation
Position error (degree)
Torque (Nm)
AC
AHC
AC
AHC
Maximum (TS)
9.163
7.491
1.163
0.531
Minimum (TS)
6.302
2.872
0.612
0.512
Mean (TS)
7.417
3.431
0.832
0.516
Adjusting time
7–11 s
3–7 s
7–11 s
3–7 s
Maximum (SS)
0.182
0.172
0.523
0.512
Minimum (SS)
0.076
0.057
0.506
0.500
Note: TS represents temporary state; SS denotes steady state.Presume the desired trajectory is selected as (in rads)(69)xd={sgn⁡(sin(0.05πt-0.5π));Δ>0,0;Δ≤0,
where Δ=sgn⁡(sin(0.05πt-0.5π)), which determines the desired trajectory.First, the adaptive PD control is applied to the simulation experiment, and the control parameters of the PD are taken asKP = 20, KD = 30, and Λ = 5I, and its simulation results are shown in Figure 3. From Figure 3, it is obvious that the adaptive PD control cannot track the desired position with the predefined control parameters. Specifically, the maximum position tracking error is about 0.8 rads; the minimum position tracking error is about 0.2 rads, and the steady mean error is about 0.6 rads.Then, an action NN is added to PD arriving at a NN-based adaptive control (NN + PD), which is applied to the simulation experiment, and its control parameters are taken asKP = 10, KD = 20, with λ = 3 (learning rate). The simulation results of the NN-based adaptive control are shown in Figure 4. Though the NN-based adaptive control can track the desired trajectory, its control performance still needs to be improved. Specifically, the maximum position tracking error is about 0.2 rads, the minimum position tracking error is less than 0.1 rads, and the steady mean error is about 0.13 rads.Thereafter, a critic NN is added to NN + PD arriving at an AC control (NN + AC), which is applied to the simulation, and its control parameters are selected as [45]. The simulation results of the AC-based robust adaptive control are shown in Figure 5. Though a favorable tracking performance can be achieved due to the effectiveness of the critic NN, the unwanted transients (e.g., 10 s, 30 s,…, 110 s, etc.) can result in unsatisfactory behavior.Finally, the AC-based inverse optimal control (NN + AHC) is applied to the simulation. At the same time, to furthermore illustrate the robustness of the proposed controller, external disturbanced=0.05sin(3πt)cos⁡(πt) is considered. The initial position is (0,0). Both critic NN and actor NN contain 16 nodes in hidden layer. For weight updating, the weights V and W are set to initial random [0,1]. All the activation functions (sigmoid) for hidden neurons are selected as
(70)φ(z)=11+exp⁡(-z).The gains for inverse optimal controller was selected as(71)k1=2.1,k2=1.8,kv=1.3,H=3,λ=0.6.The control gains for the critic NN and actor NN are selected as(72)ΥIw-1=Υaw-1=ΥAw-1=0.1I,ΥCv-1=ΥCw-1=1I.The choice of these parameters is through some trails and all the gains are chosen in consideration of the requirement of stability. The results of the AC-based inverse optimal control are shown in Figure6. A favorable high tracking performance can be achieved; the unwanted transients are almost avoided with little control torque. Specifically, the steady position tracking error of the proposed AHC decreases less than 0.001 rads in 43 s. Figure 2 shows that the meaningful cost function is positive, and thus the controller u01 in (29) can minimize a meaningful cost, which means the proposed AHC is optimal.(a) The meaningful cost functional. (b) The integral of the meaningful cost functional.
(a)
(b)Position tracking, tracking error, control torque, and weights of actor NN with PD.
(a)
(b)
(c)
(d)Position tracking, tracking error, control torque, and weights of actor NN with NN + PD.
(a)
(b)
(c)
(d)Figure 5
Position tracking, tracking error, control torque, weights of actor NN, and weights of critic NN with NN + AC.Figure 6
Position tracking, tracking error, control torque, weights of actor NN, and weights of critic NN with the proposed NN + AHC.Figures3–6 show the simulation results of traditional control and our proposed controller. From Figures 3–6, NN + PD, NN + AC, and the proposed controller can make the tracking error decrease during the learning process since these controllers have the learning ability. However, the proposed controller has a faster reduction rate in tracking errors than NN + PD, NN + AC with slightly reduced control torque. The control torque of the proposed controller is smoother and has a smaller vibration than NN + AC to achieve the requested level of performance. It means that NN + AC controller augmented by an inverse optimal can adjust the performance versus control input by minimizing a cost functional. It is concluded that the approximation ability of the proposed NN + AHC is better than the NN + AC control system, and the proposed AHC is optimal.Table1 indicates that the position tracking error with NN + AHC is approximately 9% smaller than the one with NN + AC, and resulting force is approximately smaller by 15%, and convergence speed increases by 36%. The control accuracy from the simulation is sufficient for typical functional tasks. Nonetheless, the performance of the proposed method may vary when implemented in the real rehabilitation robot, since the simulation condition is too ideal compared to the real application.


### 4.2. Tracking Experiment

Tracking experiments with two controllers (i.e., NN + AC and NN + AHC) were conducted on both arms of five volunteers (two females and three males, ages 22 to 36 yrs.). Designated as A, B, C, D, and E, subject A was used as an example and Figure7 is the experimental setup.Figure 7
A subject sitting on a chair with back support, holding the handle of the four-link robot manipulator. The interaction force generated by the subject onto the handle and the robot. The actual and reference handle positions were displayed on a monitor in front of the subject.To achieve better the control objective, each volunteer (e.g., A) first undergoes a standard assessment session before the experiment, in which he/she performs no less than 30 seconds reaching movements in eight specific movement directions only by himself/herself [51]. Then, according to the evaluation result, we make a suitable training task, as shown in Figure 8. Thereafter, after having a rest, he/she (e.g., A) is asked to track the desired trajectory (i.e., the green line and the red circle); in the first phase of the experiment (0–50 s), subject A draws circle and in the second phase of the experiment (50–100 s), subject A draws line, according to the instruction on a monitor as accurately as possible. However, since individual motor behavior varies, subject A may not be able to track the desired motion trajectory all along. Once the user’s motor behavior varies, the robot will adjust automatically the assistance modes (i.e., assistance or resistance) using the proposed controller. The experiment results of subject A are shown in Figures 8, 9, 10, and 11, and the whole experimental results of five subjects tested for the desired trajectories are summarized in Table 2.Table 2
Summarized experimental results of five volunteers.
Sub.
Maximum
RMS error
NN + AC
NN + AHC
NN + AC
NN + AHC
PEr
F
PEr
F
PEr
F
PEr
F
Units
cm
N
cm
N
cm
N
cm
N
A (L)
2.12
9.96
2.02
9.33
0.67
0.26
0.66
0.26
A (R)
2.16
10.15
2.10
9.82
0.68
0.27
0.65
0.27
B (L)
1.92
7.08
1.71
6.36
0.75
0.28
0.56
0.19
B (R)
1.96
7.15
1.73
6.53
0.77
0.28
0.48
0.16
C (L)
2.33
9.13
2.23
8.70
0.86
0.34
0.73
0.27
C (R)
2.31
9.03
2.27
8.64
0.78
0.31
0.64
0.25
D (L)
1.78
4.64
1.73
4.53
0.42
0.11
0.56
0.16
D (R)
1.76
4.59
1.72
4.57
0.39
0.10
0.41
0.11
E (L)
2.02
5.83
2.02
5.62
0.62
0.23
0.56
0.17
E (R)
2.03
5.86
1.96
5.65
0.58
0.21
0.58
0.18
Mean
2.04
7.34
1.95
6.98
0.65
0.24
0.58
0.20
S.D.
0.20
2.34
0.16
1.98
0.12
0.07
0.09
0.05
Note: A-B denotes five volunteers, L represents left arm, R represents right arm, PEr represents position tracking error, andF represents the result force; S.D. denotes statistically significant value.Figure 8
Trajectory tracking with NN + AC.Figure 9
Trajectory tracking with NN + AHC.(a) Resultant force with NN + AC. (b) Velocity of resultant force with NN + AC.
(a)
(b)(a) Resultant force with NN + AHC. (b) Velocity of resultant force with NN + AHC.
(a)
(b)Figures8–11 show the experiment results of NN + AC and our proposed controller. From Figures 8–11, NN + AC and the proposed controller can track the desired trajectory during the learning process since both controllers have the learning ability. However, the proposed controller has a faster trajectory tracking velocity than NN + AC, and its control torque is obviously smaller than the NN + AC; this is a favorable property, because high control torque can cause arm to fatigue faster.In Table2, the maximum steady-state position error (PEr) is defined as the maximum absolute value of error that occurs after 2 seconds of the trial. The maximum steady-state position errors range from 1.71 to 2.27 cm with a mean of 1.95 cm and a standard deviation of 0.16 cm. The RMS position errors range from 0.41 to 0.73 cm with a mean RMS error of 0.58 cm and a standard deviation of 0.09 cm. The maximum steady-state force (F) is defined as the maximum absolute value of force that occurs after 2 seconds of the trial. The maximum forces range from 4.53 to 9.82 N with a mean of 6.98 N and a standard deviation of 1.98 N. The RMS forces range from 0.11 to 0.27 N with a mean RMS error of 0.20 N and a standard deviation of 0.05 N. Table 2 indicates that the control accuracy from the experiment is sufficient for typical functional tasks. Nonetheless, these tests were conducted on some healthy individuals; thus, future efforts will focus on the real patients.Based on these results, it can be concluded that our NN + AHC is able to evaluate the patient’s performance during training and automatically change its assistance/resistance based on the volunteer’s requirement. However, at initial time, the control performance with the AHC is not better than the one with NN + AC, since RISE feedback structure in AC is more sensible to the disturbance and nonlinear than the inverse optimal controlleru01. Nevertheless, with the proceeding of NN + AHC learning, the inverse optimal controller begins to show better performance than the NN + AC’s; the control performance of the NN + AC becomes degraded due to its inherent accumulated error.


## 4.1. Simulation

To illustrate the effectiveness and robustness of the proposed control algorithm on robotic rehabilitation system (3), a comparison among the adaptive proportional-derivative (PD) control, the NN-based adaptive control (NN + PD), the AC-based adaptive robust control (NN + AC), and the proposed AC-based inverse optimal control (NN + AHC) is made. Relevant simulation results are shown in Figures 3–6; summary results between NN + AC and NN + AHC are shown in Table 1.Table 1
Summarized simulation results.
Simulation
Position error (degree)
Torque (Nm)
AC
AHC
AC
AHC
Maximum (TS)
9.163
7.491
1.163
0.531
Minimum (TS)
6.302
2.872
0.612
0.512
Mean (TS)
7.417
3.431
0.832
0.516
Adjusting time
7–11 s
3–7 s
7–11 s
3–7 s
Maximum (SS)
0.182
0.172
0.523
0.512
Minimum (SS)
0.076
0.057
0.506
0.500
Note: TS represents temporary state; SS denotes steady state.Presume the desired trajectory is selected as (in rads)(69)xd={sgn⁡(sin(0.05πt-0.5π));Δ>0,0;Δ≤0,
where Δ=sgn⁡(sin(0.05πt-0.5π)), which determines the desired trajectory.First, the adaptive PD control is applied to the simulation experiment, and the control parameters of the PD are taken asKP = 20, KD = 30, and Λ = 5I, and its simulation results are shown in Figure 3. From Figure 3, it is obvious that the adaptive PD control cannot track the desired position with the predefined control parameters. Specifically, the maximum position tracking error is about 0.8 rads; the minimum position tracking error is about 0.2 rads, and the steady mean error is about 0.6 rads.Then, an action NN is added to PD arriving at a NN-based adaptive control (NN + PD), which is applied to the simulation experiment, and its control parameters are taken asKP = 10, KD = 20, with λ = 3 (learning rate). The simulation results of the NN-based adaptive control are shown in Figure 4. Though the NN-based adaptive control can track the desired trajectory, its control performance still needs to be improved. Specifically, the maximum position tracking error is about 0.2 rads, the minimum position tracking error is less than 0.1 rads, and the steady mean error is about 0.13 rads.Thereafter, a critic NN is added to NN + PD arriving at an AC control (NN + AC), which is applied to the simulation, and its control parameters are selected as [45]. The simulation results of the AC-based robust adaptive control are shown in Figure 5. Though a favorable tracking performance can be achieved due to the effectiveness of the critic NN, the unwanted transients (e.g., 10 s, 30 s,…, 110 s, etc.) can result in unsatisfactory behavior.Finally, the AC-based inverse optimal control (NN + AHC) is applied to the simulation. At the same time, to furthermore illustrate the robustness of the proposed controller, external disturbanced=0.05sin(3πt)cos⁡(πt) is considered. The initial position is (0,0). Both critic NN and actor NN contain 16 nodes in hidden layer. For weight updating, the weights V and W are set to initial random [0,1]. All the activation functions (sigmoid) for hidden neurons are selected as
(70)φ(z)=11+exp⁡(-z).The gains for inverse optimal controller was selected as(71)k1=2.1,k2=1.8,kv=1.3,H=3,λ=0.6.The control gains for the critic NN and actor NN are selected as(72)ΥIw-1=Υaw-1=ΥAw-1=0.1I,ΥCv-1=ΥCw-1=1I.The choice of these parameters is through some trails and all the gains are chosen in consideration of the requirement of stability. The results of the AC-based inverse optimal control are shown in Figure6. A favorable high tracking performance can be achieved; the unwanted transients are almost avoided with little control torque. Specifically, the steady position tracking error of the proposed AHC decreases less than 0.001 rads in 43 s. Figure 2 shows that the meaningful cost function is positive, and thus the controller u01 in (29) can minimize a meaningful cost, which means the proposed AHC is optimal.(a) The meaningful cost functional. (b) The integral of the meaningful cost functional.
(a)
(b)Position tracking, tracking error, control torque, and weights of actor NN with PD.
(a)
(b)
(c)
(d)Position tracking, tracking error, control torque, and weights of actor NN with NN + PD.
(a)
(b)
(c)
(d)Figure 5
Position tracking, tracking error, control torque, weights of actor NN, and weights of critic NN with NN + AC.Figure 6
Position tracking, tracking error, control torque, weights of actor NN, and weights of critic NN with the proposed NN + AHC.Figures3–6 show the simulation results of traditional control and our proposed controller. From Figures 3–6, NN + PD, NN + AC, and the proposed controller can make the tracking error decrease during the learning process since these controllers have the learning ability. However, the proposed controller has a faster reduction rate in tracking errors than NN + PD, NN + AC with slightly reduced control torque. The control torque of the proposed controller is smoother and has a smaller vibration than NN + AC to achieve the requested level of performance. It means that NN + AC controller augmented by an inverse optimal can adjust the performance versus control input by minimizing a cost functional. It is concluded that the approximation ability of the proposed NN + AHC is better than the NN + AC control system, and the proposed AHC is optimal.Table1 indicates that the position tracking error with NN + AHC is approximately 9% smaller than the one with NN + AC, and resulting force is approximately smaller by 15%, and convergence speed increases by 36%. The control accuracy from the simulation is sufficient for typical functional tasks. Nonetheless, the performance of the proposed method may vary when implemented in the real rehabilitation robot, since the simulation condition is too ideal compared to the real application.


## 4.2. Tracking Experiment

Tracking experiments with two controllers (i.e., NN + AC and NN + AHC) were conducted on both arms of five volunteers (two females and three males, ages 22 to 36 yrs.). Designated as A, B, C, D, and E, subject A was used as an example and Figure7 is the experimental setup.Figure 7
A subject sitting on a chair with back support, holding the handle of the four-link robot manipulator. The interaction force generated by the subject onto the handle and the robot. The actual and reference handle positions were displayed on a monitor in front of the subject.To achieve better the control objective, each volunteer (e.g., A) first undergoes a standard assessment session before the experiment, in which he/she performs no less than 30 seconds reaching movements in eight specific movement directions only by himself/herself [51]. Then, according to the evaluation result, we make a suitable training task, as shown in Figure 8. Thereafter, after having a rest, he/she (e.g., A) is asked to track the desired trajectory (i.e., the green line and the red circle); in the first phase of the experiment (0–50 s), subject A draws circle and in the second phase of the experiment (50–100 s), subject A draws line, according to the instruction on a monitor as accurately as possible. However, since individual motor behavior varies, subject A may not be able to track the desired motion trajectory all along. Once the user’s motor behavior varies, the robot will adjust automatically the assistance modes (i.e., assistance or resistance) using the proposed controller. The experiment results of subject A are shown in Figures 8, 9, 10, and 11, and the whole experimental results of five subjects tested for the desired trajectories are summarized in Table 2.Table 2
Summarized experimental results of five volunteers.
Sub.
Maximum
RMS error
NN + AC
NN + AHC
NN + AC
NN + AHC
PEr
F
PEr
F
PEr
F
PEr
F
Units
cm
N
cm
N
cm
N
cm
N
A (L)
2.12
9.96
2.02
9.33
0.67
0.26
0.66
0.26
A (R)
2.16
10.15
2.10
9.82
0.68
0.27
0.65
0.27
B (L)
1.92
7.08
1.71
6.36
0.75
0.28
0.56
0.19
B (R)
1.96
7.15
1.73
6.53
0.77
0.28
0.48
0.16
C (L)
2.33
9.13
2.23
8.70
0.86
0.34
0.73
0.27
C (R)
2.31
9.03
2.27
8.64
0.78
0.31
0.64
0.25
D (L)
1.78
4.64
1.73
4.53
0.42
0.11
0.56
0.16
D (R)
1.76
4.59
1.72
4.57
0.39
0.10
0.41
0.11
E (L)
2.02
5.83
2.02
5.62
0.62
0.23
0.56
0.17
E (R)
2.03
5.86
1.96
5.65
0.58
0.21
0.58
0.18
Mean
2.04
7.34
1.95
6.98
0.65
0.24
0.58
0.20
S.D.
0.20
2.34
0.16
1.98
0.12
0.07
0.09
0.05
Note: A-B denotes five volunteers, L represents left arm, R represents right arm, PEr represents position tracking error, andF represents the result force; S.D. denotes statistically significant value.Figure 8
Trajectory tracking with NN + AC.Figure 9
Trajectory tracking with NN + AHC.(a) Resultant force with NN + AC. (b) Velocity of resultant force with NN + AC.
(a)
(b)(a) Resultant force with NN + AHC. (b) Velocity of resultant force with NN + AHC.
(a)
(b)Figures8–11 show the experiment results of NN + AC and our proposed controller. From Figures 8–11, NN + AC and the proposed controller can track the desired trajectory during the learning process since both controllers have the learning ability. However, the proposed controller has a faster trajectory tracking velocity than NN + AC, and its control torque is obviously smaller than the NN + AC; this is a favorable property, because high control torque can cause arm to fatigue faster.In Table2, the maximum steady-state position error (PEr) is defined as the maximum absolute value of error that occurs after 2 seconds of the trial. The maximum steady-state position errors range from 1.71 to 2.27 cm with a mean of 1.95 cm and a standard deviation of 0.16 cm. The RMS position errors range from 0.41 to 0.73 cm with a mean RMS error of 0.58 cm and a standard deviation of 0.09 cm. The maximum steady-state force (F) is defined as the maximum absolute value of force that occurs after 2 seconds of the trial. The maximum forces range from 4.53 to 9.82 N with a mean of 6.98 N and a standard deviation of 1.98 N. The RMS forces range from 0.11 to 0.27 N with a mean RMS error of 0.20 N and a standard deviation of 0.05 N. Table 2 indicates that the control accuracy from the experiment is sufficient for typical functional tasks. Nonetheless, these tests were conducted on some healthy individuals; thus, future efforts will focus on the real patients.Based on these results, it can be concluded that our NN + AHC is able to evaluate the patient’s performance during training and automatically change its assistance/resistance based on the volunteer’s requirement. However, at initial time, the control performance with the AHC is not better than the one with NN + AC, since RISE feedback structure in AC is more sensible to the disturbance and nonlinear than the inverse optimal controlleru01. Nevertheless, with the proceeding of NN + AHC learning, the inverse optimal controller begins to show better performance than the NN + AC’s; the control performance of the NN + AC becomes degraded due to its inherent accumulated error.


## 5. Conclusions

A Lyapunov-based stability analysis indicates that the developed adaptive AC-based inverse optimal hybrid control method yields global asymptotic tracking for an unknown nonlinear rehabilitation robot and human motor behavior dynamics, even in the presence of uncertain additive disturbances. Simulation and experimental results clearly illustrate that the proposed AHC controller enables a person to achieve a desired functional task while minimizing a meaningful cost and the proposed AHC performs better in terms of reduced error and convergence speed in comparison to AC controller. However, the performance of the controller on volunteers or patients remains to be seen. Future efforts will focus on improving the practicable intelligent control algorithms for rehabilitation robot and experimental trials on more volunteers, potentially including stroke/hemiplegia patients and so on.



---

*Source: 285248-2014-08-28.xml*