# Decentralized Reinforcement Learning Robust Optimal Tracking Control for Time Varying Constrained Reconfigurable Modular Robot Based on ACI andQ-Function

Authors: Bo Dong; Yuanchun Li
Journal: Mathematical Problems in Engineering
 (2013)

---

## Abstract

A novel decentralized reinforcement learning robust optimal tracking control theory for time varying constrained reconfigurable modular robots based on action-critic-identifier (ACI) and state-action value function (Q-function) has been presented to solve the problem of the continuous time nonlinear optimal control policy for strongly coupled uncertainty robotic system. The dynamics of time varying constrained reconfigurable modular robot is described as a synthesis of interconnected subsystem, and continuous time state equation and Q-function have been designed in this paper. Combining with ACI and RBF network, the global uncertainty of the subsystem and the HJB (Hamilton-Jacobi-Bellman) equation have been estimated, where critic-NN and action-NN are used to approximate the optimal Q-function and the optimal control policy, and the identifier is adopted to identify the global uncertainty as well as RBF-NN which is used to update the weights of ACI-NN. On this basis, a novel decentralized robust optimal tracking controller of the subsystem is proposed, so that the subsystem can track the desired trajectory and the tracking error can converge to zero in a finite time. The stability of ACI and the robust optimal tracking controller are confirmed by Lyapunov theory. Finally, comparative simulation examples are presented to illustrate the effectiveness of the proposed ACI and decentralized control theory.


---

## Body

## 1. Introduction

Reconfigurable modular robot could transform its configuration depending on the different external situations and the requirements of the tasks. According to the concept of modular design and the decentralized control theory of the subsystem, reconfigurable modular robot can complete the task by changing its structure efficiently in different situations, without redesigning the control law. At the same time, reconfigurable modular robot possessed a good accuracy and flexibility.Many scholars have studied the dynamics and the control method of reconfigurable modular robot. A novel VGSTA-ESO based decentralized ADRC control method for reconfigurable modular robot has been proposed in [1]. Through designing a high-precision VGSTA-ESO to estimate the dynamic model nonlinear terms and the interconnection terms of the subsystem, the joint trajectory tracking control is implemented. Based on calculating torque, a robust fuzzy neural network controller is proposed in [2], which is used to solve the problem of model uncertainty in the process of model generating. In [3], it shows a decentralized adaptive fuzzy sliding mode control method of reconfigurable modular robot. The fuzzy logic system is used to approximate the unknown dynamics of the subsystem, and a sliding mode controller with an adaptive scheme is designed to avoid both the interconnection term and the fuzzy approximation error. A decentralized adaptive neural network control algorithm for reconfigurable manipulators is proposed in [4], where the neural networks are used to approximate the unknown dynamic functions and interconnections in the subsystem by using the adaptive algorithm. A new distributed control method is proposed in [5], which uses a decomposition algorithm to decompose the robot dynamic system into a number of dynamical systems, and the adaptive sliding mode controller is designed to offset the impact of model uncertainty. An observer based decentralized adaptive fuzzy controller for reconfigurable manipulator is proposed in [6]; by designing the state observer, the adaptive fuzzy systems which are used to model the unknown dynamics of the subsystem and the interconnection term can be constructed by using the state estimations. Nevertheless, the requirement of the dynamics of the reconfigurable modular robot system is hard to be satisfied, either fully or even partially know. Moreover, because of there are strong coupling model uncertainties and interconnection terms of subsystems in the reconfigurable modular robot system, besides the processing load on the controller would be increased, the fact that the greater time delay and calculation error are easy to produce, so that it is too complicated to design the controllers by using the methods and algorithms above.In recent years, as one of the most effective methods to solve the control problems for continuous time which strongly coupled with nonlinear system, the reinforcement learning algorithm has received extensive attention from scholars. Reinforcement learning [7, 8] is a kind of learning method mapping situations to actions, so as to maximize a numerical reward signal. Compared with supervised learning, reinforcement learning does not need to predict the mentor signal in various states, but learns in the process of interaction with the situation. Because of its adaptive optimization capability in the nonlinear model under the condition of uncertainty, reinforcement learning has a unique advantage to solve the problems of optimization strategies and the control method in terms of the complex models [9–11]. Zhang and his team presented an infinite time optimal tracking control scheme for discrete-time nonlinear system via the greedy HDP iteration algorithm [12–14]. According to the system transformation, the optimal tracking problem is transformed into an optimal regulation problem, and the greedy HDP iteration algorithm is introduced to deal with the regulation problem with the rigorous convergence analysis. Then, a data-driven robust approximate optimal tracking control is proposed by using the adaptive dynamic programming, as well as a data-driven model which is established by a recurrent neural network to reconstruct the unknown system dynamics by using available input-output date [15]. After this, they design a fuzzy critic estimator which is used to estimate the value function for nonlinear continuous-time system [16]. On this basis, a synchronization problem for an array of neural networks with hybrid coupling and interval time varying delay is concerned with an augmented Lyapunov-Krasovskii functional method [17]. The FRL scheme only using immediate reward and sufficient conditions is adopted to analyze the convergence of the optimal task performance. Bhasin presents a neural network control of a robot interacting with an uncertain viscoelastic environment [18]. In [18], a continuous controller is developed for a robot that moved in free space and then regulated the new coupled dynamic system to a desired setpoint. Khan et al. present an implementation of a model-free Q-learning based on the discrete model reference compliance controller for a humanoid robot arm [19], where reinforcement learning scheme uses a recently developed Q-learning scheme to develop an optimal policy online. Patchaikani et al. propose an adaptive critic-based real-time redundancy resolution scheme for kinematic control of the redundant manipulator [20]. The kinematic control of the redundant manipulator is formulated as a discrete-time input affine system; and then, an optimal real-time redundancy resolution scheme is proposed. Although the research of reinforcement learning algorithm has been rapidly developed in recent years, there are still some deficiencies. For example, when there are multiple subsystems in the global system, the methods above could not handle the impacts of the interconnection terms between the subsystems. Meanwhile, the methods above are mostly used to solve the learning and optimization problems of the system itself, but when the external constraints exist in the system, these methods are no longer applicable. Therefore, it is an urgent issue to solve the problem of how to design a kind of robust reinforcement learning optimal control method in the case of the external constraints existing and multiple subsystems coupled in the system are the urgent problems to be solved.In this paper, we presented a novel continuous time decentralized reinforcement learning robust optimal tracking control theory for the time varying constrained reconfigurable modular robot. Combining with ACI and RBF-NN, the critic-NN is used to estimate the optimalQ-function; the action-NN is proposed to approximate the optimal control policy; and then, the identifier is adopted to identify the global uncertainty, so that the HJB equation can be estimated and the estimation error is bounded and converged. Firstly, since the decentralized control method is adopted in this paper, which means that each joint subsystem owns a separate controller, thus the processing loads of the controllers are reduced greatly. Secondly, due to the fact that the time varying constraints can be compensated in the subsystems, therefore the proposed method in this paper is suitable for the reconfigurable modular robot in the time varying constrained outside environment. Thirdly, the proposed control method could compensate for the impacts of the model uncertainties and the interconnection terms on the system, so that it can make the subsystems track the desired trajectories and the tracking error can converge to zero in finite time.


## 2. Problem Formulation

Assume that the time varying external constraints for the end of reconfigurable modular robot is shown as(1)Ψ(q,t)=0.Hereq∈Rn is the vector of joint displacements. Function Ψ:Rn→Rm, m is the dimension of the external limiting conditions. With the time varying constraints, the dynamics of a reconfigurable modular robot can be presented as follows:
(2)M(q)q¨+C(q,q˙)q˙+G(q)+F(q,q˙)=u+JΨT(q,t)f.M
(
q
)
∈
R
n
×
n is the inertia matrix, C(q,q˙)∈Rn is the Coriolis and centripetal force, G(q)∈Rn is the gravity term, F(q,q˙) is the unmodeled dynamics including friction terms and external disturbances, u∈Rn is the applied joint torque, and JΨT(q,t)f is the contact force generated by the contact of the end of the reconfigurable modular robot and external constraints.After introducingmth constraints for the robot which works in the free space, because of the limitation of (1), the system lost mth degrees of freedom. Therefore, the degrees of freedom of the robot change from n to (n-m), so that only (n-m) independent joint displacements are needed to describe the system of restricted movement fully.Define(3)q=[q1q2]q1∈Rn-m,q2∈Rm.
Putting the equation above into (1); then we can get that
(4)Ψ(q1,Θ(q1,t),t)=0,
where
(5)q2=Θ(q1,t).Therefore, (3) can be described by joint displacement q1 fully, shown as follows:
(6)q=[q1Θ(q1,t)].
The derivation of (6) is
(7)q˙=[q˙1∂Θ(q1,t)∂q1q˙1+∂Θ(q1,t)∂t]=[In-m0∂Θ(q1,t)∂q1Im][q˙10]+[0∂Θ(q1,t)∂t]=Tθ˙+H.
In (7),
(8)T=[In-m0∂Θ(q1,t)∂q1Im]∈Rn×n,θ=[q˙10]∈Rn,H=[0∂Θ(q1,t)∂t]∈Rn.
Therefore, the second derivation of q can be achieved easily as
(9)q¨=Tθ¨+T˙θ˙+H˙.
Putting (7) and (9) into (2), we can get
(10)u+JΨT(q,t)f=M(q)(Tθ¨+T˙θ˙+H˙)+C(q,q˙)(Tθ˙+H˙)+G(q)+F(q,q˙).
Define
(11)E=[I(n-m)×(n-m)0m×(n-m)]∈Rn×(n-m).
Therefore
(12)θ=[q10]=Eq1
So, (2) can be decomposed into the following form:
(13)∑j=1nMij(q)[(TEq¨1)j+(T˙Eq˙1)j+H˙j]+∑j=1nCij(q,q˙)[(TEq˙1)j+Hj]+G-i(q)+Fi(qi,q˙i)-fi=ui.
In the equation above, (TEq¨1)j, (T˙Eq˙1)j, (TEq˙1)j, and Hj are the jth element of (TEq¨1), (T˙Eq˙1), (TEq˙1) and H, respectively; G-i(q), Fi(qi,q˙i), and ui are the ith element of G(q), F(q,q˙), and u; fi is the constraint force which suffered by the ith joint. Mij(q) and Cij(q,q˙) are the ijth element of M(q) and C(q,q˙), respectively. So, as shown in Figure 1, each subsystem dynamical model can be formulated in joint space as follows:
(14)Mi(qi)q¨i+Ci(qi,q˙i)q˙i+Gi(qi)+Fi(qi,q˙i)+Zi(q,q˙,q¨)=ui,(15)Zi(q,q˙,q¨)=∑j=1j≠inMij(q)[(TEq¨1)j+(T˙Eq˙1)j+H˙j]+Mii(q)[(TEq¨1)i+(T˙Eq˙1)i+H˙i]-Mi(qi)q¨i+∑j=1j≠inCij(q,q˙)[(TEq˙1)j+Hj]+Cii(q,q˙)[(TEq˙1)j+Hj]-Ci(qi,q˙i)q˙i+[G-i(q)-Gi(qi)].
Let xi=[xi1,xi2]T=[qi,q˙i]T for i=1,…,n; then (10) can be presented by the following state equation:
(16)Si:{x˙i1=xi2x˙i2=-f(xi,ui)-hi(q,q˙,q¨)-fi,yi=xi1,
where xi is the state vector of subsystem Si, yi is the output of subsystem Si, and hi(q,q˙,q¨) is the interconnection term of the subsystem. f(xi,ui) and hi(q,q˙,q¨) can be defined as
(17)f(xi,ui)=Mi-1(qi)[Ci(qi,q˙i)q˙i+Gi(qi)+Fi(qi,q˙i)-ui]hi(q,q˙,q¨)=-Mi-1(qi)Zi(q,q˙,q¨).Figure 1
The architecture of the time varying constrained reconfigurable modular robot system.In response to the time varying constrained reconfigurable modular robot system, we need to design a decentralized robust optimal tracking control policy to make the subsystem track the desired trajectory, as well as the tracking error is converged and bounded.


## 3. Decentralized Reinforcement Learning Robust Optimal Tracking Control Based on ACI andQ-Function

Assumption 1.
Desired trajectoryyid, y˙id, y¨id and input gain matrix bi(xi) are bounded.
Then (16) can be transformed to the below. Consider
(18)Si:{x˙i1=xi2x˙i2=-[F(xi,ui)+hi(q,q˙,q¨)+fi]+bi(xi)ui,yi=xi1,
where F(xi,ui)=f(xi,ui)+bi(xi)ui.Assumption 2.
The interconnection terms are bounded, satisfying the following equation:(19)|hi(q,q˙,q¨)|≤δi0+∑j=1nδij(|sij|),
where δi0>0 is an unknown constant and δij(|sij|)≥0 is an unknown smooth Lipschitz function.
The trajectory tracking error of the joint subsystemc can be defined as
(20)ei(t)=xi-yid.
With regard to the continuous time state equation of the subsystem in (18) with the nonlinear function and interconnection terms, generally, the value function can be defined as
(21)Viui(ei(t))(ei(t))=∫0∞ri(ei(t),ui(ei(t)))dt.
In order to facilitate the equation, we useei, ui instead of ei(t), ui(ei(t)). Since the trajectory yid relies upon the control of the subsystem ui for updating, in order to avoid the infinity results by using (21), we need to transform the value function into the following form:
(22)Viui(ei)=∫0∞ri(ei(τ),ui(ei(τ)))dτ,t≤τ<∞.
Thus, the optimal value function of the subsystem can be defined as follows:
(23)Vi*(ei)=minuit≤τ<∞∫0∞ri(ei(τ),ui(ei(τ)))dτ.
Here ri(ei,ui) represents the reward function for the current state, shown as
(24)ri(ei,ui)=eiTQeei+uiTRui,
where Qe and R are the positive definite matrixes.Typically, recording the value of state-action pairs is more useful than recording the value of state only, since the state-action pairs are the predictions of the reward. Even if the reward value of a state is low, it does not mean that the value of state-action pairs is low too. If the state of the subsystem in a period time produces a higher reward, then it can still get a higher state-action value. Therefore, from a long term perspective, defining a suitable state-action value function (Q-function) can make actions produce more rewards [21, 22].According to (23) and (24), the continuous-time optimal Q-function can be defined as
(25)Qi*(ei,ai,ui)=ri(ei,ai,ui)+Vi*(ei,ui)=ri(ei,ai,ui)+minuit≤τ<∞∫0∞ri(ei(τ),ui(ei(τ)))dτ.Assumption 3.
The partial derivation ofQi* and ri(ei,ai,ui) exist and they are continuous in the domain. According to (18) and (24), by using the control policy ui, the optimal Q-function can satisfy the following Hamiltonian-Jacobi-Bellman equation [23]:
(26)HJBi(ei,ui,∇Qi*)=minui(ei)[ri(ei,ai,ui)+∇Qi*(-F(ei,ui)-hi(e,e˙,e¨)-fi+bi(ei)ui)]=minui(ei)[ri(ei,ai,ui)+∇Qi*Φi(ei,ui)],
where Φi(ei,ui)=-F(ei,ui)-hi(e,e˙,e¨)-fi+bi(ei)ui means the global uncertainty including the unknown dynamics of the subsystem and the interconnection term, and ∇Qi*=∂Qi*/∂ei means the gradient of the optimal Q-function.Lemma 4 (see [24]).
Considering dynamics of the subsystem of time varying constrained reconfigurable modular robot in (14), in order to ensure the minimum of the HJB equation (26) possessing the stationary point with respect to ui, the optimal Q-function and the optimal control policy must satisfy the following conditions: (1)
∂HJB(ei,ui,∇Qi)/∂ui=0(2)
∂2HJBi(ei,ui,∇Qi)/(∂ui×∂uiT)≥0.
The necessary conditions above lead us to the following results.(a)
The bounded control policy can guarantee a local minimum of the HJB equation (26) and satisfy the constraints imposed on the control inputs.(b)
The Hessian matrix is positive-definite, and the control policeui can render the global minimum of the HJB equation.(c)
If an optimal algorithm exists, it is unique.
According to Lemma4, if the reward function is smooth, and the optimal control ui* is adopted, then the HJB equation satisfies the following equation:
(27)HJBi*(ei,ui*,∇Qi*)=minui*(ei)[ri(ei,ai,ui*)+∇Qi*Φi(ei,ui*)]=0.
And the optimal control can be expressed as follows:
(28)ui*(ei)=argui*min[HJBi*(ei,ui*,∇Qi*)]=12R-1biT(ei)∂Qi*(ei,ai,ui)T∂ei.If the optimalQ-function Qi* is continuous, derivable, and known, and the initial value Qi*(0)=0, as well as the optimal control policy ui*(ei), and the global uncertainty of the subsystem Φi(ei,ui*) is known, then the HJB equation in (27) is held and solvable. However, in the actual situation, Qi* is not derivable everywhere, and ui*(ei) and Φi(ei,ui*) are unknown. Therefore, it is not feasible to solve the HJB equation by using average method. In this paper, we combine the action-critic identifier (ACI) with RBF neural network to estimate the optimal control policy, the optimal Q-function, and the global uncertainty of the subsystem. Action-NN is used to estimate ui*(ei) and is denoted as u^i(ei); Qi* is estimated by critic-NN and expressed as Q^i; then we use the robust neural network identifier to identify Φi(ei,ui*), denoted as Φ^i(ei,ui*). The block diagram of the ACI architecture is shown in Figure 2.Figure 2
The architecture of action-critic-identifier.The estimated HJB equation can be expressed as follows:(29)HJ^Bi*(ei,u^i,∇Q^i)=minui(ei)[ri(ei,ai,u^i)+∇Q^iΦ^i(ei,u^i)].
The identification error of the HJB equation above can be expressed as
(30)δhi=HJ^Bi*(ei,u^i,∇Q^i)-HJBi*(ei,ui*,∇Qi*).
A classic radial basis function of the neural network is proposed in [25], shown as (31):
(31)N(x)=W*TS(x)+ε(x),
where W* means the ideal neural network weights and ε(x) represents the estimation error. In the case of using sufficient number of nodes, if the center and width of the nodes are built appropriately, then any kind of continuous function could be approximated by RBF-NN. Therefore, the optimal Q-function and the optimal control policy can be expressed as follows:
(32)Qi*=WiTSi(ei)+εic(ei)ui*(ei)=-12R-1biT(ei)[S˙i(ei)TWi+εia(ei)],
where Si(ei)=[si1(ei)⋯sin(ei)]T indicates the smooth basis function of the neural network, Wi means the ideal unknown neural network weight, and εic(ei) and εia(ei) are the estimation error. By using Q^i and u^i(ei) to estimate Qi* and ui*(ei), we can get the following equations:
(33)Q^i=W^icTSic(ei)(34)u^i(ei)=-12R-1biT(ei)S˙ia(ei)TW^ia.
According to the equations above, W^ic(t) and W^ia(t) indicated the weights of critic-NN and action-NN. And the estimation errors of weights are shown as follows:
(35)W~ic(t)=Wi-W^ic(t)(36)W~ia(t)=Wi-W^ia(t).
The update law of the weight for the critic-NN is a gradient descent algorithm, which is shown as follows:
(37)W^˙ic(t)=-n1Li(LiTW^ic+eiTQeei+uiTRui).
In the equation above, ni>0 is the adaptive gain of the neural network. Li and li are defined as
(38)Li=liliTli+1,li=∇Sic(ei)ei.
Therefore, according to the definition above, the following inequalities can be obtained:
(39)Lim≤Li≤LiM,Sicm≤Sic(ei)≤SicM,Siam≤Sia(ei)≤SiaM.
Combining (35) with (38), we can get that
(40)W~˙ic(t)=-n1Li(LiTW~ic+δhi).
The update law of the weight for the action-NN is developed by a gradient descent algorithm, expressed as follows:
(41)W^˙ia(t)=-n2Sia(ei)×((W^iaTSia(ei)+12R-1bi(ei)∇Sic(ei)TW^ic))T.
According to the estimation error of action-NN in (36), the optimal control ui*(ei) can minimize the optimal Q-function, and we can get the following equation:
(42)WiaTSia(ei)+12R-1bi(ei)∇Sic(ei)TWic+12R-1bi(ei)∇εic(ei)+εia(ei)=0.
Putting (41) into (42), we can get that
(43)W~˙ia=-n2Sia(ei)(W~iaTSia(ei)+12R-1bi(ei)∇Sic(ei)TW~ic-12R-1bi(ei)∇εic(ei)-εia(ei)).
After using critic-NN and action-NN to estimate Q^i and u^i(ei), we need to design a kind of robust RBF-NN identifier to identify the nonlinear uncertainties of the subsystem. Here Φi(ei,u^i) can be expressed as follows:
(44)Φi(ei,u^i)=e˙iF=WiFTκ(ΛiFTeiF)+εiF(eiF)+bi(ei)u^i,
where κ(·) means the basic function of neural network, and WiF, ΛiF indicate the unknown ideal neural network weights. Equation (44) can be identified by using robust RBF-NN identifier, so we can get
(45)Φ^i(e^i,u^i)=e^˙iF=WiFTκ^iF+bi(ei)u^i+μi.
Here κ^iF indicates the estimated value of the basic function of the neural network WiF, ΛiF are expressed as the estimated value of neural network. μi∈ℝ means the feedback error term, shown as follows [26]:
(46)μi=k(eiF(t)-e^iF(t))-k(eiF(0)-e^iF(0))+ϑ=k(e~iF(t)-e~iF(0))+ϑϑ˙=(kα+γ)eiF+β1sat(eiF),
where k, α, β1, and γ are the positive control gain constants and sat(·) is a saturation function. Therefore, the state estimation error of the identifier-NN can be expressed as follows:
(47)e~˙iF=e˙iF-e^˙iF=WiFTκiF-W^iFTκ^iF+εiF(eiF)-μi.
A filtered identification error is defined as follows:
(48)Ei=e~˙iF+αe~iF.
The derivation of the equation above is shown as
(49)E˙i=WiFTκ˙iFΛiFTe˙iF-W^iFTκ^˙iFΛ^˙iFTe^iF+αe~˙iF-kEi-γe~iF-W^˙iFTκ^iF+ε˙iF(eiF)-β1sat(e~iF)-W^iFTκ^˙iFΛ˙iFTe^˙iF.
Here, the weight WiF, ΛiF of the identification-NN can be updated by
(50)W^˙iF=proj(ΓiWFκ^˙iFΛ^iFTe^˙iFe~iFT)Λ^˙iF=proj(ΓiΛFe^˙iFe~iFTW^iFTκ^˙iF),
where ΓiWF, ΓiΛF are positive constant adaptation gain matrices. In order to analyze the convergence of the filtered identification error, W^iFTκ^˙iFΛ˙iFTe^˙iF can be divided into the following form:
(51)W^iFTκ^˙iFΛ˙iFTe^˙iF=12κ^˙iFe^˙iF[(ΛiFT-Λ~iFT)(WiFT-W~iFT)+(WiFT-W~iFT)(Λ~iFT-ΛiFT)]=12κ^˙iFe^˙iF[W~iFT(ΛiFT-Λ~iFT)+(WiFT-W~iFT)Λ~iFT-WiFT(ΛiFT-Λ~iFT)-(WiFT-W~iFT)ΛiFT]=12κ^˙iFe^˙iF[W~iFT(ΛiFT-Λ~iFT)+(WiFT-W~iFT)Λ~iFT]-12κ^˙iFe^˙iF[WiFT(ΛiFT-Λ~iFT)+(WiFT-W~iFT)ΛiFT]=12WiFTκ^˙iFΛ^iFTe~˙iF+12W^iFTκ^˙iFΛiFTe~˙iF-12WiFTκ^˙iFΛ^iFTe˙iF-12W^iFTκ^˙iFΛiFTe˙iF+12W~iFTκ^˙iFΛ^iFTe^˙iF+12W^iFTκ^˙iFΛ~iFTe^˙iF,
where W~iFT=WiFT-W^iFT, Λ~iFT=ΛiFT-Λ^iFT. Putting (51) into (49), then (49) can be reduced to the following form:
(52)E˙i=PF1+PF2+PF3-kEi-γe~iF-β1sat(e~iF).
Among the equations above, PF1+PF2+PF3 can be expressed, respectively, as follows:
(53)PF1=12WiFTκ^˙iFΛ^iFTe~˙iF+12W^iFTκ^˙iFΛiFTe~˙iF-W^iFTκ^˙iFΛ^˙iFTe^iF+αe~˙iF-W^iFTκ^iF(54)PF2=-12WiFTκ^˙iFΛ^iFTe˙iF-12W^iFTκ^˙iFΛiFTe˙iF+WiFTκ˙iFΛiFTe˙iF+ε˙iF(eiF)(55)PF3=12W~iFTκ^˙iFΛ^iFTe^˙iF+12W^iFTκ^˙iFΛ~iFTe^˙iF.
According to Assumption 1, (48), and (50), the upper bounds of PF1, PF2, PF3 are shown as
(56)∥PF1∥≤J1(∥φi(e~iFT,EiT)∥)∥φi(e~iFT,EiT)∥∥PF2∥≤ς1∥PF3∥≤ς2.
Combining (53) and (54) with (55), then we can get that:
(57)∥P˙F2+P˙F3∥≤ς3+ς4J2(∥φi(e~iFT,EiT)∥)∥φi(e~iFT,EiT)∥,
where φi(e~iFT,EiT)=[e~iFTEiT]T and Ji(·) is a global invertible nondecreasing function. ςi, (i=1,2,3,4) are computable positive constants.Theorem 5.
Considering dynamics of the subsystem of time varying constrained reconfigurable modular robot in (14) and the state equation (18), if the designed identifier and the corresponding weight update laws are adopted, then the global uncertainty of the subsystem which depends explicitly on the error term can be identified, and the identification error is converged and bounded.Proof.
Define the Lyapunov function as the follows:(58)ViL(eiF,Ei)=12EiTEi+12γe~iFTe~iF+ϖi(t)+ϕi(t).
In the equation above, ϖi(t) and ϕi(t) can be expressed as follows:
(59)ϖ˙i(t)=-[EiT(PF2-β1sat(e~iF))+e~˙iFTPF3-β2J2(∥φi(e~iFT,EiT)∥)∥φi(e~iFT,EiT)∥∥e~iF∥],ϖi(0)=β1|e~iF(0)|-e~iFT(0)(PF2(0)+PF3(0))(60)ϕi(t)=14α[tr(W~iFTΓiWF-1W~iF)+tr(Λ~iFTΓiΛF-1Λ~iF)],
where tr(·) represents the trace of matrix. Defining d=[EiTe~iFTϖi1/2ϕi1/2], β1, β2∈ℝ are positive adaptation gains which are chosen to ensure ϖi(t)≥0, so we can get
(61)U1(d)≤ViL(eiF,Ei)≤U2(d),
where
(62)U1(d)=12min(1,γ)∥d∥2U2(d)=max(1,γ)∥d∥2.
The derivation of (58) is shown as follows:
(63)V˙iL(eiF,Ei)=∇ViLTK[E˙ie~˙iFT12ϖi1/2ϖ˙i12ϕi1/2ϕ˙i]T,
where K[·] is expressed as a Filipov set [27].
SoV˙iL(eiF,Ei) can be deformed as the following form:
(64)V˙iL(eiF,Ei)=[EiTγe~iFT2ϖi1/22ϕi1/2]K[E˙ie~˙iFT12ϖi1/2ϖ˙i12ϕi1/2ϕ˙i]T≤γT(12WiFTκ^˙iFΛ^iFTe~˙iF+12W^iFTκ^˙iFΛiFTe~˙iF-W^iFTκ^˙iFΛ^˙iFTe^iF+αe~˙iF-12WiFTκ^˙iFΛ^iFTe˙iF-12W^iFTκ^˙iFΛiFTe˙iF-γe~iF-W^iFTκ^iF+WiFTκ˙iFΛiFTe˙iF+ε˙iF(eiF)+12W~iFTκ^˙iFΛ^iFTe^˙iF+12W^iFTκ^˙iFΛ~iFTe^˙iF-kEi-β1K[sat(e~iF)])-EiT(-12WiFTκ^˙iFΛ^iFTe˙iF-12W^iFTκ^˙iFΛiFTe˙iF+WiFTκ˙iFΛiFTe˙iF+ε˙iF(ei)-β1K[sat(e~iF)])-e~˙iFT12(W~iFTκ^˙iFΛ^iFTe^˙iF+12W^iFTκ^˙iFΛ~iFTe^˙iF)+γe~iFT(Ei-αe~iF)+β2J2(∥φi(e~iFT,EiT)∥)∥φi(e~iFT,EiT)∥∥e~iF∥-12αtr(W~iFTΓiWF-1W^˙iF)-12αtr(Λ~iFTΓiΛF-1Λ^˙iF).
Put (53), (54), and (55) into (64); then we can get:
(65)V˙iL(eiF,Ei)=EiT(PF1+PF2+PF3-β1K[sat(e~iF)]-kEi-γe~iF)+γe~iFT(Ei-αe~iF)-EiT(PF2-β1K[sat(e~iF)])-e~˙iFTPF3+β2J2(∥φi(e~iFT,EiT)∥)∥φi(e~iFT,EiT)∥∥e~iF∥-12αtr(W~iFTΓiWF-1W^˙iF)-12αtr(Λ~iFTΓiΛF-1Λ^˙iF)=-αγe~iFTe~iF+(EiT-e~˙iFT)PF3J1(∥φi(e~iFT,EiT)∥)24k3-12αtr(W~iFTκ^˙iFΛ~iFTe^˙iFe~iFT)-12αtr(Λ~iFTe^˙iFe~iFTW^iFTκ^˙iF)≤-k1∥e~iF∥2-k2∥Ei∥2+J1(∥φi(e~iFT,EiT)∥)24k3∥φi(e~iFT,EiT)∥2+β22J2(∥φi(e~iFT,EiT)∥)24αk4∥φi(e~iFT,EiT)∥2,
where kmin=min{k1,k2}, ξ=min{k3,αk4/β22}, and J(∥φi(e~iFT,EiT)∥)2  =  J1(∥φi(e~iFT,EiT)∥)2  +  J2(∥φi(e~iFT,EiT)∥)2, so the following conclusion can be obtained:
(66)V˙iL(eiF,Ei)≤-kmin∥φi(e~iFT,EiT)∥2+J(∥φi(e~iFT,EiT)∥)2∥φi(e~iFT,EiT)∥24ξ≤-c∥φi(e~iFT,EiT)∥2.
Therefore, for an arbitrary constantc, -c∥φi(e~iFT,EiT)∥2 is a negative semidefinite function which is defined in the adjustable interval D, expressed as follows:
(67)D={d(t)∣∥d∥≤J-1(2kminξ)},
so that Lyapunov stability theory shows that the system is stable. In order to make the subsystem of time varying constrained reconfigurable modular robot tracking the desired trajectory progressively, in this paper, a novel decentralized reinforcement learning robust optimal tracking controller has been designed by using the robust term to compensate the neural network approximation errors. Design the robust control term as(68)uirb=NrbeieiTei+ζ.
In the equation above,ζ>0 is a constant. And Nrb can be expressed as
(69)Nrb≥[δhi22n1+n1(-εia(ei)-R-1bi(ei)(∇εic(ei)/2))22n2+n1n2∥bi(ei)∥2(-εia(ei)-R-1bi(ei)∇εic(ei)2)2δhi22n1+n1(-εia(ei)-R-1bi(ei)(∇εic(ei)/2))22n2]·(eiTei+ζ)2n1n2∥bi(ei)∥eiTei≥[n12(-εia(ei)-R-1bi(ei)∇εic(ei)2)2+n2δhi2+2n12n22∥bi(ei)∥2·(-εia(ei)-R-1bi(ei)∇εic(ei)2)2]·(eiTei+ζ)4n12n22∥bi(ei)∥eiTei.
Therefore, the global control law can be designed as follows:(70)umix=ui+uirb=-12R-1biT(ei)S˙ia(ei)TW^ia+NrbeieiTei+ζ.Theorem 6.
Considering dynamics of the subsystem of time varying constrained reconfigurable modular robot in (14), if the system parameters conditions and the assumptions are held, the critic-NN, action-NN, and identifier are given by (33), (34), and (45), respectively, and the decentralized robust optimal tracking controller of the subsystem in (70) is adopted; then, the system is closed-loop stability and the desired trajectory can be tracked asymptotically by the actual output.Proof.
Design the Lyapunov function as follows:(71)Viu(ei,umix)=12n1tr{W~icTW~ic}+n12n2tr{W~iaTW~ia}+n1n2[eiFTeiF+Ξ∫0∞ri(ei,umix)dτ],
where Ξ>0 is the undetermined parameter, t≤τ<∞. The derivation of (71) is shown as follows:
(72)V˙iu(ei,umix)=12n1tr{W~icTW~˙ic}+n12n2tr{W~iaTW~˙ia}+n1n2[eiFTe˙iF+Ξri(ei,umix)]=12n1tr{W~icT(-n1Li(LiTW~ic+δhi))}+n12n2tr×{W~iaT[-n2Sia(ei)(W~iaTSia(ei)-εia(ei)+12R-1bi(ei)∇Sic(ei)TW~ic-12R-1bi(ei)∇εic(ei))]}+n1n2{eiFT(WiFTκ(ΛiFTei)+εiF(ei)+bi(ei)u^mix)+Ξ(eiTQeei+umixTRumix)}≤-(Lim2-n12LiM2)∥W~ic∥2+12n1δhi2-(n1Siam2-34n1n2SiaM2)∥W~ia∥2+n14n2∥R-1∥2∥bi(ei)∥2∥∇SicM2∥∥W~ic∥2+n12n2(εia(ei)+R-1bi(ei)∇εic(ei)2)T·(εia(ei)+R-1bi(ei)∇εic(ei)2)+n1n2∥bi(ei)∥2εia(ei)Tεia(ei)+n1n2SiaM2∥bi(ei)∥2∥W~ia∥2+n1n2Ξλmin(Qe)∥eiF∥2+n1n2(∥bi(ei)∥2-Ξλmin(R))∥umix∥2.
If the following inequalities can satisfy
(73)λmin{Qe}∥eiF∥22≤eiFTQeeiF≤λmax{Qe}∥eiF∥22λmin{R}∥umix∥22≤umixTRumix≤λmax{R}∥umix∥22,Ξ>∥bi(ei)∥2λmin{R},
then, V˙iu(ei,umix) can be further transformed as
(74)V˙iu(ei,umix)≤-(Lim2-n12LiM2-n14n2∥R-1∥2∥bi(ei)∥2∇SicM2)∥W~ic∥2-(n1Siam2-34n1n2SiaM2-n1n2SiaM2∥bi(ei)∥2)∥W~ia∥2-n1n2(∥bi(ei)∥2+Ξλmin(R))∥umix∥2-n1n2Ξλmin(Qe)∥eiF∥2≤-n1n2Ξλmin(Qe)∥eiF∥2.
Therefore, we can get the conclusion that V˙iu(ei,umix)<0.


## 4. Simulations

In order to verify the validity and convergence of the proposed decentralized reinforcement learning robust optimal tracking control method based on ACI and to study the convergence of the error by comparing the simulation result, in this paper, two different configurations of the time varying external constrained reconfigurable modular robot have been applied, shown in Figures3 and 4.Figure 3
Configuration A for simulation.Figure 4
Configuration B for simulation.For the sake of the facilitation of the analysis of the configurations above, we can transform them into a form of analytic charts, which are shown in Figures5 and 6, where L1, L2, and L4 are the length of the links. L3 is the distance between the time varying constraint joint and the base modular.Figure 5
The analytic chart of configuration A.Figure 6
The analytic chart of configuration B.The time varying constraint can be defined as a kind of column which rotated about with a certain degree of freedom. The constraint equations of configuration A and configuration B are shown as follows:(75)ΨA(q,t)=L1cosq1+L2cosq2-[L3+L4cotα(t)]ΨB(q,t)=L1+L2cosq2-[L3+L4cotα(t)].
In the equation above, the angle α(t) between the time varying constraint and the X-axis can be defined as follows:
(76)α(t)=0.75π+0.2sint2.The initial positions of joint models areq1(0)=2, q2(0)=2 in configuration A and q1(0)=2, q2(0)=2 in configuration B. The initial velocities of joints are zeros. The dynamic model of configurations A and B is designed as follows:
(77)MA(q)=[0.36cos(q2)+0.60660.18cos(q2)+0.12330.18cos(q2)+0.12330.1233]MB(q)=[0.17-0.1166cos2(q2)-0.06cos(q2)-0.06cos(q2)0.1233]CA(q,q˙)=[-0.36sin(q2)q˙2-0.18sin(q2)q˙20.18sin(q2)(q˙1-q˙2)0.18sin(q2)q˙1]CB(q,q˙)=[0.1166sin(2q2)q˙20.06sin(q2)q˙20.06sin(q2)q˙20]GA(q)=[-5.88sin(q1+q2)-17.64sin(q1)-5.88sin(q1+q2)]GB(q)=[0-5.88cos(q2)]FA(q,q˙)=[q˙1+10sin(3q1)+2sgn(q˙1)1.2q˙2+5sin(2q2)+sgn(q˙2)]FB(q,q˙)=[01.5q˙2+sin(q2)+1.2sgn(q˙2)].
The desired trajectory of configurations A and B is shown asConfiguration A:(78)y1d=0.5cos(t)+0.2sin(3t)y2d=Θ(y1d,t)=arcsin[L1sin(α(t)-y1d)-L3sin(α(t))L2]+α(t),Configuration B:(79)y1d=0y2d=Θ(y1d,t)=arcsin[L1sin(α(t))-L3sin(α(t))L2]+α(t).Due to the limit of dynamic external constraints, the variety of joint 1 in configuration B is zero.In order to confirm that the adopted method can be applied in different configurations and to verify the tracking performance for the subsystem desired trajectory by using the ACI and RBF-NN based decentralized reinforcement learning robust optimal tracking control method, in this part, the comparative simulations which include the classic RBF neural network control method and ACI based decentralized reinforcement learning robust optimal tracking control method have been adopted, respectively.From Figures7, 8, 9, 10, 11, 12, 13, and 14, the joint tracking and error curves are shown by using classic RBF neural network [4] to compensate the effect of the dynamic nonlinear term and the interconnection term in the subsystem. Figures 7 to 10 show that the actual output trajectories take about 2 seconds to track the desired trajectories. This is due to the fact that the classical neural network method requires a longer training process and parameter adaptive process. The error curves in Figures 11–14 show that the joint subsystem constraints cannot be well compensated by adopting the classical neural control method for the reconfigurable modular robot when the time varying constrains exist. When the joint output variables turn larger, the reconfigurable modular system cannot exhibit a good robustness, and the tracking errors are larger than before.Figure 7
Trajectory tracking curve of configuration A joint 1 with RBF neural network.Figure 8
Trajectory tracking curve of configuration A joint 2 with RBF neural network.Figure 9
Trajectory tracking curve of configuration B joint 1 with RBF neural network.Figure 10
Trajectory tracking curve of configuration B joint 2 with RBF neural network.Figure 11
Tracking error curve of configuration A joint 1 with RBF neural network.Figure 12
Tracking error curve of configuration A joint 2 with RBF neural network.Figure 13
Tracking error curve of configuration B joint 1 with RBF neural network.Figure 14
Tracking error curve of configuration B joint 2 with RBF neural network.Figures15, 16, 17, 18, 19, 20, 21, and 22 showed the joint tracking and error curves by adopting ACI to identify the optimal Q-function, optimal control policy, and global dynamic nonlinear terms of the HJB equation in the subsystem. Figures 15 to 18 show that the actual output trajectories of the subsystems can track the desired trajectories in 0.5 seconds by using the proposed robust reinforcement learning optimal control method. This is due to the excellent identifying ability of ACI which can identify the uncertainties contained in the subsystems in a short time. The error curves in Figures 19–22 show that the tracking error is very small and it can converge to zero in a short time. Besides, when the proposed decentralized control method is adopted for the joint subsystems, the robustness is manifested.Figure 15
Trajectory tracking curve of configuration A joint 1 with ACI based reinforcement learning.Figure 16
Trajectory tracking curve of configuration A joint 2 with ACI based reinforcement learning.Figure 17
Trajectory tracking curve of configuration B joint 1 with ACI based reinforcement learning.Figure 18
Trajectory tracking curve of configuration B joint 2 with ACI based reinforcement learning.Figure 19
Tracking error curve of configuration A joint 1 with ACI based reinforcement learning.Figure 20
Tracking error curve of configuration A joint 2 with ACI based reinforcement learning.Figure 21
Tracking error curve of configuration B joint 1 with ACI based reinforcement learning.Figure 22
Tracking error curve of configuration B joint 2 with ACI based reinforcement learning.Figures23 and 24 showed the 3D-tip trajectory curves by using ACI algorithm. These two figures show that the proposed decentralized control method can fully satisfy the accessibility requirement of the reconfigurable modular robot. Besides, the singular displacements of the joint subsystems and the end-effector would not appear. The parameters defined in ACI are shown in Table 1.Table 1
Parameter list of action-critic-identifier.
k
α
υ
η
a
1
η
a
2
η
c
β
1
β
2
γ
800
300
0.005
10
50
20
0.2
2
0.5Figure 23
3D-tip trajectory curve of configuration A with ACI.Figure 24
3D-tip trajectory curve of configuration B with ACI.The simulation results show that, compared with the situation of using classic RBF neural network controller, the decentralized robust optimal tracking control method based on ACI and reinforcement learning can be applied into different configurations of the time varying constrained reconfigurable modular robot. And the joint variables can track the desired trajectory within a very short time in different configurations, and the fluctuation of the error convergence range is minimal.


## 5. Conclusions and Future Work

In this paper, combining ACI with RBF neural network, a novel decentralized reinforcement learning robust optimal tracking control theory has been proposed for time varying constrained reconfigurable modular robots. Moreover, this theory is used to solve the problem of the continuous time nonlinear optimal control policy for strongly coupled uncertainty robotic system. Firstly, we build the model of subsystem with the time varying external constraints and described the global robot system as a synthesis of interconnected subsystem. Secondly, ACI is used to estimate the HJB equation and the global uncertainty, where a continuous-time optimalQ-function is adopted to take the place of traditional optimal value function, the optimal Q-function and the optimal control police are approximated by critic-NN and action-NN, and the global uncertainty is identified by the identifier. Thirdly, we design a novel decentralized robust optimal tracking controller, so the desired trajectory can be tracked, and the tracking error could converge to zero in finite time. On this basis, two kinds of Lyapunov functions are designed to confirm the stability of ACI and the subsystem. Finally, in order to confirm the superiority of the proposed control theory, the comparative simulation examples have been presented combining two different configurations of the robot.In the future, more complex configuration for time varying external constrained reconfigurable modular robot can be included. Therefore, the decentralized controller with higher control and error precision will be considered.



---

*Source: 387817-2013-12-19.xml*