# Adaptive Reinforcement Learning-Enhanced Motion/Force Control Strategy for Multirobot Systems

Authors: Phuong Nam Dao; Duy Khanh Do; Dinh Khue Nguyen
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

This paper presents an adaptive reinforcement learning- (ARL-) based motion/force tracking control scheme consisting of the optimal motion dynamic control law and force control scheme for multimanipulator systems. Specifically, a new additional term and appropriate state vector are employed in designing the ARL technique for time-varying dynamical systems with online actor/critic algorithm to be established by minimizing the squared Bellman error. Additionally, the force control law is designed after obtaining the computation of constraint force coefficient by the Moore–Penrose pseudo-inverse matrix. The tracking effectiveness of the ARL-based optimal control is verified in the closed-loop system by theoretical analysis. Finally, simulation studies are conducted on a system of three manipulators to validate the physical realization of the proposed optimal tracking control design.


---

## Body

## 1. Introduction

In recent years, the tracking control problem for robotic systems has been remarkable, considered from not only academia but also industrial automation [1, 2]. Among different schemes, robotic systems can mainly be separated into two categories: (1) mobile robotics [3, 4] and (2) robot manipulators [5]. However, because of the growing requirements for high productivity, the networked robotic systems that are known as the collaboration of many robots can fulfill different tasks by mobile manipulators [6], tractor trailers [3], bilateral teleoperators [5], and cooperating mobile manipulators (CMMs) [7]. It is worth emphasizing that, unlike nonlinear multiagent systems [8, 9] studying the relation between agents, CMMs consider the interaction with the rigid object. In the literature of CMMs’ control objectives, they can be classified into two categories, i.e., multiple mobile robot manipulators in cooperation carrying a common object with unknown parameters and disturbances [7, 10–18]; one of them tightly holds the object by the end effector, and the remaining mobile manipulators’ end effector follows a trajectory on the surface of the object [19, 20]. Most of the existing control literature of networked robotics mainly focuses on the implementation of nonlinear controllers. According to Lyapunov’s direct technique, authors in [5] presented the robust controller for a bilateral teleoperation under time-varying delays and without relative motion. Recently, the backstepping technique of networked robotics has achieved much attention [3, 11, 21]. In [3], a backstepping method is utilized to implement an adaptive trajectory tracking control design for a tractor trailer, where the proposed control scheme can stabilize for cascade control systems. For the purpose of easily computing the backstepping method and handling the full-state constraint, authors in [21, 22] pointed out the appropriate term α in the asymmetric barrier Lyapunov function (ABLF). This backstepping method is extended in [11] to deal with unknown dead zone nonlinearity, time-varying term using the input-driven filter term and Nussbaum function, respectively. Another consideration addressed in the existing literature is to mention the holonomic-nonholonomic constraint [12, 19, 20, 23, 24]. Motivated by this approach, the corresponding motion dynamic model and constraint force equation are established for finding appropriate controllers [12, 19, 20, 23, 24]. As a result, the robust adaptive controller was realized by estimating the derivative of the Lyapunov function and combining the constraint force control to obtain the motion/force control scheme for multimanipulator systems [12]. Due to dynamic uncertainties and external disturbance, fuzzy and neural network (NN) method are popularly considered to estimate dynamic uncertainties for guaranteeing system stability [7, 10, 20]. However, the work in [23] utilized an estimation in the absence of fuzzy and NNs for designing a decentralized controller for cooperating manipulators. In contrast to this technique, authors in [7, 12, 25] proposed the centralized control law based on the consideration of dynamics of m manipulators. The scheme in [7] extends the neural network-based centralized coordination control to achieve the tracking of each mobile manipulator by the consideration of additional actuators using servo motors. However, these nonlinear control solutions do not take into account the challenges, such as input-state saturation, and control objectives are different from the traditional tracking control problem. Therefore, we propose motion/force control with a progressive optimality principle to improve the tracking performance.In the implementation of the optimality principle, an important aspect is developed for robotic systems with the consideration of the robot-environment interaction [26–28]. In control design for robotic systems, and so on, there always exists an interaction between the end-effector and the environment, which may lead to various challenges of system performance. Therefore, the control scheme should maintain a solution for considering the system interactions. Needless to say, the control systems of the robot-environment interaction have been paid much attention by many researchers for several decades [28]. Due to the effect of the interaction between the robot’s end effector and the unknown environment, the desired trajectory in the joint space needs to be modified by the optimality approach [26–28]. In [26], the method to generate the virtual desired trajectory can be implemented by adaptive output feedback optimal control for the unknown impedance model. A different approach for finding the virtual reference trajectory can be developed by Q-learning [27]. Furthermore, based on ARL-based admittance control, authors in [28] proposed the completed control structure with the additional term of torque estimation and adaptive controller. An extension of the robot-environment interaction can be known as the physical human-robot interaction [15, 29]. In [29], the dynamic model of an n−dimensional manipulator system is established under the task space with the consideration of the end effector of the robot and human or environment. Moreover, the dynamic model established more a desired admittance model in the task space for obtaining virtual desired trajectory under constraint region by a soft saturation function [29]. A different discussion of the human-robot interaction with the estimation of human intention was mentioned in [15]. Furthermore, the motion/force control law was proposed using the computation of constraint force without the optimality problem [15]. An approach of optimization for the dual-arm manipulator with the consideration of relative Jacobian was mentioned in [16] by the QP solver. The extension from dual-arm manipulators to multiarm robots is discussed with the classical backstepping structure and optimal control law being constructed to tackle optimal behaviors involving obstacle avoidance and energy saving [11]. Additionally, authors in [17] studied multiarm systems under time-varying communication delay by task-space synchronization with a fixed threshold. For different networked systems of bilateral teleoperation systems, the optimality principle was also considered for implementing control design [30]. However, the disadvantage of the control law in [30] is no consideration in solving the Riccati equation. Furthermore, the optimality principle in robotic systems has been almost focused on the interaction of the human-robot’s end effector and the environment. This article proposes the consideration of optimal control in trajectory tracking in the motion/force control problem.For finding an optimal control problem to minimize a given performance index in dynamical systems, it is necessary to solve the so-called Hamilton–Jacobi–Bellman (HJB) equation obtained from the dynamic principle. However, it is hard to analytically solve this HJB equation due to leading to a nonlinear partial differential equation. Among the numerical methods which have been considered to solve the HJB equation, the remarkable iterative structure has been developed to solve via online based on the reinforcement learning (RL) principle being inspired by machine learning [27, 31–39]. In order to implement the numerical algorithm for solving the HJB equation, there are two major directions, including online actor/critic [31, 32] and off-policy technique using integral reinforcement learning (IRL) [38, 39]. The first approach using online actor/critic is to develop based on the property of Hamiltonian Hx,u∗,∂V∗/∂x being 0. As a result, the iterative algorithm can be considered by the Bellman error with solving the optimization problem. For designing the ARL scheme in linear dynamical systems [31] and in nonlinear systems [32, 40], the methods are realized to find the optimal control input being Kronecker product and approximating neural networks (NNs), respectively. Furthermore, this technique is extended for several situations, such as goal representation heuristic dynamic programming (GRHDP) with the multivariable tracking scheme [35] and uncertain discrete-time systems by using NNs [33]. The second approach using the off-policy technique is implemented by considering the deviation of the performance index in two different times with appropriate data collection [38, 39]. It is worth emphasizing that almost existing works discussed the ARL design in time-invariant systems [27, 33, 35, 36, 38, 39]. In contrast, for establishing the optimal control problem in robotic systems, it requires the consideration of ARL in time-varying systems because of the desired time-varying trajectory [31, 32]. Furthermore, the ARL-based optimal control with the actor/critic learning structure has just been discussed in [41] for time-invariant systems. The key idea is to establish simultaneously the learning of neural networks (NNs) in the actor and critic part [41]. The work in [32] developed the actor/critic learning structure for manipulators by combining the sliding mode control (SMC) technique in the absence of constraint force. Regarding the multimanipulator systems, the proposed control law in [12] only implemented the robust adaptive control based on the traditional nonlinear control technique and model separation technique without considering the optimality principle.In this paper, we investigate the ARL-based motion/force tracking control problem for a multimanipulator system in the presence of disturbance. The control goal is to obtain the unification of the optimality principle and tracking problem. The main contributions of this paper can be summarized as follows:(1)
First, it is obviously different from [26, 27, 34, 41, 42] studying optimal control for time-invariant systems; we propose ARL-based optimal control design in the situation of the time-varying nonlinear dynamical system under the influence of the time-varying desired trajectory by utilizing the online actor/critic technique for the motion dynamic model of multimanipulator systems. Furthermore, this work is still proposed for the motion/force control problem by extending more the force control term in the control scheme.(2)
Second, different from numerous existing nonlinear control approaches [10, 12, 13, 17, 20, 23–25, 43, 44], the proposed optimal control algorithm is investigated with the unification of the optimization and tracking problem to be discussed through theoretical analysis and simulation studies.The rest of this article is organized as follows. Section2 provides the problem statement of this paper. Section 3 describes the ARL-based motion/force control design for multimanipulator systems with tracking analysis. Simulation studies on the multirobots with 3 manipulators are presented in Section 4, and the conclusion remarks are pointed out in Section 5.


## 2. Preliminaries and Problem Statement

We investigaten two-link manipulators, as shown in Figure 1, which includes n robots interacted by each pair of manipulators, the unknown environment with the original Cartesian coordinate system OX,Y,Z, and the corresponding coordinate Ei of the end effector of each robot. The interaction between each pair of manipulators leads to a constraint condition as described in the following equation:(1)Ξij=OEi−OEj−dij=0∀1≤i<j≤n,where OEi is a vector with the first point being the original point and the end effector Ei under the Cartesian coordinate system and dij is the distance between end effectors of the ith robot manipulator and the jth robot manipulator. According to (1), combining all the constraints in this multimanipulator, we can obtain the constraint vector as follows:(2)Ξη=…,Ξijηi,ηj,…T=0,where Ξη∈ℝm×1, η is the joint vector of n two-link manipulator systems. It can be seen that there are two main cases in general robots. First, there exist no constraint forces due to no interaction between the robot and the rigid object. Second, the interaction leads to the existence of constraint force, and we only consider the constraint motion under the following assumption.Figure 1
The cooperating manipulator system.Assumption 1.
The number of joints is more than the number of constraint conditions, i.e.,2n>m.
In [12], the dynamic equation of n two-link robot manipulators with constraint force by the interaction between each couple of manipulators can be described as(3)Mηη¨+Cη,η˙η˙+Fη,η˙,t=τ−JTηλ,where Jη=∂Ξ/∂η∈ℝm×2nandλ∈ℝm×1 is the vector of Lagrange multiplier.
The control objective is to obtain not only the tracking of joints to desired trajectories but also guaranteeing the constant distance between end effectors of each pair of manipulators. Moreover, the motion/force control objective also mentions the tracking of Lagrangian coefficientλ in (3). On the contrary, due to the proposed optimality-based motion/force control design, it requires the unification of the optimal control and tracking problem.Remark 1.
It is worth emphasizing that different from classical nonlinear control approaches [7, 17, 19, 20, 23, 24, 43], the approach in this paper is developed by the optimal control technique for uncertain systems based on adaptive/approximate reinforcement learning (ARL). Moreover, regarding the mathematical model of multimanipulators, this work is the extension of Chen et al. [11] by considering the holonomic constraint force with separation of the joint vector.


## 3. Motion/Force Control Design with ARL

In this paper, a completed control structure in Figure2 is investigated with 2 independent parts, including ARL-based optimal control for the motion dynamic model and the force control scheme for tracking the constraint coefficient. Moreover, the convergence and stability of the whole system (Figure 2) are discussed for determining the performance of the proposed motion/force controller.Figure 2
Control structure of the surface vessel.


### 3.1. ARL-Based Motion Controller

For the purpose of designing the motion controller in multimanipulator systems, the motion dynamic model is discussed after eliminating the constraint forces. Moreover, it is due to the description of time-varying systems in trajectory tracking control; the modification of the tracking error model needs to be implemented for developing the ARL-based optimal control design.Because ofΞη∈ℝm×1 with mm<2n independent constraint conditions, there exists an algebraic map φ:ℝ2n−m↦ℝ2n satisfying(4)η=φξ,where ξ∈ℝ2n−m×1 is the vector of independent joint variables. Taking the derivative of (4), we imply that(5)η˙=Λξ˙,η¨=Λξ¨+Λ˙ξ˙,where Λ=∂φ/∂ξ∈ℝ2n×2n−m. With the purpose of eliminating the constraint force in the motion dynamic model, we use the Moore–Penrose pseudo-inverse matrix of Jη being J+=JTJJT−1 to achieve the matrix Q=I−J+J satisfying QJT=0. Multiplying both sides of (3) with matrix Q, it leads to the following equation after utilizing QJT=0:(6)QMη¨+QCη˙+QF=Qτ−JTλ=Qτ−QJTλ=Qτ=τ1.According to (5) and (6), the motion dynamic can be expressed as(7)QMΛξ¨+QMΛ˙+QCΛξ¨+QF=τ1.Regarding the independent joint coordinates, the dynamic model can be represented after multiplying with matrixI2n−m,0∈ℝ2n−m×2n on both sides of (6) as follows:(8)ξ˙=υ,M¯¯υ˙+C¯¯υ+F¯¯=τind1,where(9)M¯¯=I2n−m,0QMΛ,C¯¯=I2n−m,0QMΛ˙+QCΛ,F¯¯=I2n−m,0QF,τind1=I2n−m,0τ1.Remark 2.
This paper only discusses the situation that the number of constraint conditions is smaller than the number of joints2n>m as mentioned in Assumption 1. For the case of 2n=m, it leads to no existence of independent joints. As a result, the above modifications are not implemented, and the control method is developed with different approaches [45]. Additionally, no constraint forces are considered in the special case m=0.
The motion control design is proposed as(10)τind1=τd+u,where(11)τd=M¯¯υ˙d+C¯¯υd+F¯¯,υd=ξ˙d−βzξ,zξ=ξ−ξd,zυ=υ−υd.
Therefore, with the assumption thatd/dtξd is the function of ξd as d/dtξd=h1ξd, we can obtain that(12)ddtzυ=ddtυ−υd=M¯¯−1u−M¯¯−1C¯¯υ−υd,ddtzξ=υ−ddtξd,ddtξd=h1ξd.
Furthermore,υd is the function of zξ,ξd as υd=ξ˙d−βzξ=h1ξd−βzξ⇒υ=zυ+υd is the function of zυ,zξ,ξd. Combining with (12), a continuous-time affine system has been obtained with the typical form(13)Z˙=FZ+GZu,where the vector of state variables is Z=zυ,zξ,ξdT∈ℝ12.Remark 3.
Due to the time-varying reference in trajectory tracking, the closed system needs to be known as the time-varying property. However, the framework of using term (11) and choosing the state variable vector Z is able to obtain time-invariant systems (12) and (13). Therefore, in contrast to the existing methods in [36], the proposed control easily handles the situation of time-varying systems by the elimination of the term −∂V∗/∂t.


#### 3.1.1. The Optimal Control Problem

In this section, we consider the general class of time-varying systems:(14)ddtZ=FZ,u,t,with the associated cost function to be defined as(15)JZ,u=∫0∞rZτ,uτdt,where rZτ,uτ=ZTQZ+uTRu under the weighting matrices Q=QT≥0 and R=RT>0 to be defined with appropriate dimensions. It can be seen that the optimal problem for the motion model of multimanipulators (12) and (13) is to realize an admissible control policy [31, 32] for the purpose of obtaining the minimal value. This is because that the control policy u^Z needs to guarantee the existence of the estimated Bellman function V^Z obtained from the HJB equation.Definition 1.
A control lawuZ is considered being “admissible” with respect to the performance index (15) on a compact set Ω, which is known as a continuous signal uZ∈ΥΩ, if the properties of the stabilization of (14) and the limitation of JZ,u for every Z∈Ω are satisfied.
It is worth emphasizing that, in the general case of time-varying system (14), the optimal control strategy needs to be considered as a time-varying function u∗Z,t. Hence, the time-varying Bellman function with respect to arbitrary time V∗Zt,t can be given as(16)V∗Zt=minuZ∈ΥΩJZt,ut.
The HJB equation can be achieved by taking the time derivative ofV∗Xt under two different methods. Firstly, it is directly computed along system (14) as(17)ddtV∗Zt,t=∂V∗∂ZdZdt+∂V∗∂t=∂V∗∂ZFZ,u∗,t+∂V∗∂t.
Secondly, thanks to the dynamic programming principle, we also compute the derivative ofV∗Zt with respect to time as(18)ddtV∗Zt,t=−rZτ,u∗τ.
According to (17) and (18), the first part of the HJB problem can be given:(19)rZτ,u∗τ+∂V∗∂ZFZ,u∗,t=−∂V∗∂t.
Moreover, the second part of the HJB problem can be developed with the performance index to be formulated as(20)V∗Zt,t=minuZ∈ΥΩ∫t∞rZτ,u∗τdτ,(21)V∗Zt,t=minu∗Z∈ΥΩ∫tt+ΔtrZ,u∗dτ+minu∗Z∈ΥΩ∫t+Δt∞rZ,u∗dτ.
Because of utilizing the dynamic programming principle, the performance index is represented as(22)V∗Zt,t=minuZ∈ΥΩ∫tt+ΔtrZτ,uτdτ+V∗Zt+Δt,t+Δt.
Therefore, we imply that the Bellman function can be rewritten as(23)0=minuZ∈ΥΩ1Δt∫tt+ΔtrZ,u∗dτ+V∗Zt+Δt,t+Δt−V∗Zt,tΔt.
As the convergence ofΔt⟶0+, one can derive that(24)minuZ∈ΥΩrZt,u∗t+∂V∗∂ZFZ,u∗,t=−∂V∗∂t.Remark 4.
It is worth noting here that the term∂/∂tV∗Zt,t in (19) and (24) is the disadvantage in developing the ARL algorithm, and there has been very little research considering the direct method for this challenge. Authors in [36] developed the solutions of the ARL-based time-varying optimal controller using the framework of the data-driven and Newton–Leibniz formula-based function approximation method. In this paper, we indirectly handle the time-varying tracking error model through the consideration of systems (12) and (13). Therefore, the two parts of the HJB problem can be modified as(25)rZ,u∗+∂V∗∂ZFZ+GZu∗=0,(26)minuX∈ΥΩrX,u+∂V∗∂XFX+GXu=0.


#### 3.1.2. ARL-Based Control Design for Independent Joints

It is because of difficulties in directly solving HJB equation (25) to obtain corresponding optimal control algorithm. Hence, the approximate solution with a neural network (NN) is mentioned to develop the ARL design in the motion dynamic model of multimanipulators. Thanks to the smooth property of functions V∗Z and u∗Z, they can be described over any compact domain C⊆ℝ12:(27)V∗Z=WTψZ+ϵZ,(28)u∗Z=−12R−1GTZ∂ψ∂ZTW+∂εZ∂ZT,where W∈ℝN is an uncertain ideal NN weight vector, N is chosen as the number of neurons, ψZ∈ℝN denotes an activation function vector satisfying ψj0=0 and ∂ψj/∂Z|Z=0=0,∀j=1,…,N, and ϵZ∈ℝ is known as the estimation error of the Bellman function V∗Z. Due to the unknown ideal NN weight vector W∈ℝN, it is necessary to obtain the corresponding adjusting mechanism W^a,W^c for estimating the actor/critic NNs to achieve the optimal control design in the absence of finding the HJB equation analytically (for more details, see [41]). Moreover, it is able to choose the smooth NN activation function vector ψZ∈ℝN based on the description of multimanipulators (see Section 4). In [41], thanks to the Weierstrass approximation theorem, we can uniformly approximate not only V∗Z but also ∂V∗Z/∂Z with ϵZ,∂ϵZ/∂Z⟶0 as N⟶∞. For a fixed number N, the approximated Bellman function of critic NN V^Z and the approximated optimal control policy of actor NN u^X are represented as(29)V^Z=W^cTψZ,(30)u^Z=−12R−1GTZ∂ψ∂ZTW^a.It should be noted that properties (19) and (24) of Hamiltonian HZ,u,∂V/∂Z=rZτ,uτ+∂V/∂ZFZ+GZu with the optimal controller u∗Z and corresponding value function V∗Z guarantee that the adjusting mechanisms of actor NN W^a and critic W^c weights are simultaneously tuned to minimize the squared Bellman error δhjb and the corresponding integral, respectively. According to the property of HJB problem (25), it leads to H∗Z,u∗,∂V∗/∂Z=0. After obtaining the error between approximated functions V^Z,u^Z and optimal control input V∗Z, u∗Z, the Bellman error δhjb is represented as(31)δhjb=H^Z,u^,∂V^∂Z−H∗Z,u∗,∂V∗∂Z=H^Z,u^,∂V^∂Z=W^cTσZ,u^+ZTQTZ+u^TRu^,where σZ,u^=∂ψ/∂ZFZ+GZu^ is the regression vector of the critic part.Based on the results in [31], the adjusting mechanism of the critic NN is computed:(32)ddtW^c=−kcλσ1+νσTλσδhjb,where ν,kc∈ℝ are constant positive coefficients and λt∈ℝN×N is an estimated symmetric matrix being the solution of the differential equation as(33)ddtλ=−kcλσσT1+νσTλσλ,λts+=λ0=φ0I,where ts+ denotes the resetting time satisfying that λminλt≤φ1,φ0>φ1>0. In [31], because of the condition that λt is positive definite and preventing the covariance wind-up problem, the covariance matrix λt is chosen as(34)φ1I≤λt≤φ0I.Additionally, the adjusting mechanism of the actor NN part is represented by minimizing the squared Bellman error:(35)ddtW^a=−ka11+σTσ∂ψ∂ZGR−1GT∂ψT∂ZW^a−W^cδhjb−ka2W^a−W^c.Remark 5.
ARL technique has been investigated with many different approaches, such as off-policy integral reinforcement learning (IRL) [46, 47] and Q-learning [42]. However, there is a wide difference between the online actor/critic technique in this paper and IRL and Q-learning. The off-policy IRL requires the complete data collection between the two sequential sampling times in computing the optimal law [46, 47]. Meanwhile, it is obviously different from the off-policy IRL technique; the online actor/critic method in (29), (30), (35), (33) requires the initial values to implement the computation. This could be exemplified by this article design control law under HJB problems (25) and (26), and the off-policy IRL method considers the relation between the deviation of integrals at two sampling times and the collected data under the dynamic programming law. In addition, the Q-learning method mentioned the Q-function in terms of not only state variables but also control inputs, being different from the Bellman function [42]. This property can result in the difficulties for implementing the Q-learning technique of the continuous-time system. Additionally, compared with the data-driven method in [48], this work provides a neural network-based technique to avoid the Kronecker product in estimating the actor/critic term. The actor/critic-based approaches have been discussed in [49, 50] for nonlinear affine systems using residual error δhjb. However, in view of the consideration of the identifier in [49, 50], it implies the difference in the computation of residual error δhjb and training laws in actor/critic weights between the proposed method and the work in [49, 50].


#### 3.1.3. The Control Design of Dependent Joints

After completing the tracking control design of independent joints, we can achieve the remaining dependent joints’ controller based on the relation between two groups of joint variables, including independent joints and dependent joints. According to (8), it leads to the following representation:(36)ξ¨=M¯¯−1τind1−C¯¯ξ−F¯¯.On the contrary, thanks to the relationτ1=τind1T,τdep1TT, multiplying both sides of (6) with 0,Im∈ℝm×2n, we achieve the dependent joints to be computed as(37)M″ξ¨+C″ξ˙+F″=τdep1,where(38)M″=0,ImQMΛ,C″=0,ImQMΛ˙+QCΛ,F″=0,ImQF,τdep1=0,Imτ1.Substituting (36) into (37), we obtain the control law of dependent joint variables to be implemented as follows:(39)τdep1=M″M¯¯τind1−C¯¯ξ−F¯¯+C″ξ˙+F″.It is worth noting that the problem of synchronization between multiple joints can be considered by the relation between dependent and independent joints. Therefore, this problem is able to obtain the desired trajectories of independent joints and the trajectories of remaining joints depending on these independent joints. Furthermore, it leads to the relationship between the control input of independent joints and dependent joints (39).


### 3.2. Constraint Force Control Design

For the purpose of designing the motion/force control scheme forn manipulators, the constraint force controller and motion control law are necessary to be established based on the computation of lambda coefficient in (3). Before demonstrating the motion/force control design, we need to compute the coefficient λ to be dependent on control input τ as follows. According to (3)–(5), we achieve the following equation:(40)J∗Tξλ=τ−M∗ηξ¨+C∗η,η˙ξ˙+F∗η,η˙,t,where(41)J∗Tξ=JTφξ,M∗η=MηΛ,C∗η,ξ˙=Cη,η˙Λ+MηΛ˙,F∗η,η˙,t=Fη,η˙,t.Moreover, we find the method to compute coefficientλ after eliminating the term ξ by multiplying both sides with matrix N satisfying that(42)NMηΛ=0,NJ∗Tξ=I.Therefore, we can obtain the following coefficientλ:(43)λ=Nτ−NC∗η,η˙ξ˙+NF∗η,η˙,t,with N to be chosen as(44)N=AJ∗Tξ−1A,where A=J∗ξM−1η.Based on the proposed motion/force, control design is described in the following equation:(45)τ=τind1τdep1−JTητ2.This leads to the following relations that(46)Qτ1=Qτind1τdep1=τ1,(47)λ=Nτ1−τ2−NC∗η,η˙ξ˙+NF∗η,η˙,t.It should be noted that the constraint force coefficientλ depends on the ARL-based motion dynamic control τ1=τind1τdep1 to be described in (47). Therefore, with the purpose of tracking the coefficient λ, we can propose the remaining term τ2 in (47) as given in the following:(48)τ2=Kfeλ−λd−NC∗η,η¨ξ˙d.The tracking problem will be proved in the next section.


### 3.3. Convergence and Stability Analysis

With the purpose of considering the stability and convergence of the closed system under the proposed motion-force control design, we need to implement several following steps. First, the Bellman errorδhjb, critic estimated weight error W˜c=W−W^c, and the persistence of excitation (PE) condition need to be given. Second, thanks to the optimal Bellman function and PE condition, the corresponding Lyapunov function candidate is chosen. Third, the stability of the whole system (Figure 1) and the role of constraint force control and dependent joints’ motion control are discussed.According to (27)–(31), the Bellman error function δhjb is dependent on state variable vector Zt as(49)δhjb=W^cTσZ,u^+ZTQTZ+u^TRu^−WTσZ,u∗−u∗TRu∗−∂ϵ∂ZFZ+GZu∗−ZTQTZ=−W˜cTσZ,u^+14W˜aT∂ψ∂ZGR−1GT∂ψ∂ZTW˜a−14∂ϵ∂ZGR−1GT∂ϵ∂ZT−∂ϵ∂ZFZ+GZu∗.Combining (49) with (32), we imply the dynamic equation of critic error W˜c=W−W^c to be given as(50)ddtW˜c=−kcλΠΠTW˜c+kcλσZ,u^1+νσTλσ14W˜aT∂ψ∂ZGR−1GT∂ψ∂ZTW˜a−14∂ϵ∂ZGR−1GT∂ϵ∂ZT−∂ϵ∂ZFZ+GZu∗,where W˜a=W−W^a and Πt=σZ,u^/1+νσZ,u^TλσZ,u^ is satisfied by(51)Π≤1νφ1.The nominal system is given after eliminating the term of NN errors:(52)ddtW˜c=−kcλΠΠTW˜c.Due to the purpose of exponential convergence ofW˜c, it is necessary to guarantee the PE condition as follows:(53)μ2I≥∫t0t0+δΠsΠsTds≥μ1I,∀t0≥0.The following main theorem is proposed with the assumptions being discussed in [41] to estimate the attraction region of tracking errors.Theorem 1.
Consider the cooperating manipulators in Figure1 with the assumption of the bound conditions of the NN including ideal weights, activation function ψ., its derivative to be described in [41], and PE condition of the signal vector Πt, and the selection of parameters is implemented with the following condition:(54)c3ka1>k1k2,where the coefficients in (54) are mentioned in (55), (56), and (35). Let us consider the control structure (Figure 2), (10) with ARL-based control scheme (46), (10), (39), and (30), the associated adjusting mechanisms (35) and (33) for the actual controller, and constraint force control vector (48); then, (1) the actor-critic weight errors W˜a and W˜c are UUB; (2) the tracking effectiveness of not only z˜v=v−vd but also z˜η=ξ−ξd in the closed control system (Figure 1) is also UUB; (3) the tracking of constraint force coefficient vector λ and the remaining terms of joint variables’ vector η2n=ξ2n−m,pm is also UUB.Proof.
First, the tracking effectiveness of motion tracking control is considered under the ARL-based optimal control scheme for corresponding tracking error motion models (12) and (13). Due to the satisfaction of PE condition (53), there exists a time-varying function Vc:ℝN×0,∞⟶ℝ satisfying several following inequalities:(55)c1W˜c2≤VcW˜c,t≤c2W˜c2,∂Vc∂t+∂Vc∂W˜c−kcλΥΥTW˜c≤−c3W˜c2,∂Vc∂W˜c≤c4W˜c,where c1,c2,c3,c4∈ℝ are positive constant coefficients. The function VcW˜c,t is one part of the Lyapunov function candidate. Moreover, according to the bound conditions of the proposed NN [41], we can achieve the following results with the purpose of estimating the time derivative of the Lyapunov function in the next steps:(56)i.W˜a≤k1,i2.∂ψ∂ZGR−1GT∂ψ∂Z≤k2,i3.14W˜aT∂ψ∂ZGR−1GT∂ψ∂ZTW˜a−14∂ϵ∂ZGR−1GT∂ϵ∂ZT−∂ϵ∂ZFZ+GZu∗≤k3,i4.12WT∂ψ∂ZGR−1GT∂ϵ∂ZT+12∂ϵ∂ZGR−1GT∂ϵ∂ZT+12WT∂ψ∂ZGR−1GT∂ψ∂ZTW˜a+12∂ϵ∂ZGR−1GT∂ψ∂ZTW˜a≤k4.
Thanks to Bellman functionV∗Z (20) and time-varying VcW˜c,t (55), a Lyapunov function candidate can be chosen to determine the stability of the whole of the cascade control system (Figure 2) and the tracking of the weights of the actor NN and critic NN:(57)VL≜12ρzξTzξ+V∗Z+VcW˜c,t+12W˜aTW˜a.
It is worth noting that the term1/2ρzξTzξ (ρ is a positive constant coefficient) is the additional term for the purpose of tracking of the whole motion control system. Because V∗Z is a smooth function and positive definite, there exist two K-class functions γ1andγ2 such that(58)γ1Z≤V∗Z≤γ2Z.
Because of (55) and (8), we can determine the strict proper representation of Lyapunov function VLZ:(59)12ρzξ2+α1Z+c1W˜c2+12W˜a2≤VL≤12ρzξ2+α2Z+c2W˜c2+12W˜a2.
Implementing the derivative ofVL along motion system (13) under the control input u^Z, we imply that(60)ddtVL=ρzξTzv−βzξ+∂V∗∂ZFZ+GZu^+∂Vc∂t+∂Vc∂W˜cΘ1+Θ2−W˜aTW^˙a+∂V∗∂ZGZΔ,where(61)Θ1=−kcλΠΠTW˜cΘ2=kcλσ1+νσTλσ14W˜aT∂ψ∂ZGR−1GT∂ψ∂ZTW˜a−14∂ϵ∂ZGR−1GT∂ϵ∂ZT−∂ϵ∂ZFZ+GZu∗.
According to (19), the relation between the Bellman function and the optimal control law can be represented as(62)∂V∗∂ZFZ=−∂V∗∂ZGZu∗Z−ZTQTZ−u∗TZRu∗Z.
According to (62) and optimal law (28), (35), and (55) in (60), we can conclude that(63)ddtVL≤−ρβzξ2+ρzξTzv−zvTQzv−u∗TRu∗+2u∗TRu∗−u^−c3W˜c2+c4W˜cΘ2+ka2W˜aTW^a−W^c−2u∗TRΔ+ka11+σTσW˜aT∂ψ∂ZGZR−1GTZ∂ψ∂ZTW^a−W^cδhjb.
Using Young’s inequality, one can obtain that(64)ρzξTzv≤ρ2zξ2+ρ2zvTzv=ρ2zξ2+ρ2zv2.
From (30), (19), and (56), the following inequality is given:(65)2u∗TRu∗−u^=12WT∂ϕ∂ZGR−1GT∂ϵ∂ZT+12WT∂ϕ∂ZGR−1GT∂ϕ∂ZTW˜a+12∂ϵ∂ZGR−1GT∂ϕ∂ZTW˜a+12∂ϵ∂ZGR−1GT∂ϵ∂ZT≤k4,and the term Θ2 is bounded by(66)Θ2≤kcφ02νφ1k3.
Following (56) and (49), we have(67)ka11+σTσW˜aT∂ψ∂ZGZR−1GTZ∂ψ∂ZTW^a−W^cδhjb=ka11+σTσW˜aT∂ψ∂ZGZR−1GTZ∂ψ∂ZTW˜c−W˜a×−W˜cTσ+14W˜aT∂ψ∂ZGR−1GT∂ψ∂ZTW˜a−14∂ϵ∂ZGR−1GT∂ϵ∂ZT−∂ϵ∂ZFZ+GZu∗≤ka1k1k2W˜c2+ka1k12k2W˜c+ka1k1k2k3W˜c+ka1k12k2k3.
It is obvious that(68)ka2W˜aTW^a−W^c=ka2W˜aTW˜c−W˜a≤ka2k1W˜c−ka2W˜a2.
Moreover, it can be seen that(69)−u∗TRu∗−2u∗TRΔ≤ΔTRΔ≤λmaxRΔ¯2.
Replacing (64) and (68) in (63), we can conclude that(70)ddtVL≤−ρβ−12zξ2−zvTQ−0.5ρIzv−c3−ka1k1k2W˜c2−ka2W˜a2+ka1k12k2k3+k4+c4kcφ02νφ1k3+ka1k1k2k3+ka1k12k2+ka2k1W˜c+λmaxRΔ¯2.
Based on the estimationab≤γa2+1/4γb2, we can obtain that(71)ddtVL≤−ρβξ−12zη2−zvTQ−0.5ρIzv+λmaxRΔ¯2−1−θc3−ka1k1k2W˜c2−ka2W˜a2+ka1k12k2k3+k4+14θc3−ka1k1k2c4ηcφ02νφ1k3+ka1k1k2k3+ka2k1+ka1k12k22.
Let us select the coefficients to be satisfied such thatβ>1/2,0<ρ<2λminQ,0<θ<1,andc3>ka1k1k2.
With the state vectorz=zηT,zvT,W˜cT,W˜aTT, it is able to determine that there exist two K-class functions α3andα4 satisfying(72)α3z≤ρβη−12zη2+zvTQ−0.5ρIzv+1−θc3−ka1k1k2W˜c2+ka2W˜a2≤α4z.
Based on (72), inequality (71) can be written as(73)ddtVL≤−α3z+ka1k12k2k3+k4+λmaxRΔ¯2+14θc3−ka1k1k2c4kcφ02νφ1k3+ka1k1k2k3+ka1k12ka2+ka2k12.
It is obvious thatd/dtVL is negative if zt lies outside the attraction region as(74)Ωz≜z:z≤α3−114θc3−ka1k1k2c4kcφ02νφ1k3+ka1k1k2k3+ka1k12k2+ka2k12+ka1k12k2k3+k4+λmaxRΔ¯2.
As a result of this, we can conclude thatz is UUB with attraction region (74) depending on the number of neurons in the critic NN.
Second, the tracking of constraint force is considered through the dynamic ofeλ=λ−λd obtained from (47) and (48) as(75)I+Kfeλ=Nτ1−NC∗ξ˙−ξ˙d−NF∗ξ,ξ˙,t.
According toz˙ξ=ξ˙−ξ˙d=zv−βzξ and zv,zξ are UUB, τ1,N,andF∗η,η˙,t are bounded; then, eλ is UUB with the attraction region depending on Kf and the bound of ARL-based motion dynamic control τ1=τind1τdep1. The proposed controller is absolutely implemented for uncertain multi-manipulators with attraction region (74) depending on model error Δ¯. This completes the proof.


## 3.1. ARL-Based Motion Controller

For the purpose of designing the motion controller in multimanipulator systems, the motion dynamic model is discussed after eliminating the constraint forces. Moreover, it is due to the description of time-varying systems in trajectory tracking control; the modification of the tracking error model needs to be implemented for developing the ARL-based optimal control design.Because ofΞη∈ℝm×1 with mm<2n independent constraint conditions, there exists an algebraic map φ:ℝ2n−m↦ℝ2n satisfying(4)η=φξ,where ξ∈ℝ2n−m×1 is the vector of independent joint variables. Taking the derivative of (4), we imply that(5)η˙=Λξ˙,η¨=Λξ¨+Λ˙ξ˙,where Λ=∂φ/∂ξ∈ℝ2n×2n−m. With the purpose of eliminating the constraint force in the motion dynamic model, we use the Moore–Penrose pseudo-inverse matrix of Jη being J+=JTJJT−1 to achieve the matrix Q=I−J+J satisfying QJT=0. Multiplying both sides of (3) with matrix Q, it leads to the following equation after utilizing QJT=0:(6)QMη¨+QCη˙+QF=Qτ−JTλ=Qτ−QJTλ=Qτ=τ1.According to (5) and (6), the motion dynamic can be expressed as(7)QMΛξ¨+QMΛ˙+QCΛξ¨+QF=τ1.Regarding the independent joint coordinates, the dynamic model can be represented after multiplying with matrixI2n−m,0∈ℝ2n−m×2n on both sides of (6) as follows:(8)ξ˙=υ,M¯¯υ˙+C¯¯υ+F¯¯=τind1,where(9)M¯¯=I2n−m,0QMΛ,C¯¯=I2n−m,0QMΛ˙+QCΛ,F¯¯=I2n−m,0QF,τind1=I2n−m,0τ1.Remark 2.
This paper only discusses the situation that the number of constraint conditions is smaller than the number of joints2n>m as mentioned in Assumption 1. For the case of 2n=m, it leads to no existence of independent joints. As a result, the above modifications are not implemented, and the control method is developed with different approaches [45]. Additionally, no constraint forces are considered in the special case m=0.
The motion control design is proposed as(10)τind1=τd+u,where(11)τd=M¯¯υ˙d+C¯¯υd+F¯¯,υd=ξ˙d−βzξ,zξ=ξ−ξd,zυ=υ−υd.
Therefore, with the assumption thatd/dtξd is the function of ξd as d/dtξd=h1ξd, we can obtain that(12)ddtzυ=ddtυ−υd=M¯¯−1u−M¯¯−1C¯¯υ−υd,ddtzξ=υ−ddtξd,ddtξd=h1ξd.
Furthermore,υd is the function of zξ,ξd as υd=ξ˙d−βzξ=h1ξd−βzξ⇒υ=zυ+υd is the function of zυ,zξ,ξd. Combining with (12), a continuous-time affine system has been obtained with the typical form(13)Z˙=FZ+GZu,where the vector of state variables is Z=zυ,zξ,ξdT∈ℝ12.Remark 3.
Due to the time-varying reference in trajectory tracking, the closed system needs to be known as the time-varying property. However, the framework of using term (11) and choosing the state variable vector Z is able to obtain time-invariant systems (12) and (13). Therefore, in contrast to the existing methods in [36], the proposed control easily handles the situation of time-varying systems by the elimination of the term −∂V∗/∂t.


### 3.1.1. The Optimal Control Problem

In this section, we consider the general class of time-varying systems:(14)ddtZ=FZ,u,t,with the associated cost function to be defined as(15)JZ,u=∫0∞rZτ,uτdt,where rZτ,uτ=ZTQZ+uTRu under the weighting matrices Q=QT≥0 and R=RT>0 to be defined with appropriate dimensions. It can be seen that the optimal problem for the motion model of multimanipulators (12) and (13) is to realize an admissible control policy [31, 32] for the purpose of obtaining the minimal value. This is because that the control policy u^Z needs to guarantee the existence of the estimated Bellman function V^Z obtained from the HJB equation.Definition 1.
A control lawuZ is considered being “admissible” with respect to the performance index (15) on a compact set Ω, which is known as a continuous signal uZ∈ΥΩ, if the properties of the stabilization of (14) and the limitation of JZ,u for every Z∈Ω are satisfied.
It is worth emphasizing that, in the general case of time-varying system (14), the optimal control strategy needs to be considered as a time-varying function u∗Z,t. Hence, the time-varying Bellman function with respect to arbitrary time V∗Zt,t can be given as(16)V∗Zt=minuZ∈ΥΩJZt,ut.
The HJB equation can be achieved by taking the time derivative ofV∗Xt under two different methods. Firstly, it is directly computed along system (14) as(17)ddtV∗Zt,t=∂V∗∂ZdZdt+∂V∗∂t=∂V∗∂ZFZ,u∗,t+∂V∗∂t.
Secondly, thanks to the dynamic programming principle, we also compute the derivative ofV∗Zt with respect to time as(18)ddtV∗Zt,t=−rZτ,u∗τ.
According to (17) and (18), the first part of the HJB problem can be given:(19)rZτ,u∗τ+∂V∗∂ZFZ,u∗,t=−∂V∗∂t.
Moreover, the second part of the HJB problem can be developed with the performance index to be formulated as(20)V∗Zt,t=minuZ∈ΥΩ∫t∞rZτ,u∗τdτ,(21)V∗Zt,t=minu∗Z∈ΥΩ∫tt+ΔtrZ,u∗dτ+minu∗Z∈ΥΩ∫t+Δt∞rZ,u∗dτ.
Because of utilizing the dynamic programming principle, the performance index is represented as(22)V∗Zt,t=minuZ∈ΥΩ∫tt+ΔtrZτ,uτdτ+V∗Zt+Δt,t+Δt.
Therefore, we imply that the Bellman function can be rewritten as(23)0=minuZ∈ΥΩ1Δt∫tt+ΔtrZ,u∗dτ+V∗Zt+Δt,t+Δt−V∗Zt,tΔt.
As the convergence ofΔt⟶0+, one can derive that(24)minuZ∈ΥΩrZt,u∗t+∂V∗∂ZFZ,u∗,t=−∂V∗∂t.Remark 4.
It is worth noting here that the term∂/∂tV∗Zt,t in (19) and (24) is the disadvantage in developing the ARL algorithm, and there has been very little research considering the direct method for this challenge. Authors in [36] developed the solutions of the ARL-based time-varying optimal controller using the framework of the data-driven and Newton–Leibniz formula-based function approximation method. In this paper, we indirectly handle the time-varying tracking error model through the consideration of systems (12) and (13). Therefore, the two parts of the HJB problem can be modified as(25)rZ,u∗+∂V∗∂ZFZ+GZu∗=0,(26)minuX∈ΥΩrX,u+∂V∗∂XFX+GXu=0.


### 3.1.2. ARL-Based Control Design for Independent Joints

It is because of difficulties in directly solving HJB equation (25) to obtain corresponding optimal control algorithm. Hence, the approximate solution with a neural network (NN) is mentioned to develop the ARL design in the motion dynamic model of multimanipulators. Thanks to the smooth property of functions V∗Z and u∗Z, they can be described over any compact domain C⊆ℝ12:(27)V∗Z=WTψZ+ϵZ,(28)u∗Z=−12R−1GTZ∂ψ∂ZTW+∂εZ∂ZT,where W∈ℝN is an uncertain ideal NN weight vector, N is chosen as the number of neurons, ψZ∈ℝN denotes an activation function vector satisfying ψj0=0 and ∂ψj/∂Z|Z=0=0,∀j=1,…,N, and ϵZ∈ℝ is known as the estimation error of the Bellman function V∗Z. Due to the unknown ideal NN weight vector W∈ℝN, it is necessary to obtain the corresponding adjusting mechanism W^a,W^c for estimating the actor/critic NNs to achieve the optimal control design in the absence of finding the HJB equation analytically (for more details, see [41]). Moreover, it is able to choose the smooth NN activation function vector ψZ∈ℝN based on the description of multimanipulators (see Section 4). In [41], thanks to the Weierstrass approximation theorem, we can uniformly approximate not only V∗Z but also ∂V∗Z/∂Z with ϵZ,∂ϵZ/∂Z⟶0 as N⟶∞. For a fixed number N, the approximated Bellman function of critic NN V^Z and the approximated optimal control policy of actor NN u^X are represented as(29)V^Z=W^cTψZ,(30)u^Z=−12R−1GTZ∂ψ∂ZTW^a.It should be noted that properties (19) and (24) of Hamiltonian HZ,u,∂V/∂Z=rZτ,uτ+∂V/∂ZFZ+GZu with the optimal controller u∗Z and corresponding value function V∗Z guarantee that the adjusting mechanisms of actor NN W^a and critic W^c weights are simultaneously tuned to minimize the squared Bellman error δhjb and the corresponding integral, respectively. According to the property of HJB problem (25), it leads to H∗Z,u∗,∂V∗/∂Z=0. After obtaining the error between approximated functions V^Z,u^Z and optimal control input V∗Z, u∗Z, the Bellman error δhjb is represented as(31)δhjb=H^Z,u^,∂V^∂Z−H∗Z,u∗,∂V∗∂Z=H^Z,u^,∂V^∂Z=W^cTσZ,u^+ZTQTZ+u^TRu^,where σZ,u^=∂ψ/∂ZFZ+GZu^ is the regression vector of the critic part.Based on the results in [31], the adjusting mechanism of the critic NN is computed:(32)ddtW^c=−kcλσ1+νσTλσδhjb,where ν,kc∈ℝ are constant positive coefficients and λt∈ℝN×N is an estimated symmetric matrix being the solution of the differential equation as(33)ddtλ=−kcλσσT1+νσTλσλ,λts+=λ0=φ0I,where ts+ denotes the resetting time satisfying that λminλt≤φ1,φ0>φ1>0. In [31], because of the condition that λt is positive definite and preventing the covariance wind-up problem, the covariance matrix λt is chosen as(34)φ1I≤λt≤φ0I.Additionally, the adjusting mechanism of the actor NN part is represented by minimizing the squared Bellman error:(35)ddtW^a=−ka11+σTσ∂ψ∂ZGR−1GT∂ψT∂ZW^a−W^cδhjb−ka2W^a−W^c.Remark 5.
ARL technique has been investigated with many different approaches, such as off-policy integral reinforcement learning (IRL) [46, 47] and Q-learning [42]. However, there is a wide difference between the online actor/critic technique in this paper and IRL and Q-learning. The off-policy IRL requires the complete data collection between the two sequential sampling times in computing the optimal law [46, 47]. Meanwhile, it is obviously different from the off-policy IRL technique; the online actor/critic method in (29), (30), (35), (33) requires the initial values to implement the computation. This could be exemplified by this article design control law under HJB problems (25) and (26), and the off-policy IRL method considers the relation between the deviation of integrals at two sampling times and the collected data under the dynamic programming law. In addition, the Q-learning method mentioned the Q-function in terms of not only state variables but also control inputs, being different from the Bellman function [42]. This property can result in the difficulties for implementing the Q-learning technique of the continuous-time system. Additionally, compared with the data-driven method in [48], this work provides a neural network-based technique to avoid the Kronecker product in estimating the actor/critic term. The actor/critic-based approaches have been discussed in [49, 50] for nonlinear affine systems using residual error δhjb. However, in view of the consideration of the identifier in [49, 50], it implies the difference in the computation of residual error δhjb and training laws in actor/critic weights between the proposed method and the work in [49, 50].


### 3.1.3. The Control Design of Dependent Joints

After completing the tracking control design of independent joints, we can achieve the remaining dependent joints’ controller based on the relation between two groups of joint variables, including independent joints and dependent joints. According to (8), it leads to the following representation:(36)ξ¨=M¯¯−1τind1−C¯¯ξ−F¯¯.On the contrary, thanks to the relationτ1=τind1T,τdep1TT, multiplying both sides of (6) with 0,Im∈ℝm×2n, we achieve the dependent joints to be computed as(37)M″ξ¨+C″ξ˙+F″=τdep1,where(38)M″=0,ImQMΛ,C″=0,ImQMΛ˙+QCΛ,F″=0,ImQF,τdep1=0,Imτ1.Substituting (36) into (37), we obtain the control law of dependent joint variables to be implemented as follows:(39)τdep1=M″M¯¯τind1−C¯¯ξ−F¯¯+C″ξ˙+F″.It is worth noting that the problem of synchronization between multiple joints can be considered by the relation between dependent and independent joints. Therefore, this problem is able to obtain the desired trajectories of independent joints and the trajectories of remaining joints depending on these independent joints. Furthermore, it leads to the relationship between the control input of independent joints and dependent joints (39).


## 3.1.1. The Optimal Control Problem

In this section, we consider the general class of time-varying systems:(14)ddtZ=FZ,u,t,with the associated cost function to be defined as(15)JZ,u=∫0∞rZτ,uτdt,where rZτ,uτ=ZTQZ+uTRu under the weighting matrices Q=QT≥0 and R=RT>0 to be defined with appropriate dimensions. It can be seen that the optimal problem for the motion model of multimanipulators (12) and (13) is to realize an admissible control policy [31, 32] for the purpose of obtaining the minimal value. This is because that the control policy u^Z needs to guarantee the existence of the estimated Bellman function V^Z obtained from the HJB equation.Definition 1.
A control lawuZ is considered being “admissible” with respect to the performance index (15) on a compact set Ω, which is known as a continuous signal uZ∈ΥΩ, if the properties of the stabilization of (14) and the limitation of JZ,u for every Z∈Ω are satisfied.
It is worth emphasizing that, in the general case of time-varying system (14), the optimal control strategy needs to be considered as a time-varying function u∗Z,t. Hence, the time-varying Bellman function with respect to arbitrary time V∗Zt,t can be given as(16)V∗Zt=minuZ∈ΥΩJZt,ut.
The HJB equation can be achieved by taking the time derivative ofV∗Xt under two different methods. Firstly, it is directly computed along system (14) as(17)ddtV∗Zt,t=∂V∗∂ZdZdt+∂V∗∂t=∂V∗∂ZFZ,u∗,t+∂V∗∂t.
Secondly, thanks to the dynamic programming principle, we also compute the derivative ofV∗Zt with respect to time as(18)ddtV∗Zt,t=−rZτ,u∗τ.
According to (17) and (18), the first part of the HJB problem can be given:(19)rZτ,u∗τ+∂V∗∂ZFZ,u∗,t=−∂V∗∂t.
Moreover, the second part of the HJB problem can be developed with the performance index to be formulated as(20)V∗Zt,t=minuZ∈ΥΩ∫t∞rZτ,u∗τdτ,(21)V∗Zt,t=minu∗Z∈ΥΩ∫tt+ΔtrZ,u∗dτ+minu∗Z∈ΥΩ∫t+Δt∞rZ,u∗dτ.
Because of utilizing the dynamic programming principle, the performance index is represented as(22)V∗Zt,t=minuZ∈ΥΩ∫tt+ΔtrZτ,uτdτ+V∗Zt+Δt,t+Δt.
Therefore, we imply that the Bellman function can be rewritten as(23)0=minuZ∈ΥΩ1Δt∫tt+ΔtrZ,u∗dτ+V∗Zt+Δt,t+Δt−V∗Zt,tΔt.
As the convergence ofΔt⟶0+, one can derive that(24)minuZ∈ΥΩrZt,u∗t+∂V∗∂ZFZ,u∗,t=−∂V∗∂t.Remark 4.
It is worth noting here that the term∂/∂tV∗Zt,t in (19) and (24) is the disadvantage in developing the ARL algorithm, and there has been very little research considering the direct method for this challenge. Authors in [36] developed the solutions of the ARL-based time-varying optimal controller using the framework of the data-driven and Newton–Leibniz formula-based function approximation method. In this paper, we indirectly handle the time-varying tracking error model through the consideration of systems (12) and (13). Therefore, the two parts of the HJB problem can be modified as(25)rZ,u∗+∂V∗∂ZFZ+GZu∗=0,(26)minuX∈ΥΩrX,u+∂V∗∂XFX+GXu=0.


## 3.1.2. ARL-Based Control Design for Independent Joints

It is because of difficulties in directly solving HJB equation (25) to obtain corresponding optimal control algorithm. Hence, the approximate solution with a neural network (NN) is mentioned to develop the ARL design in the motion dynamic model of multimanipulators. Thanks to the smooth property of functions V∗Z and u∗Z, they can be described over any compact domain C⊆ℝ12:(27)V∗Z=WTψZ+ϵZ,(28)u∗Z=−12R−1GTZ∂ψ∂ZTW+∂εZ∂ZT,where W∈ℝN is an uncertain ideal NN weight vector, N is chosen as the number of neurons, ψZ∈ℝN denotes an activation function vector satisfying ψj0=0 and ∂ψj/∂Z|Z=0=0,∀j=1,…,N, and ϵZ∈ℝ is known as the estimation error of the Bellman function V∗Z. Due to the unknown ideal NN weight vector W∈ℝN, it is necessary to obtain the corresponding adjusting mechanism W^a,W^c for estimating the actor/critic NNs to achieve the optimal control design in the absence of finding the HJB equation analytically (for more details, see [41]). Moreover, it is able to choose the smooth NN activation function vector ψZ∈ℝN based on the description of multimanipulators (see Section 4). In [41], thanks to the Weierstrass approximation theorem, we can uniformly approximate not only V∗Z but also ∂V∗Z/∂Z with ϵZ,∂ϵZ/∂Z⟶0 as N⟶∞. For a fixed number N, the approximated Bellman function of critic NN V^Z and the approximated optimal control policy of actor NN u^X are represented as(29)V^Z=W^cTψZ,(30)u^Z=−12R−1GTZ∂ψ∂ZTW^a.It should be noted that properties (19) and (24) of Hamiltonian HZ,u,∂V/∂Z=rZτ,uτ+∂V/∂ZFZ+GZu with the optimal controller u∗Z and corresponding value function V∗Z guarantee that the adjusting mechanisms of actor NN W^a and critic W^c weights are simultaneously tuned to minimize the squared Bellman error δhjb and the corresponding integral, respectively. According to the property of HJB problem (25), it leads to H∗Z,u∗,∂V∗/∂Z=0. After obtaining the error between approximated functions V^Z,u^Z and optimal control input V∗Z, u∗Z, the Bellman error δhjb is represented as(31)δhjb=H^Z,u^,∂V^∂Z−H∗Z,u∗,∂V∗∂Z=H^Z,u^,∂V^∂Z=W^cTσZ,u^+ZTQTZ+u^TRu^,where σZ,u^=∂ψ/∂ZFZ+GZu^ is the regression vector of the critic part.Based on the results in [31], the adjusting mechanism of the critic NN is computed:(32)ddtW^c=−kcλσ1+νσTλσδhjb,where ν,kc∈ℝ are constant positive coefficients and λt∈ℝN×N is an estimated symmetric matrix being the solution of the differential equation as(33)ddtλ=−kcλσσT1+νσTλσλ,λts+=λ0=φ0I,where ts+ denotes the resetting time satisfying that λminλt≤φ1,φ0>φ1>0. In [31], because of the condition that λt is positive definite and preventing the covariance wind-up problem, the covariance matrix λt is chosen as(34)φ1I≤λt≤φ0I.Additionally, the adjusting mechanism of the actor NN part is represented by minimizing the squared Bellman error:(35)ddtW^a=−ka11+σTσ∂ψ∂ZGR−1GT∂ψT∂ZW^a−W^cδhjb−ka2W^a−W^c.Remark 5.
ARL technique has been investigated with many different approaches, such as off-policy integral reinforcement learning (IRL) [46, 47] and Q-learning [42]. However, there is a wide difference between the online actor/critic technique in this paper and IRL and Q-learning. The off-policy IRL requires the complete data collection between the two sequential sampling times in computing the optimal law [46, 47]. Meanwhile, it is obviously different from the off-policy IRL technique; the online actor/critic method in (29), (30), (35), (33) requires the initial values to implement the computation. This could be exemplified by this article design control law under HJB problems (25) and (26), and the off-policy IRL method considers the relation between the deviation of integrals at two sampling times and the collected data under the dynamic programming law. In addition, the Q-learning method mentioned the Q-function in terms of not only state variables but also control inputs, being different from the Bellman function [42]. This property can result in the difficulties for implementing the Q-learning technique of the continuous-time system. Additionally, compared with the data-driven method in [48], this work provides a neural network-based technique to avoid the Kronecker product in estimating the actor/critic term. The actor/critic-based approaches have been discussed in [49, 50] for nonlinear affine systems using residual error δhjb. However, in view of the consideration of the identifier in [49, 50], it implies the difference in the computation of residual error δhjb and training laws in actor/critic weights between the proposed method and the work in [49, 50].


## 3.1.3. The Control Design of Dependent Joints

After completing the tracking control design of independent joints, we can achieve the remaining dependent joints’ controller based on the relation between two groups of joint variables, including independent joints and dependent joints. According to (8), it leads to the following representation:(36)ξ¨=M¯¯−1τind1−C¯¯ξ−F¯¯.On the contrary, thanks to the relationτ1=τind1T,τdep1TT, multiplying both sides of (6) with 0,Im∈ℝm×2n, we achieve the dependent joints to be computed as(37)M″ξ¨+C″ξ˙+F″=τdep1,where(38)M″=0,ImQMΛ,C″=0,ImQMΛ˙+QCΛ,F″=0,ImQF,τdep1=0,Imτ1.Substituting (36) into (37), we obtain the control law of dependent joint variables to be implemented as follows:(39)τdep1=M″M¯¯τind1−C¯¯ξ−F¯¯+C″ξ˙+F″.It is worth noting that the problem of synchronization between multiple joints can be considered by the relation between dependent and independent joints. Therefore, this problem is able to obtain the desired trajectories of independent joints and the trajectories of remaining joints depending on these independent joints. Furthermore, it leads to the relationship between the control input of independent joints and dependent joints (39).


## 3.2. Constraint Force Control Design

For the purpose of designing the motion/force control scheme forn manipulators, the constraint force controller and motion control law are necessary to be established based on the computation of lambda coefficient in (3). Before demonstrating the motion/force control design, we need to compute the coefficient λ to be dependent on control input τ as follows. According to (3)–(5), we achieve the following equation:(40)J∗Tξλ=τ−M∗ηξ¨+C∗η,η˙ξ˙+F∗η,η˙,t,where(41)J∗Tξ=JTφξ,M∗η=MηΛ,C∗η,ξ˙=Cη,η˙Λ+MηΛ˙,F∗η,η˙,t=Fη,η˙,t.Moreover, we find the method to compute coefficientλ after eliminating the term ξ by multiplying both sides with matrix N satisfying that(42)NMηΛ=0,NJ∗Tξ=I.Therefore, we can obtain the following coefficientλ:(43)λ=Nτ−NC∗η,η˙ξ˙+NF∗η,η˙,t,with N to be chosen as(44)N=AJ∗Tξ−1A,where A=J∗ξM−1η.Based on the proposed motion/force, control design is described in the following equation:(45)τ=τind1τdep1−JTητ2.This leads to the following relations that(46)Qτ1=Qτind1τdep1=τ1,(47)λ=Nτ1−τ2−NC∗η,η˙ξ˙+NF∗η,η˙,t.It should be noted that the constraint force coefficientλ depends on the ARL-based motion dynamic control τ1=τind1τdep1 to be described in (47). Therefore, with the purpose of tracking the coefficient λ, we can propose the remaining term τ2 in (47) as given in the following:(48)τ2=Kfeλ−λd−NC∗η,η¨ξ˙d.The tracking problem will be proved in the next section.


## 3.3. Convergence and Stability Analysis

With the purpose of considering the stability and convergence of the closed system under the proposed motion-force control design, we need to implement several following steps. First, the Bellman errorδhjb, critic estimated weight error W˜c=W−W^c, and the persistence of excitation (PE) condition need to be given. Second, thanks to the optimal Bellman function and PE condition, the corresponding Lyapunov function candidate is chosen. Third, the stability of the whole system (Figure 1) and the role of constraint force control and dependent joints’ motion control are discussed.According to (27)–(31), the Bellman error function δhjb is dependent on state variable vector Zt as(49)δhjb=W^cTσZ,u^+ZTQTZ+u^TRu^−WTσZ,u∗−u∗TRu∗−∂ϵ∂ZFZ+GZu∗−ZTQTZ=−W˜cTσZ,u^+14W˜aT∂ψ∂ZGR−1GT∂ψ∂ZTW˜a−14∂ϵ∂ZGR−1GT∂ϵ∂ZT−∂ϵ∂ZFZ+GZu∗.Combining (49) with (32), we imply the dynamic equation of critic error W˜c=W−W^c to be given as(50)ddtW˜c=−kcλΠΠTW˜c+kcλσZ,u^1+νσTλσ14W˜aT∂ψ∂ZGR−1GT∂ψ∂ZTW˜a−14∂ϵ∂ZGR−1GT∂ϵ∂ZT−∂ϵ∂ZFZ+GZu∗,where W˜a=W−W^a and Πt=σZ,u^/1+νσZ,u^TλσZ,u^ is satisfied by(51)Π≤1νφ1.The nominal system is given after eliminating the term of NN errors:(52)ddtW˜c=−kcλΠΠTW˜c.Due to the purpose of exponential convergence ofW˜c, it is necessary to guarantee the PE condition as follows:(53)μ2I≥∫t0t0+δΠsΠsTds≥μ1I,∀t0≥0.The following main theorem is proposed with the assumptions being discussed in [41] to estimate the attraction region of tracking errors.Theorem 1.
Consider the cooperating manipulators in Figure1 with the assumption of the bound conditions of the NN including ideal weights, activation function ψ., its derivative to be described in [41], and PE condition of the signal vector Πt, and the selection of parameters is implemented with the following condition:(54)c3ka1>k1k2,where the coefficients in (54) are mentioned in (55), (56), and (35). Let us consider the control structure (Figure 2), (10) with ARL-based control scheme (46), (10), (39), and (30), the associated adjusting mechanisms (35) and (33) for the actual controller, and constraint force control vector (48); then, (1) the actor-critic weight errors W˜a and W˜c are UUB; (2) the tracking effectiveness of not only z˜v=v−vd but also z˜η=ξ−ξd in the closed control system (Figure 1) is also UUB; (3) the tracking of constraint force coefficient vector λ and the remaining terms of joint variables’ vector η2n=ξ2n−m,pm is also UUB.Proof.
First, the tracking effectiveness of motion tracking control is considered under the ARL-based optimal control scheme for corresponding tracking error motion models (12) and (13). Due to the satisfaction of PE condition (53), there exists a time-varying function Vc:ℝN×0,∞⟶ℝ satisfying several following inequalities:(55)c1W˜c2≤VcW˜c,t≤c2W˜c2,∂Vc∂t+∂Vc∂W˜c−kcλΥΥTW˜c≤−c3W˜c2,∂Vc∂W˜c≤c4W˜c,where c1,c2,c3,c4∈ℝ are positive constant coefficients. The function VcW˜c,t is one part of the Lyapunov function candidate. Moreover, according to the bound conditions of the proposed NN [41], we can achieve the following results with the purpose of estimating the time derivative of the Lyapunov function in the next steps:(56)i.W˜a≤k1,i2.∂ψ∂ZGR−1GT∂ψ∂Z≤k2,i3.14W˜aT∂ψ∂ZGR−1GT∂ψ∂ZTW˜a−14∂ϵ∂ZGR−1GT∂ϵ∂ZT−∂ϵ∂ZFZ+GZu∗≤k3,i4.12WT∂ψ∂ZGR−1GT∂ϵ∂ZT+12∂ϵ∂ZGR−1GT∂ϵ∂ZT+12WT∂ψ∂ZGR−1GT∂ψ∂ZTW˜a+12∂ϵ∂ZGR−1GT∂ψ∂ZTW˜a≤k4.
Thanks to Bellman functionV∗Z (20) and time-varying VcW˜c,t (55), a Lyapunov function candidate can be chosen to determine the stability of the whole of the cascade control system (Figure 2) and the tracking of the weights of the actor NN and critic NN:(57)VL≜12ρzξTzξ+V∗Z+VcW˜c,t+12W˜aTW˜a.
It is worth noting that the term1/2ρzξTzξ (ρ is a positive constant coefficient) is the additional term for the purpose of tracking of the whole motion control system. Because V∗Z is a smooth function and positive definite, there exist two K-class functions γ1andγ2 such that(58)γ1Z≤V∗Z≤γ2Z.
Because of (55) and (8), we can determine the strict proper representation of Lyapunov function VLZ:(59)12ρzξ2+α1Z+c1W˜c2+12W˜a2≤VL≤12ρzξ2+α2Z+c2W˜c2+12W˜a2.
Implementing the derivative ofVL along motion system (13) under the control input u^Z, we imply that(60)ddtVL=ρzξTzv−βzξ+∂V∗∂ZFZ+GZu^+∂Vc∂t+∂Vc∂W˜cΘ1+Θ2−W˜aTW^˙a+∂V∗∂ZGZΔ,where(61)Θ1=−kcλΠΠTW˜cΘ2=kcλσ1+νσTλσ14W˜aT∂ψ∂ZGR−1GT∂ψ∂ZTW˜a−14∂ϵ∂ZGR−1GT∂ϵ∂ZT−∂ϵ∂ZFZ+GZu∗.
According to (19), the relation between the Bellman function and the optimal control law can be represented as(62)∂V∗∂ZFZ=−∂V∗∂ZGZu∗Z−ZTQTZ−u∗TZRu∗Z.
According to (62) and optimal law (28), (35), and (55) in (60), we can conclude that(63)ddtVL≤−ρβzξ2+ρzξTzv−zvTQzv−u∗TRu∗+2u∗TRu∗−u^−c3W˜c2+c4W˜cΘ2+ka2W˜aTW^a−W^c−2u∗TRΔ+ka11+σTσW˜aT∂ψ∂ZGZR−1GTZ∂ψ∂ZTW^a−W^cδhjb.
Using Young’s inequality, one can obtain that(64)ρzξTzv≤ρ2zξ2+ρ2zvTzv=ρ2zξ2+ρ2zv2.
From (30), (19), and (56), the following inequality is given:(65)2u∗TRu∗−u^=12WT∂ϕ∂ZGR−1GT∂ϵ∂ZT+12WT∂ϕ∂ZGR−1GT∂ϕ∂ZTW˜a+12∂ϵ∂ZGR−1GT∂ϕ∂ZTW˜a+12∂ϵ∂ZGR−1GT∂ϵ∂ZT≤k4,and the term Θ2 is bounded by(66)Θ2≤kcφ02νφ1k3.
Following (56) and (49), we have(67)ka11+σTσW˜aT∂ψ∂ZGZR−1GTZ∂ψ∂ZTW^a−W^cδhjb=ka11+σTσW˜aT∂ψ∂ZGZR−1GTZ∂ψ∂ZTW˜c−W˜a×−W˜cTσ+14W˜aT∂ψ∂ZGR−1GT∂ψ∂ZTW˜a−14∂ϵ∂ZGR−1GT∂ϵ∂ZT−∂ϵ∂ZFZ+GZu∗≤ka1k1k2W˜c2+ka1k12k2W˜c+ka1k1k2k3W˜c+ka1k12k2k3.
It is obvious that(68)ka2W˜aTW^a−W^c=ka2W˜aTW˜c−W˜a≤ka2k1W˜c−ka2W˜a2.
Moreover, it can be seen that(69)−u∗TRu∗−2u∗TRΔ≤ΔTRΔ≤λmaxRΔ¯2.
Replacing (64) and (68) in (63), we can conclude that(70)ddtVL≤−ρβ−12zξ2−zvTQ−0.5ρIzv−c3−ka1k1k2W˜c2−ka2W˜a2+ka1k12k2k3+k4+c4kcφ02νφ1k3+ka1k1k2k3+ka1k12k2+ka2k1W˜c+λmaxRΔ¯2.
Based on the estimationab≤γa2+1/4γb2, we can obtain that(71)ddtVL≤−ρβξ−12zη2−zvTQ−0.5ρIzv+λmaxRΔ¯2−1−θc3−ka1k1k2W˜c2−ka2W˜a2+ka1k12k2k3+k4+14θc3−ka1k1k2c4ηcφ02νφ1k3+ka1k1k2k3+ka2k1+ka1k12k22.
Let us select the coefficients to be satisfied such thatβ>1/2,0<ρ<2λminQ,0<θ<1,andc3>ka1k1k2.
With the state vectorz=zηT,zvT,W˜cT,W˜aTT, it is able to determine that there exist two K-class functions α3andα4 satisfying(72)α3z≤ρβη−12zη2+zvTQ−0.5ρIzv+1−θc3−ka1k1k2W˜c2+ka2W˜a2≤α4z.
Based on (72), inequality (71) can be written as(73)ddtVL≤−α3z+ka1k12k2k3+k4+λmaxRΔ¯2+14θc3−ka1k1k2c4kcφ02νφ1k3+ka1k1k2k3+ka1k12ka2+ka2k12.
It is obvious thatd/dtVL is negative if zt lies outside the attraction region as(74)Ωz≜z:z≤α3−114θc3−ka1k1k2c4kcφ02νφ1k3+ka1k1k2k3+ka1k12k2+ka2k12+ka1k12k2k3+k4+λmaxRΔ¯2.
As a result of this, we can conclude thatz is UUB with attraction region (74) depending on the number of neurons in the critic NN.
Second, the tracking of constraint force is considered through the dynamic ofeλ=λ−λd obtained from (47) and (48) as(75)I+Kfeλ=Nτ1−NC∗ξ˙−ξ˙d−NF∗ξ,ξ˙,t.
According toz˙ξ=ξ˙−ξ˙d=zv−βzξ and zv,zξ are UUB, τ1,N,andF∗η,η˙,t are bounded; then, eλ is UUB with the attraction region depending on Kf and the bound of ARL-based motion dynamic control τ1=τind1τdep1. The proposed controller is absolutely implemented for uncertain multi-manipulators with attraction region (74) depending on model error Δ¯. This completes the proof.


## 4. Simulation Results

In this section, an example with 3 manipulators (Figure3) is given to validate the effectiveness of the proposed control structure (Figure 2) by the m-file script in MATLAB software. This example has been considered in [12] by the classical robust nonlinear controller. However, it is obviously different from the existing method in [12], and the proposed motion/force control based on ARL is first structured to obtain optimality satisfaction. Similar to the model in [12], the constrained multimanipulator system is represented by dynamic equation (3) with the following constraints:(76)ϕ12=xE1−xE22+zE1−zE221/2−D12=0ϕ13=xE1−xE32+yE1−yE32+zE1−zE321/2−D13=0.Figure 3
The constrained multimanipulator system [12].In this example, the matrices in (3) can be represented as in [12]:(77)M=M1M2M3,M1=M11M12M21M22,M2=M33M34M43M44,M3=M55M56M65M66,C=C1C2C3,C1=C11C12C21C22,C2=C33C34C43C44,C3=C55C56C65C66.Moreover, to easily compare with the existing work [12], the parameters of multimanipulators are given as D12=7m,D13=3.5m,l1=l2=1m,m1=m2=1kg,r1=r2=0.9m,l3=l4=1.5m,m3=m4=2kg,r3=r4=1.2m,l5=l6=1.2m,m5=m6=1.5kg,andr5=r6=1m. Because of two constraints (76), this multimanipulator has 4 independent joints η1,η2,η3,andη5 with the desired trajectories η1d=π/12,η2d=1.91π+0.2sint,η3d=0.51π,andη5d=0.191π, and the initial values of these independent joints are η10=π/6,η20=1.92π,η30=2π/3,η40=−0.0565,η50=π/4,andη60=−0.0853. According to the given multimanipulator and above parameters, we achieve the necessary models for investigating the control design, including motion dynamic model (7), (37), and constraint force (43). In light of Theorem 1, the control parameters are given as follows: k1=0.5,ηc=5,ηa1=5,ηa2=10,andν=0.01. The dynamic model of 3manipulators with the above parameters and ARL-based motion/force control scheme is established by the m-file script and Simulink in MATLAB software. The purpose of simulations is to verify the tracking effectiveness of not only joint variables but also constraint force coefficients. Additionally, due to the implementation of optimal control for the motion dynamic model by the radial basis function (RBF) network-based actor/critic reinforcement learning algorithm, the convergence of the adjusting mechanism of actor NN (35) and critic NN (33) should be verified, and the activation function is chosen depending on multimanipulator systems:(78)ψZ=z12,z1z2,z1z3,z1z4,z22,z2z3,z2z4,z32,z3z4,z42,z12z92,z22z102,z32z112,z42z122,z12z52,z22z62,z32z72,z42z82,where the vector of state variables isZ=zυ,zξ,ξdT∈ℝ12. Moreover, the adjusting mechanisms of weights in actor and critic are implemented as in (32 and 35). On the contrary, in order to complete the proposed motion/force control structure, the remaining terms need to be realized, (39) and (48). In order to illustrate the performance of the ARL scheme, we examine to obtain the actor/critic weight response and the tracking effectiveness of joint variables. Referring to Figures 4 and 5, it is seen that the response of joint variables 1–6 with tracking errors converging to zero is 10 s, which leads to that the effectiveness of closed systems is highly precise under the ARL technique. Additionally, the convergence of adjusting mechanisms of the actor NN and critic NN to the ideal weight is also obtained (0.0006, −0.0055, 0.0045, 0.0038, 0.014, −0.018, −0.009, 0.019, 0.011, 0.0055, 0.000041, 0.00051, 0.0492, 0.002, 0.000015, 0.0024, 0.0047, 0.00019) (Figures 6–11). Finally, the high-performance responses of constraint force and control inputs are shown in Figures 12 and 13. It should be noted that these simulation results are conducted for three manipulators to be described in the work [12]. However, they are obviously different with [12]; due to the ARL-based approach, the proposed control scheme also considers the convergence of weights in actor/critic neural networks (Figures 6–11).Figure 4
The response of 6 joint variables.Figure 5
The tracking errors of 6 joint variables.Figure 6
Weights 1–6 of the actor NN.Figure 7
Weights 7–12 of the actor NN.Figure 8
Weights 13–18 of the actor NN.Figure 9
Weights 1–6 of the critic NN.Figure 10
Weights 7–12 of the critic NN.Figure 11
Weights 13–18 of the critic NN.Figure 12
The response of constraint coefficients.
(a)(b)Figure 13
The control inputs of multimanipulator systems.


## 5. Conclusions

This paper presented an ARL-based motion/force tracking control for multimanipulator systems, which consists of the ARL-based motion control law and nonlinear force controller. Both the minimization of the performance index and closed-loop stability are guaranteed by theoretical analysis. Moreover, the proposed control structure is able to handle the time-varying model using the additional term. The simulation studies are conducted to illustrate the efficacy of the proposed control structure. Future work is to develop completely uncertain multimanipulator systems by integrating model-free reinforcement learning.



---

*Source: 5560277-2021-06-01.xml*