# Adaptive Teleoperation System with Neural Network-Based Multiple Model Control

Authors: Vu Trieu Minh; Fakhruldin Mohd Hashim
Journal: Mathematical Problems in Engineering
 (2010)

---

## Abstract

This paper presents an adaptive teleoperation which is robust to time-delay and environmental uncertainties while assuring the transparent performance. A novel theoretical framework and algorithms for the teleoperation system have been built up with neural network-based multiple model control and time forward state observer. Conditions for stability and transparency performance are also identified. Simulations show that the system is stable and in good performance.


---

## Body

## 1. Introduction

Applications of master-slave teleoperation system can be found in many areas from micro to macro scales. A teleoperation system or a bilateral telemanipulation system is a complex electro-mechanical system with a master and a slave device, interconnected by a communication channel and controller. Through interaction with the master device, the human operator is able to communicate control signals for the slave. The slave device actually interacts with the remote environments, thereby staying under full control of the human operator. Information gathered at the remote environments is transmitted back to the human operator through the master device. Usually a camera is installed at the slave device to provide the visual information to the operator.The existence of varying time delays in the communication channel and the uncertainty of the remote environments are the most important problems regarding the stability of the teleoperation systems since they will cause bad performance or instability to the system. Another important aspect is the transparency or the transparent performance of the systems. The transparency is defined as the impedance felt by the human operator on the local site. This allows the operator to feel the real sense of the remote environments. It can be applied in the case, for example, where the medical doctors cannot approach the patients directly.Delays in the communication channel make the standard control law nonpassive system. But by mimicking the lossless transmission line, the system becomes passive and, hence, stable. Kikuchi el al. [1] proposed a teleoperation system in dynamic environment with varying communication time delays. The proposed system consists of the stable bilateral teleoperation subsystem that offers pictures in remote site and the environment prediction display subsystem that offers the prediction of the slave manipulator and the environment. Zhu and Salcudean [2] introduced a novel stability guaranteed controller design for bilateral teleoperation under both position and rate control modes with arbitrary motion/force scaling. Boukhnifer and Ferreira [3] presented a bilateral controller for a micro-teleoperation system using passivity approaches. They showed that the application of wave variable formation allows the passivity of the system in spite of the communication delays and the varying scaling factors.Transparency and stability in bilateral teleoperation systems have also been analyzed when communication delays are present. Lawrence [4] explored the trade-off algorithm between transparency and stability based on the concept of impedance, a quantity that maps the input position of a system to the output force. When a teleoperation system is ideal, the operator feels as if he is operating directly the remote environments. Slawiñski and Mut [5] proposed defining transparency in the time domain and established a quantitative measure of how the human operator feels the remote system. It allows analyzing the effect of the time-varying delays on the system transparency.The complexity of the uncertain remote environments makes the development of such exact models for all potential environments impossible. For the model detection, various methods have been developed. One of the most effective approaches is the use of multiple-model neural network. By very fast calculation, the neural network can select the most reliable model and provide the online optimal control action for the system. Most of the current employed neural networks for artificial intelligence are based on statistical estimation, optimization, and control theory. Chen et al. [6] proposed a neural network-based multiple-model adaptive control for teleoperation system. Decision controller is designed to adaptive switch among all predictive controllers according to the performance target. This method can ensure the stability and transparent performance of the system. Smith and Hashtrudi-Zaad [7] used two neural networks at the master and slave devices to improve the transparency and to compensate the effect of the time delays.This paper presents a novel scheme to control the displacement and force for a bilateral teleoperation system in variant time delays and environmental uncertainties. The aim of the paper is to develop a simple and low cost teleoperation connected via the global internet system (Internet Protocol Suite—TCP/IP) and without a camera in the slave device. The system can be supported by a telephone or voice chat over Internet Protocol (VoIP) for a medical doctor who can use the hands/fingers to examine the remote patients. The idea of the system is initiated from the design of a medical teleanalyzer in Suebsomran and Parnichkun [8] who introduced a hybrid teleoperation controller of displacement and force, which is robust to internal and external disturbances, model uncertainty, load, and fiction. The teleoperation transparency in time delays and the effect of local force feedback were also investigated by Hashtrudi-Zaad and Salcudean [9] where the stability robustness of the system was analyzed and the transparency conditions for a general teleoperation system are presented. The environment uncertainty modeling and verification are referred to by Minh V.T et al. in [10]. The stability of uncertain cellular neural networks and the stability criterion for bidirectional associate memory neural networks with interval time-varying delays are also referred to by Kwon and Park in [11, 12].The followings are the content of this paper: Section2 introduces the system modeling; Section 3 develops the system time forward observer; Section 4 sets up stability and transparency conditions for the system controller; Section 5 illustrates simulation results; and finally conclusions are drawn in Section 6.


## 2. System Modeling

As mentioned in the introduction part, a teleoperation system consists of four fundamental components: the human operator, the communication channel, the slave manipulator, and the environment. Through these five components, information is exchanged in two directions. The control channel enables the human operator to assign tasks to the slave and control the slave as desired. Then, information about the task execution is fed back to the human operator. Both channels are interconnected via the normal internet system. For the simplification and low cost purpose, there is no camera installed in the slave device—a voice chat over Internet Protocol can support the communication between the doctor and the patient instead. Figure1 shows the basic configuration for a bilateral control setting adopted by the position-force controller architecture by Lawrence [4]: the operator moves the master manipulator and via the Internet channel communication, the slave manipulator follows. Forces exerted by the environment on the slave are transmitted back to the master and felt by the operator.Figure 1
Teleopration system configuration.The human force on the masterfh  and the master motion xm   should have the same relationship with the force on the environment fe  and the slave motion xs, that is, for the same forces, fe=fh, the motions should be the same, xs=xm. This requirement assures that the system is completely transparent. So the operator feels the real sense of the remote environments. Notations, u, udm, ud, and us, are the signals transmitted among the system’s components.The master and slave are all modeled as mechanical devices with masses, dampers, and springs, as shown in Figure2.Figure 2
Physical model of master and slave manipulator.The human forcefh  is applied to the master manipulator and xm is the position of the master; fe is the contact force between the slave manipulator and the environment, and xs is the position of the slave; udm is the feedback signal from the slave, and ud is the control signal given by the master through the Internet. Notations, mm, ms, bm, bs, km, and ks, are the inertia, damping, and spring stiffness of the master and slave manipulator, respectively.The dynamics of the master and the slave are given by the following equations:(2.1)fh-udm=mmẍm+bmẋm+kmxm,(2.2)ud-fe=msẍs+bsẋs+ksxs.
For the variant time delays and the environmental uncertainties, it is assumed that the dynamics of the environment are also regulated by a finite set of mechanical devices with masses, dampers, and springs, shown in Figure 3: xsi denotes the position of the slave device and fei denotes the force applied on the environment i. Similarly, mei, kei, and bei are mass, stiffness and damping parameters of the unknown and time-variant environment models.Figure 3
Physical Model of Environment.The dynamics of the environment model are formulated by the following equation:(2.3)fe=meẍs+beẋs+kexsorfei=meiẍsi+beiẋsi+keixsi,fori=1,…,n.
Since the environment models are uncertain, a neural network with the radial basis function (RBF) is used to detect the environmental dynamics since RBF network is easy to approximate the parameters and the training speed is fast. For n given environment models, the input vectors U is the movement of the slave device
(2.4)U=[xs1⋯xsnẋs1⋱ẋsnẍs1⋯ẍsn],
and the target vectors Y  is the environment model parameters:
(2.5)Y=[ke1⋯kenbe1⋱benme1⋯men].
From the online archived slave movements, [xsiẋsiẍsi]′, RBF will on line calculate the corresponding environment parameters,[k̂eib̂eim̂ei]′, and, then, the corresponding model errors ∈i=[mei-m̂ei]2+[bei-b̂ei]2+[kei-k̂ei]2. If the ∈m is the smallest, the mth environment model which fits the best to the current dynamics is selected. The environment modeling and verification are referred to by Minh. et al. in [10]. It is assumed that the environmental uncertainties can be represented by mathematical models. To each model, the system behavior changes and should be estimated by a different modes. The system mode may jump up or vary continuously in a discrete set.Since the master and the slave devices are connected via the Internet, the time delay is significant and variant. Thus, the equations for signal transmitted via communication channel must include the forward time delay,TR(t), and the backward time delay, TL(t):(2.6)ud=u(t-TR(t)),udm=us(t-TL(t)).
Finally the overall schematic diagram of the proposed system is show in Figure 4.Figure 4
Schematic Diagram of the Teleoperation System.The operator moves the master plant, which causes a transmitted signal,u(t). In conventional methods proposed by Lawrence [4] and Hashtrudi-Zaad and Salcudean [9], the transmitted signals include position, velocity, and acceleration. But in this proposed system, the operator (medical doctor) wants to directly and gently touch the patient, and the acceleration is difficult to be realized and can be omitted from the dynamic equations in order to guarantee the high transparency for the operator. Thus,   (2.7)u(t)=f11xm(t)+f12ẋm(t)+c11fh(t),
where f11, f12, and c11 are the feedback coefficients. Because the forward time delay, TR(t), causes ud(t)≠u(t), we use xs(t), ẋs(t), and fe(t)  to adjust  ud(t) as follows:(2.8)ud(t)=f11xm(t-TR(t))+f12ẋm(t-TR(t))+c11fh(t-TR(t))+f13xs(t)+f14ẋs(t)+c12fe(t),
where f13, f14, and c12 are also the feedback coefficients.Similarly, the slave transmitted signal,us(t), should include force, position, and velocity:
(2.9)us(t)=f23xs(t)+f24ẋs(t)+c22fe(t),
where f23, f24, and c22 are the feedback coefficients. And since udm(t)≠us(t) is caused by the backward time delay, TL(t), we use xm(t-T(t)), ẋm(t=T(t)), and fh(t-T(t)) to adjust udm(t) as follows:(2.10)udm(t)=f23xs(t-TL(t))+f24ẋs(t-TL(t))+c22fe(t-TL(t))+f21xm(t-T(t))+f22ẋm(t)+c21fh(t-T(t)),
where T(t)=TR(t)+TL(t)  is the total of time delay and f21, f22, and c21 are the feedback coefficients. In this equation, udm(t) can respond the control effect of u(t-T(t)). As a result, if we use the predicted value of udm(t+T(t))  as the feedback value from the slave, then the operator can feel that the time delay does not exist since the time delay on the transparency has been eliminated.Predicted value ofudm(t+T(t))  is labeled as ûdm(t) and(2.11)ûdm(t+T(t))=f21xm(t)+f22ẋm(t)+c21fh(t)+f23x̂s(t+TR(t))+f24x̂̇m(t+TR(t))+c22f̂e(t+TR(t)),
where x̂s(t+TR(t)), x̂̇m(t+TR(t)), and f̂e(t+TR(t))  are the predicted values of xs(t+TR(t)), ẋm(t+TR(t)), and fe(t+TR(t)).According to the environment model,f̂e(t+TR(t))  can be estimated by the following equation:(2.12)f̂e(t+TR(t))=mex̂̈s(t+TR(t))+bex̂̇s(t+TR(t))+kex̂s(t+TR(t)).
Then, the predicted values of x̂s(t+TR(t)) and x̂̇s(t+TR(t)) can be achieved through the time forward observer design in the next section.


## 3. Observer Design

Teleoperation observer is used to estimate the system parameters for the time delay compensation.  The system can only achieve a high level of transparency with a good observer based on the predicted system dynamics. In order to predictxs(t+TR(t))  and ẋs(t+TR(t)), (2.8) is substituted into (2.2):(3.1)f11xm(t-TR(t))+f12ẋm(t-TR(t))+c11fn(t-TR(t))+f13xs(t)+f14ẋm(t)+c12fe(t)-fe(t)=msẍs(t)+bsẋs(t)+ksxs(t).
We define(3.2)u̅d(t)=f11xm(t-TR(t))+f12ẋm(t-TR(t))+c11fh(t-TR(t)).
Equation (3.1) becomes
(3.3)u̅d(t)=msẍs(t)+(bs-f14)ẋs(t)+(ks-f13)xs(t)+(1-c12)fe(t).
Substituting (2.3) into (3.3),(3.4)u̅d(t)=(ms+(1-c12)me)ẍs(t)+((bs-f14)+(1-c12)be)ẋs(t)+((ks-f13)+(1-c12)ke)xs(t).
Converting (3.4) into state-space form, we have(3.5)x̅̇s(t)=Asx̅s(t)+Bsu̅d(t),y̅(t)=Csx̅s(t),
where
(3.6)x̅s(t)=[xs(t)ẋs(t)],Cs=[10],Bs=[01/ms(1-c12)me],As=[01-(ks-f13)+(1-c12)kems+(1-c12)me-(bs-f14)+(1-c12)bems+(1-c12)me].
From (3.5), in order to predict udm(t+T(t)), we have to prognosticate xs(t+TR(t)) and ẋs(t+TR(t))  by forwarding into the future with TR(t)  units:(3.7)x̅̇s(t+TR(t))=(1+ṪR(t))Asx̅s(t+TR(t))+(1+ṪR(t))Bsu̅d(t+TR(t)),y̅(t+TR(t))=Csx̅s(t+TR(t)).
However, it is difficult to estimate ṪR(t). According o the characteristic of the Internet, we can model TR(t)  as uncertain parameters, and (3.7) becomes(3.8)x̅̇s(t+TR(t))=(As+ΔAs)x̅s(t+TR(t))+(Bs+ΔBs)u̅d(t+TR(t)),y̅(t+TR(t))=Csx̅s(t+TR(t)),
where Δt=ṪR(t) and ΔAs and ΔBs are modeled as uncertain parameters, ΔAs=ΔtAs and ΔBs=ΔtBs. In (3.8), when the time delay is constant, ṪR(t)=0, these uncertain parameters do not exist.Value ofx̅s(t+TR(t)) can be also calculated directly by the current master dynamics xm(t),ẋm(t), and fh(t)  via the following observer:(3.9)ż(t)=Asz(t)+Bs(f11x̅m(t)+c11fh(t))+L(y̅(t+TR(t))-ys(t)),ys(t)=Csz(t),
where L is the observer gain, x̅m(t)=[xm(t)ẋm(t)], and z(t)=[x̂s(t)x̂̇s(t)].Since the master output,y̅(t+TR(t)), cannot be measured, we can use the available adjustment of t-TL(t) as follows:(3.10)ż(t)=Asz(t)+Bs(f11x̅m(t)+c11fh(t))+L(y̅(t-TL(t))-ys(t-TR(t)-TL(t))),ys(t)=Csz(t).
Let the observing error e(t)=x̅s(t+TR(t)-z(t)), then (3.11)ė(t)=Ase(t)+ΔAsxs(t+TR(t))-LCse(t-TR(t)-TL(t))+ΔBs(f11x̅m(t)+c11fh(t))
RBF will on line calculate the smallest em(t), and the mth environment model which fits the best to the current environmental dynamics is selected for the system controller. Conditions for a stabilized and transparent controller are developed in the next section.


## 4. Controller Design

In this part, the controller design with guaranteed conditions for the system transparency and stability is established. The time forward state calculation for the slave state at(t+TR(t)) has been performed in the previous section. Using ûdm(t+T(t)) in (2.11) to take place of udm(t) in (2.1), we have:(4.1)fh(t)-(f21xm(t)+f22ẋm(t)+f23x̂s(t+TR(t))+f24x̂̇s(t+TR(t))+c21fh(t)+c22f̂e(t+TR(t)))=mmẍm(t)+bmẋm(t)+kmxm(t).
Rearranging (4.1) yields(4.2)mmẍm(t)+(bm+f22)ẋm(t)+(km+f21)xm(t)=-(f23x̂s(t+TR(t))+f24x̂̇s(t+TR(t))+c22f̂e(t+TR(t))).
Substituting (2.12) into (4.2) and modifying, we have(4.3)mmẍm(t)+(bm+f22)ẋm(t)+(km+f21)xm(t)=-((f23+c22ke)x̂s(t+TR(t))+(f24+c22be)x̂̇s(t+TR(t))+c22mex̂̈s(t+TR(t)))+(1-c21)fh(t).
Using z(t)  in (3.9) replacing x̂s(t)  in (4.3), we have(4.4)ẍm(t)+(bm+f22)mmẋm(t)+(km+f21)mmxm(t)=(1-c21)mmfh(t)-(Aez(t)+Beż(t)),
where(4.5)Ae=[(f23+c22ke)mm(f24+c22be)mm],Be=[0c22memm].
Substituting (3.9) into (4.4) and altering, we have(4.6)ẍm(t)+(bm+f22)mmẋm(t)+(km+f21)mmxm(t)=-(Ae+BeAs)z(t)+BeBs(f11x̅m(t)+c11fh(t))+BeLCse(t-t(T))+(1-c21)mmfh(t)
or(4.7)ẍm(t)+((bm+f22)mm+BeBsf12)ẋm(t)+((km+f21)mm+BeBsf11)xm(t)=((1-c21)mm+BeBsc11)fh(t)-((Ae+BeAs)z(t)+BeLCse(t-T(t))).
Converting (4.7) into state-space form, we have(4.8)x̅̇m(t)=Amx̅m(t)+Bmfh(t)-(Az(x̅s(t+TR(t))-e(t))+Bze(t-T(t))),ym(t)=Cmx̅m(t),
where
(4.9)Am=[01-(km+f21mm+BeBsf11)-(bm+f22mm+BeBsf12)],Cm=[10],(4.10)Bm=[01-c11mm+BeBsc11],Az=[0Ae+BeAs],Bz=[0BeLCs].
From (3.8), (3.11), and (4.8), the model of total system is established as(4.11)x(t)=Ax(t)+Atx(t-T(t))+Bfh(t),y(t)=Cx(t),
where
(4.12)x(t)=[x̅m(t)x̅s(t+TR(t))e(t)],A=[Am-AzAz(Bs+ΔBs)f11As+ΔAs0ΔBsf11ΔAsAs],C=[Cm000Cs0000],B=[Bm(Bs+ΔBs)c11ΔBsc11],At=[00-Bz00000-LCs].
The model of total system in (4.11) is now subject to conditions for the system transparency as referred to by Lawrence [4] that a teleoperation system achieves stability and ideal transparency if the following relation is maintained:(4.13)Fe(s)Xs(s)=Fh(s)Xm(s),
where Fe(s), Xs(s), Fh(s), and Xm(s)  are the Laplace transform of fe(t), xs(t), fh(t), and xm(t),  respectively.Substituting (2.2), (2.3), (2.8), and (2.10) into (4.13), following conditions for the system transparency yield:
(4.14)((1-c21)(ms+(1-c12)me)-c11c22me)me=(ms+(1-c12)me)mm,((1-c21)(ms+(1-c12)me)-c11c22me)be+(1-c12)(bs-f14+(1-c12)be-c11(f24+c22be))me=(bs-f14+(1-c12)be)mm+(ms+(1-c12)me)(bm+f22)+c22mef12,((1-c21)(ms+(1-c12)me)-c11c22me)ke+((1-c12)(bs-f14+(1-c12)be-c11(f24+c22be))be+(1-c21)(ks-f13+(1-c12)ke)-c11(f23+c22ke))me=(ks-f13+(1-c12)ke)mm+(bs-f14+(1-c12)be)(bm+f22)+(ms+(1-c12)me)(km+f21)+c22mef11+(f24+c22be)f12((1-c21)(bs-f14+(1-c12)be)-c11(f24+c22be))ke+(1-c21)(ks-f13+(1-c12)ke)-c11(f23+c22ke)be=(ks-f13+(1-c12)ke)(bm+f22)+(bs-f14+(1-c12)be)(bm+f21)+(f24+c22be)f11+(f23+c22ke)f12,((1-c21)(ks-f13+(1-c12)ke)-c11(f23+c22be))ke=(ks-f13+(1-c12)ke)(km+f21)+(f23+c22ke)f11.
Consequently, if (4.14) is feasible, the system will achieve the stability and ideal transparency. Simulations for this controller are examined in the next section.


## 5. Simulation Results

Simulation of this proposed model is developed with Matlab/Simulink for time-variant delays and environment uncertainties. The controller is also tested for the ability to reject disturbances while maintaining its closed-loop stability.In the simulation, the parameters of the master and slave manipulator are selected asmm=1.5 kg, bm=0.45 Ns/m, km=1 N/m, ms=1.5 kg, bs=0.45 Ns/m, and ks=1 N/m. It is assumed that the environment uncertainties consist of three models M1, M2, and M3 activated randomly.The parameters of environment modelM1 are selected as me1=1.1 kg, be1=0.6 Ns/m, and ke1=0.6 N/m. The initial controller parameters are chosen as c11=30, f11=2, f12=1, f21=1456, and f22=987. To assure the system transparency, other controller parameters are calculated as solutions of (4.14) as c12=-27.636, c21=-29, c22=28.636, f13=-1.8182, f14=-1.3682, f23=-1456.2, and f24=-986.63.Similarly, the parameters of environment modelM2 are selected as me2=1.0 kg, be2=0.4 Ns/m, and ke2=0.8 N/m. Then, other initial controller parameters are chosen as c11=20, f11=2, f12=1, f21=1500, and f22=1200. For transparency conditions, the remaining controller parameters are calculated as c12=-17.5, c21=-19, c22=18.5, f13=-2.2, f14=-1.15, f23=-1499.8, and f24=-1199.8.The parameters of environment modelM3 are selected as me3=1.5 kg, be3=0.7 Ns/m, and ke3=0.5 N/m. The initial controller parameters in this case are chosen as c11=30, f11=2, f12=1, f21=1497, and f22=1356. Conditions for the system transparency lead to other controller parameters as c12=-28, c21=-29, c22=29, f13=-1.5, f14=-1.25, f23=-1497.5, and f24=-1355.8.The time-variant delays in the communication channel are randomly selected with0<TR(t)≤5 s  and 0<TL(t)≤5 s. The three environmental uncertainties, M1, M2, and M3 are also randomly activated, and the RBF network is used to select the best fitted model. The maximum allowable bound of time delay, TMax_DeLay, for guaranteeing stability of the system therefore is the sum of the maximum of forward time delay, TMax_R(t), and the maximum of backward time delay, TMax_L(t),  TMax_DeLay≤TMax_R(t)+TMax_L(t)=10 s.The input human force,fh, with pulse generator activates the movement of the master device, xm. Three environment models, M1, M2, and M3, are activated at time from timeline from 0–30 s, 31–60 s, and 61–90 s, respectively. RBF identifies the environmental parameters and activates the suitable adaptive controllers. A white noise is also injected to the system to test the ability of the system to maintain its stability as shown in Figure 5.Figure 5
Matlab Simulink diagram.Results of the simulation for the time-variant delays and the environmental uncertainties are shown in Figure6. The subplots indicate the forces and movements of the master and the slave. The adaptive predictive controller is accurate, and the system is stable and in good performance since the slave profiles tracking well to the master amid the time-variant delays and the environment uncertainties.Position of master and slave with time-variant delays and environmental uncertainties.
(a)(b)Lastly, the model is tested for the ability to maintain its closed-loop stability with noise disturbances. Simulation results for the time-variant delays and the environmental uncertainties with noises are shown in Figure7.Position of master and slave with time-variant delays and noises.
(a)(b)Simulation results show that the system is stable under the white-noise disturbances. The outputs can track the input properly without steady-state error. The controller is robust to the disturbances.


## 6. Conclusion

This paper presents a novel theoretical framework to design a teleoperation system dealing with time-variant delays and environmental uncertainties. Using predictive strategies and RBF network, the stability and transparency of the system are guaranteed amid the noise disturbance. The RBF neural network is trained off line using a set of environmental models and selects the best fitted model in the current environment dynamics. Issue of communication uncertain delays in teleoperation is also addressed. A neural network-based multiple-model adaptive controller is proposed to design the time forward state observer. Simulations show that the system is stable and in good performance. However, the system becomes unstable if conditions for the stability and transparency are not feasible. Further analysis is needed for the effectiveness with respect to the achievable performance and reliability of this design.



---

*Source: 592054-2010-09-01.xml*