# A Novel Human-Like Control Framework for Mobile Medical Service Robot

Authors: Xin Zhang; Jiehao Li; Wen Qi; Xuanyi Zhou; Yingbai Hu; Hao Quan; Zhen Wang
Journal: Complexity
 (2020)

---

## Abstract

Recently, as a highly infectious disease of novel coronavirus (COVID-19) has swept the globe, more and more patients need to be isolated in the rooms of the hospitals, so how to deliver the meals or drugs to these infectious patients is the urgent work. It is a reliable and effective method to transport medical supplies or meals to patients using robots, but how to teach the robot to the destination and to enter the door like a human is an exciting task. In this paper, a novel human-like control framework for the mobile medical service robot is considered, where a Kinect sensor is used to manage human activity recognition to generate a designed teaching trajectory. Meanwhile, the learning technique of dynamic movement primitives (DMP) with the Gaussian mixture model (GMM) is applied to transfer the skill from humans to robots. A neural-based model predictive tracking controller is implemented to follow the teaching trajectory. Finally, some demonstrations are carried out in a hospital room to illustrate the superiority and effectiveness of the developed framework.


---

## Body

## 1. Introduction

In the past few months, a novel coronavirus has resulted in an ongoing outbreak of viral pneumonia in the world [1, 2]. More than 98000 people are known to be infected, and over 3400 deaths have been reported [3]. The symptoms of these patients include high temperature, cough, shortness of breath, and headache, and it is a highly infectious disease [4]. More and more patients need to be isolated in independent rooms of the hospitals. How to transport meals and medicines to patients and reduce the infection for medical staff simultaneously is a hot issue. It is a safe and useful method to transport medical supplies or meals to patients using robots. Therefore, this paper focuses on how to control the robot to the destination and to enter the door like a human at the same time.Human-robot skill transfer is currently one of the significant topics in human-assisted systems. Under different external environmental conditions, especially in the hospital room, the main challenge for assisted medical rescue robots is how to transport the medical supplies [5] safely. The robot system acquires the learning ability through the cognitive knowledge transmitted by humans [6]. At the same time, by learning the approach of human-in-the-loop, the performance of the robot is improved, making the system more intelligent [7, 8].Human-robot interaction (HRI) is dedicated to the development of more intelligent and anthropomorphic robots, which is a subregion of human-computer interaction that researches the interaction between man and robot [9, 10]. In some hazardous areas, to minimize staff participation, robots are required to perform operations. HRI technology can be used to perform remote operations on robots efficiently [11]. Therefore, not only is it widely used in the research of robotic systems, but it also plays an essential role in the implementation of robotic systems. Furthermore, in some particular activities, human-computer interaction plays an important role. For example, remote operation of human-machine interaction for medical robots is the safest method in medical processes, allowing the user to interact with the robot through tactile signals for coordinated feedback [12]. In order to improve surgeon performance, an experimental approach to characterize the human-robot-assisted surgery system is discussed. In vision-based robotic control, the visual impedance scheme was used to implement dynamic control at the activity level. To complete the integration of the vision servosystem and the conventional servosystem, Su et al. [13] proposed the visual impedance scheme in the control scheme; namely, the characteristics of the image were applied to the impedance equation. However, the flaw is that this research is still limited in terms of a self-adaptive decision. Besides, two themes are widely used in HRI [14], one of which was the human-computer interaction interface which was represented by the operation of the keyboard, and the other was the human-machine interaction represented by a touch screen operation. Therefore, this document focuses primarily on human-robot interaction with demonstration teaching.At the same time, another advanced technology for human-machine operation is the skill transmission through demonstration teaching [15, 16], and during this process, the motion control strategies and the generalized output [17] were learned to transfer the motor skills to the robot by the movement of the demonstrator. Behavior perception, behavior representation, and behavior reproduction are three processes that are imitating the process of learning. Some special feature methods can be used to program the learning process, such as Dynamic Motion Primitives (DMP) and Hidden Markov Models (HMM) [18, 19]. The stochastic models, like the Gaussian mixture model (GMM), have some powerful capability to code and process noise so that they have the ability to handle high-dimensional problems more effectively. The trajectory-level representation that is on the basis of the probability model uses the characteristics of the stochastic model to model the motion trajectory, thereby solving the problem more efficiently. Motion trajectory reproduction and motion control belong to the category of behavior reproduction, where trajectory reproduction is a process of transmitting coded data. It is mainly transmitted for some techniques of regression, such as Gaussian Process Regression (GPR) and Gaussian Hybrid Regression (GMR), and the feedback variable is a playback behavior learned from the presenter. In other words, it is a generalized output that maps to robot motion control for motion reproduction [20–22].The most widely used technologies for scientific research widely used in statistical models of human behavior are machine learning (ML) and deep learning (DL) technologies. In the previous work, in order to compare the recognition rates for identifying human activities, different combined sensors were adopted, and a deep convolutional neural network (DCNN) was applied to the HAR system [23]. In addition, the ML method is used to enhance HAR’s adaptive identification and real-time monitoring system to overcome time-consuming strategies. These classifiers have been proven to recognize more human actions in dynamic situations, thereby improving accuracy, enhancing robustness, and achieving time-saving effects [13, 24]. Moreover, a hybrid hierarchical classification algorithm combining DL and the method that is based on the threshold is proposed to differentiate complex events in order to calculate quickly. Although our previous research has proposed many effective frameworks, most acceptable results have been achieved with limited assumptions and conditions that cannot meet the complex environment in the medical room.In this article, a novel human-like control framework for mobile medical service robots is presented, where Kinect sensors are used to achieve human activity recognition to generate a designed teaching trajectory. At the same time, the learning technique of dynamic movement primitives (DMP) with the Gaussian mixture model (GMM) is applied to transfer the skill from human to machine. In addition, a neural-enhanced model predictive tracking controller is implemented to follow the teaching trajectory. Finally, some demonstrations are carried out in a medical room to account for the effectiveness and superiority of the framework. The main contribution of this paper is as described below:(1)
In order to control the mobile service robot to transport medical supplies or meals to the patient who suffers from the new coronavirus, a novel human-like control framework is discussed.(2)
Kinect sensors technology is applied to the skill transfer of the mobile medical service robot for collecting the movement points, and the method of DMP with GMM is used to congress the points.(3)
To efficiently track the teaching trajectory under uncertain disturbances, a neural network enhanced predictive tracking control scheme is presented. Also, some demonstrations are carried out to illustrate the developed structure.The structure of this essay is as described below: Section2 describes the overview of human-like control. Some demonstrations are discussed in Section 3. Finally, the conclusion and future point are summarized in Section 4.


## 2. Methodology

### 2.1. Human Activity Recognition Using Kinect Sensors

Human activity recognition technology can be applied to follow the position of an operator using a Kinect device. As shown in Figure1, the operator selects the depth message on the Kinect sensor, and the color vision is collected in the Kinect depth image [25]. To effectively construct the movement information, it is combined with the color image and the depth image in such a way that the origin is located at the center of the depth camera. Thus, we assume that the coordination system in camera space follows a right-hand convention [26].Figure 1
The overview of human activity recognition using Kinect sensors.In this case,ℳk=xk,yk,zk represents the three coordinates of the sequence of frame, and we define Di,j and ϱi,j as the depth image points and color image, respectively. Then, we can obtain the Bayes rule to evaluate the probability of ℳt|ϱ:(1)ℳt|ϱ=ℳt|ϱℳtℳϱ,where ℳϱ and ℳϱ|t denote the exercise dataset and prior probabilities of skin color, respectively.It is on account of self-occlusion or lacking of joint information at some stages that we need to incorporate other functions that can offer data about human shape so as to upgrade the precision of the classifier [6, 27]. We adopt orthogonal Cartesian planes on the depth map to obtain the positive 2D image and the profile obtained. Then, by converting Cartesian coordinates to polar coordinates, the silhouette of a person can be efficiently processed:(2)ℛi=xi−xj2−yi−yj2,θi=tan−1yi−yjxi−xj,where xi,yi and ℛi,θi represent the coordinates of the outline of the human body and Radius and angle in polar coordinates, respectively. Besides, xj,yj is the center coordinate of the human contour. The overall order of each activity with front and side views is averaged, and the average Smean from the initial frame to the final frame is defined as follows:(3)Smean=1T∑t=1TIx,y,z,t.


### 2.2. Trajectory Generation via DMP with GMM

After the Kinect device has collected the path information teaching by human demonstration, the mobile robot needs to learn the created trajectory [28]. The teaching trajectory is determined by dynamic movement primitive technology (DMP) and then rebuilt by the Gaussian mixture model (GMM) to generalize the movement trajectory:(4)ΨΦl=∑ξ=1ξΨξΨΦl|ξ,where Ψξ is the prior probability, ΨΦl/ξ is the conditional probability distribution, which follows the Gaussian distribution, and ξ is the number of Gaussian model distribution.Therefore, by using a Gaussian mixture model, the entire teaching dataset can be expressed as follows:(5)ΨΦl|ξ=NΦl,ψξ,∑ξ=12πE∑ξe−0.5Φl−ψξT∑ξ−1Φl−ψξ,where E is the dimension of the GMR and determined by πξ,ψξ,∑ξ.The Gaussian distribution can be addressed as(6)Φf,ξ|Φs,ξ∼Nψf,ξ′,∑f,ξ′,ψf,ξ′=ψf,ξ+∑fs,ξ∑s,ξ−1Φs,ξ−ψs,ξ,where ψξ=ψf,ξ,ψs,ξ and ∑ξ=∑s,f,ξ∑fs,ξ.Therefore, the averageψf′ and variance ∑f′ of GMR of the number of ξ Gaussian components can be evaluated as(7)ψf′=∑ξ=1Mηξψf,ξ′,∑f′=∑k=1Mηξ2∑f,ξ′,ζξ=GΦs|ξ∑ξ=1MGΦs|i,where ψf′ is the estimation variable and Φs is the corresponding space parameter. Φf′,ΦS is the generalized points, which produces a smooth movement trajectory under the covariance constraint ∑f′.The GMM model, including a multidimensional probability density function, is composed of multiple Gaussian probability density functions. The Gaussian model is only related to two parameters, the mean and variance. As we all know, different learning mechanisms can directly affect the accuracy, convergence, and stability of the model. Assume that anM-order GMM is weighted and summed with M Gaussian probability density functions:(8)PX|λ=∑n=1MΦnQnX,n=1,2,…,M,where X denotes D dimensional random vector and M is on behalf of the order of the model, while Phin represents the weight of each Gaussian component, satisfying sumn=1MPhin=1. Furthermore, mathcalQnX is each Gaussian component, which is a Gaussian probability density function of D dimension and can be expressed as follows:(9)QnX=12πD/2∑n1/2∗exp−0.5x−μnT∑n=1MX−μn,where mun is on behalf of the mean vector and sumn denotes the covariance matrix. Then, GMM can be expressed by the three parameters of mean vector, covariance matrix, and mixed weight. Therefore, the GMM can be described as(10)λ=μn,∑n,Φn,n=1,2,…,M.


### 2.3. Neural Approximation

In order to efficiently transfer the trajectory teaching by human demonstration, it is necessary to control the uncertain disturbance in the trajectory tracking process for the mobile robot [29–31]. To overcome the hidden safety hazards in scooter operation [32–34], a control scheme based on RBFNN was implemented for the elderly walker system. This scheme has certain disturbances and unknown dynamic characteristics. Design a constant smooth function GK:Rq⟶R to connect the approximation capability, where the RBFNN control scheme is applied for evaluating the dynamics of uncertainty, such as load friction and mechanism structure [35–37]:(11)GnnKin=JTΘKin,where Kin∈Ω⊂Rq represents the input of RBFNN; ΘKin and ΘiKin are the activation function depending on Gaussian function, respectively; and J=ξ1,ξ2,…,ξm∈Rm represents the weight in the hidden layer:(12)ΘiKin=exp−Kin−uiTKin−uiηi2,where i=1,2,…,m, ui=ui1,ui2,…,uiqT∈Rq, and ηi is the variance.Then,ΘKin can be defined as(13)ΘKin≤ϖ,where ϖ is a positive constant.Then, we have(14)GnnKin=J∗TΘKin+ε,where J∗ is the desired weight subjected to ΦKin⊂Rq and ε≤τ.Hence, we have(15)J∗=argminKin∈RqsupGnnKin−JTΘKin,where ΘKin denotes the activation function depending on the Gaussian function.


### 2.4. Neural-Based Model Predictive Tracking Control

Human motion points are collected by the Kinect sensor, and then the generated trajectory can be obtained with the method of DMP and GMM [18, 38, 39]. Finally, the next task of the mobile robot is to follow the teaching trajectory [40].Figure2 exhibits the kinematic model of the mobile medical service robot, where xr,yr and xf,yf denote the coordinate of the rear axis and front axis, respectively. P is the circle center, and R denotes the steering radius. L and M represent the wheel trajectory. vr and vf denote the rear speed and front speed, respectively. δf and φ denote the steering angle and yaw angle, respectively.Figure 2
The kinematic model and the dynamic model of the mobile medical service robot.The trajectory tracking control of the mobile scroll wheel system can be represented as(16)x˙ry˙rφ.=cosφsinφ0vr+001ω,where ξs=xr,yr,φT is the system state and uS=vr,ωT is the control state.Hence, the dynamic model of the mobile robot can be addressed as(17)mxr..=myr.Φc+Fa1cosδf+Fa2cosδf+Fa3+Fa4,myr..=−mxr.Φc+Fb1cosδf+Fb2cosδf+Fb3+Fb4,Izφ..=AFb1cosδf+Fb2cosδf−BFb3+Fb4M−Fa1cosδf+Fa2cosδf−Fa3+Fa4,where Fa1, Fa2, Fa3, and Fa4 are the wheel force of left front, right front, left rear, and right front, respectively. Φc denotes the center yaw velocity, and IZ represents the rotational inertia.Besides, we assume the following condition to evaluate the lateral force of the robot tire [14]:(18)Fb1=ψδFΓδF,Fb2=ψδBΓδB,ψδF=β+MΦrvx−δf,ψδB=β+MΦrvx,where ψδF and ψδB are tire cornering angle. ΓδF and ΓδB denote cornering stiffness and β is the slip angle.The tracking error can be addressed as(19)x˙e=x˙r−x˙d=−vdsinφdxr−xd+cosφdv−vd,y˙e=y˙r−y˙d=vdcosφdyr−yd+sinφdv−vd,φ.e=φ.−φ.d=vdLcos2δdδ−δd+tanδdLv−vd.Then, we discretize the error function as(20)X˜n+1=ℋn,tX˜n+Kn,tu˜n,subjected to ℋn,t=10−vdTsinφd01vdTcosφd001,Kn,t=Tcosφd0Tsinφd0tanδd/LTvd/Lcos2δdT and T is the sampling time.In order to reliably and smoothly grasp the desired trajectory, state errors and control parameters must be constrained:(21)Vn=∑l=1NX˜Tn+l|nZn+l+u˜Tn+l−1ℱu˜n+l−1,where Z and ℱ are weighting factors, NP is the prediction horizon, and Ne is the control horizon. Then, the actual control variable can be determined as(22)ut=ut−1+Δut∗.It is on account of considering the safety and stability of the robot [41, 42] that it is of necessity to restrict the control limit and control increment. Combined with the mobile robot system, the control constraint can be presented as follows:(23)−1.0−45≤u≤1.045,−0.1−0.2≤ΔU≤0.10.2.Based on the overall control scheme, the framework of neural approximation for tracking control using DMP with GMM is shown in Figure3. The Kinect sensor detects the human movement points and then generates the teaching trajectory using the technology of DMP with GMM. Then, the neural-based model predictive tracking controller is carried out to realize the path following.Figure 3
Block diagram of neural fuzzy-based tracking control.


## 2.1. Human Activity Recognition Using Kinect Sensors

Human activity recognition technology can be applied to follow the position of an operator using a Kinect device. As shown in Figure1, the operator selects the depth message on the Kinect sensor, and the color vision is collected in the Kinect depth image [25]. To effectively construct the movement information, it is combined with the color image and the depth image in such a way that the origin is located at the center of the depth camera. Thus, we assume that the coordination system in camera space follows a right-hand convention [26].Figure 1
The overview of human activity recognition using Kinect sensors.In this case,ℳk=xk,yk,zk represents the three coordinates of the sequence of frame, and we define Di,j and ϱi,j as the depth image points and color image, respectively. Then, we can obtain the Bayes rule to evaluate the probability of ℳt|ϱ:(1)ℳt|ϱ=ℳt|ϱℳtℳϱ,where ℳϱ and ℳϱ|t denote the exercise dataset and prior probabilities of skin color, respectively.It is on account of self-occlusion or lacking of joint information at some stages that we need to incorporate other functions that can offer data about human shape so as to upgrade the precision of the classifier [6, 27]. We adopt orthogonal Cartesian planes on the depth map to obtain the positive 2D image and the profile obtained. Then, by converting Cartesian coordinates to polar coordinates, the silhouette of a person can be efficiently processed:(2)ℛi=xi−xj2−yi−yj2,θi=tan−1yi−yjxi−xj,where xi,yi and ℛi,θi represent the coordinates of the outline of the human body and Radius and angle in polar coordinates, respectively. Besides, xj,yj is the center coordinate of the human contour. The overall order of each activity with front and side views is averaged, and the average Smean from the initial frame to the final frame is defined as follows:(3)Smean=1T∑t=1TIx,y,z,t.


## 2.2. Trajectory Generation via DMP with GMM

After the Kinect device has collected the path information teaching by human demonstration, the mobile robot needs to learn the created trajectory [28]. The teaching trajectory is determined by dynamic movement primitive technology (DMP) and then rebuilt by the Gaussian mixture model (GMM) to generalize the movement trajectory:(4)ΨΦl=∑ξ=1ξΨξΨΦl|ξ,where Ψξ is the prior probability, ΨΦl/ξ is the conditional probability distribution, which follows the Gaussian distribution, and ξ is the number of Gaussian model distribution.Therefore, by using a Gaussian mixture model, the entire teaching dataset can be expressed as follows:(5)ΨΦl|ξ=NΦl,ψξ,∑ξ=12πE∑ξe−0.5Φl−ψξT∑ξ−1Φl−ψξ,where E is the dimension of the GMR and determined by πξ,ψξ,∑ξ.The Gaussian distribution can be addressed as(6)Φf,ξ|Φs,ξ∼Nψf,ξ′,∑f,ξ′,ψf,ξ′=ψf,ξ+∑fs,ξ∑s,ξ−1Φs,ξ−ψs,ξ,where ψξ=ψf,ξ,ψs,ξ and ∑ξ=∑s,f,ξ∑fs,ξ.Therefore, the averageψf′ and variance ∑f′ of GMR of the number of ξ Gaussian components can be evaluated as(7)ψf′=∑ξ=1Mηξψf,ξ′,∑f′=∑k=1Mηξ2∑f,ξ′,ζξ=GΦs|ξ∑ξ=1MGΦs|i,where ψf′ is the estimation variable and Φs is the corresponding space parameter. Φf′,ΦS is the generalized points, which produces a smooth movement trajectory under the covariance constraint ∑f′.The GMM model, including a multidimensional probability density function, is composed of multiple Gaussian probability density functions. The Gaussian model is only related to two parameters, the mean and variance. As we all know, different learning mechanisms can directly affect the accuracy, convergence, and stability of the model. Assume that anM-order GMM is weighted and summed with M Gaussian probability density functions:(8)PX|λ=∑n=1MΦnQnX,n=1,2,…,M,where X denotes D dimensional random vector and M is on behalf of the order of the model, while Phin represents the weight of each Gaussian component, satisfying sumn=1MPhin=1. Furthermore, mathcalQnX is each Gaussian component, which is a Gaussian probability density function of D dimension and can be expressed as follows:(9)QnX=12πD/2∑n1/2∗exp−0.5x−μnT∑n=1MX−μn,where mun is on behalf of the mean vector and sumn denotes the covariance matrix. Then, GMM can be expressed by the three parameters of mean vector, covariance matrix, and mixed weight. Therefore, the GMM can be described as(10)λ=μn,∑n,Φn,n=1,2,…,M.


## 2.3. Neural Approximation

In order to efficiently transfer the trajectory teaching by human demonstration, it is necessary to control the uncertain disturbance in the trajectory tracking process for the mobile robot [29–31]. To overcome the hidden safety hazards in scooter operation [32–34], a control scheme based on RBFNN was implemented for the elderly walker system. This scheme has certain disturbances and unknown dynamic characteristics. Design a constant smooth function GK:Rq⟶R to connect the approximation capability, where the RBFNN control scheme is applied for evaluating the dynamics of uncertainty, such as load friction and mechanism structure [35–37]:(11)GnnKin=JTΘKin,where Kin∈Ω⊂Rq represents the input of RBFNN; ΘKin and ΘiKin are the activation function depending on Gaussian function, respectively; and J=ξ1,ξ2,…,ξm∈Rm represents the weight in the hidden layer:(12)ΘiKin=exp−Kin−uiTKin−uiηi2,where i=1,2,…,m, ui=ui1,ui2,…,uiqT∈Rq, and ηi is the variance.Then,ΘKin can be defined as(13)ΘKin≤ϖ,where ϖ is a positive constant.Then, we have(14)GnnKin=J∗TΘKin+ε,where J∗ is the desired weight subjected to ΦKin⊂Rq and ε≤τ.Hence, we have(15)J∗=argminKin∈RqsupGnnKin−JTΘKin,where ΘKin denotes the activation function depending on the Gaussian function.


## 2.4. Neural-Based Model Predictive Tracking Control

Human motion points are collected by the Kinect sensor, and then the generated trajectory can be obtained with the method of DMP and GMM [18, 38, 39]. Finally, the next task of the mobile robot is to follow the teaching trajectory [40].Figure2 exhibits the kinematic model of the mobile medical service robot, where xr,yr and xf,yf denote the coordinate of the rear axis and front axis, respectively. P is the circle center, and R denotes the steering radius. L and M represent the wheel trajectory. vr and vf denote the rear speed and front speed, respectively. δf and φ denote the steering angle and yaw angle, respectively.Figure 2
The kinematic model and the dynamic model of the mobile medical service robot.The trajectory tracking control of the mobile scroll wheel system can be represented as(16)x˙ry˙rφ.=cosφsinφ0vr+001ω,where ξs=xr,yr,φT is the system state and uS=vr,ωT is the control state.Hence, the dynamic model of the mobile robot can be addressed as(17)mxr..=myr.Φc+Fa1cosδf+Fa2cosδf+Fa3+Fa4,myr..=−mxr.Φc+Fb1cosδf+Fb2cosδf+Fb3+Fb4,Izφ..=AFb1cosδf+Fb2cosδf−BFb3+Fb4M−Fa1cosδf+Fa2cosδf−Fa3+Fa4,where Fa1, Fa2, Fa3, and Fa4 are the wheel force of left front, right front, left rear, and right front, respectively. Φc denotes the center yaw velocity, and IZ represents the rotational inertia.Besides, we assume the following condition to evaluate the lateral force of the robot tire [14]:(18)Fb1=ψδFΓδF,Fb2=ψδBΓδB,ψδF=β+MΦrvx−δf,ψδB=β+MΦrvx,where ψδF and ψδB are tire cornering angle. ΓδF and ΓδB denote cornering stiffness and β is the slip angle.The tracking error can be addressed as(19)x˙e=x˙r−x˙d=−vdsinφdxr−xd+cosφdv−vd,y˙e=y˙r−y˙d=vdcosφdyr−yd+sinφdv−vd,φ.e=φ.−φ.d=vdLcos2δdδ−δd+tanδdLv−vd.Then, we discretize the error function as(20)X˜n+1=ℋn,tX˜n+Kn,tu˜n,subjected to ℋn,t=10−vdTsinφd01vdTcosφd001,Kn,t=Tcosφd0Tsinφd0tanδd/LTvd/Lcos2δdT and T is the sampling time.In order to reliably and smoothly grasp the desired trajectory, state errors and control parameters must be constrained:(21)Vn=∑l=1NX˜Tn+l|nZn+l+u˜Tn+l−1ℱu˜n+l−1,where Z and ℱ are weighting factors, NP is the prediction horizon, and Ne is the control horizon. Then, the actual control variable can be determined as(22)ut=ut−1+Δut∗.It is on account of considering the safety and stability of the robot [41, 42] that it is of necessity to restrict the control limit and control increment. Combined with the mobile robot system, the control constraint can be presented as follows:(23)−1.0−45≤u≤1.045,−0.1−0.2≤ΔU≤0.10.2.Based on the overall control scheme, the framework of neural approximation for tracking control using DMP with GMM is shown in Figure3. The Kinect sensor detects the human movement points and then generates the teaching trajectory using the technology of DMP with GMM. Then, the neural-based model predictive tracking controller is carried out to realize the path following.Figure 3
Block diagram of neural fuzzy-based tracking control.


## 3. Results and Discussions

In this section, the overview scenario of the medical room to transport the meals for patients is presented in Figure4. There are two Kinect sensors (XBOX 360) used in this demonstration. The surgical medical robot (LWR4+, KUKA, Germany) is used to feed the meals to patients, where the haptic manipulator (SIGMA 7, Force Dimension, Switzerland) is applied to control the KUKA arm remotely. The main purpose of this demonstration is that the developed mobile medical service robot can safely transport the meals or medicines to the medical bed like a human without collisions.Figure 4
The overview scenario of the medical room to transport the meals for patients.The Kinect sensor can detect human activity points and generate a teaching trajectory based on the method of DMP and GMM. Then, the mobile robot can follow the teaching trajectory via human demonstration. The result of the learning method, including the DMP, GMM, and the regression result of teaching trajectory, is displayed in Figures5 and 6. It is noted that there are two demonstrations considered in this section, which aims to evaluate the proposed framework for mobile medical service robot in skill transfer via teaching by demonstration.Figure 5
The regression result of teaching by demonstration using DMP with GMM (demonstration 1): (a) DMP to encode the trajectory points, (b) Gaussian components of GMM, and (c) regression results.
(a)(b)(c)Figure 6
The regression result of teaching by demonstration using DMP with GMM (demonstration 2): (a) DMP to encode the trajectory points, (b) Gaussian components of GMM, and (c) regression results.
(a)(b)(c)Figure7 exhibits teaching results of demonstration 1 in x-position, y-position, x-error, y-error, robot tracking velocity, roll angle, pitch angle, and tracking performance. It can be concluded that the mobile medical service robot can follow the teaching trajectory collected by Kinect sensors. The y-position error and x-position error can be constrained in a reasonable range within ±0.03 meters, indicating that the mobile robot can avoid the medical devices and surgeons. On the other hand, because of the neural-based predictive tracking controller, the velocity response of the mobile robot under uncertain disturbance is smooth. In particular, the roll angle and pitch angle can maintain a stable range.Figure 7
Teaching results of demonstration 1 inx-position, y-position, x-error, y-error, robot tracking velocity, roll angle, pitch angle, and tracking performance.In addition, to further illustrate the improvement of skill transfer scheme using multisensors fusion technology, demonstration 2 to avoid obstacles such as medical devices and medical staff is carried out. Figure8 displays the teaching performance in x-position, y-position, x-error, y-error, robot tracking velocity, roll angle, pitch angle, and tracking performance. From the tracking performance of x-position and y-position, the mobile medical service robot can efficiently follow the teaching trajectory and avoid obstacles. The x-position error and y-position error also can be maintained at a high accuracy, which is within ±0.06 meters in x-position and ±0.03 meters in y-position. At the same time, the neural-based predictive controller can constrain the mobile robot body, and the pitch angle and roll angle are within ±0.02 degrees and ±0.03 degrees, respectively.Figure 8
Teaching results of demonstration 2 inx-position, y-position, x-error, y-error, robot tracking velocity, roll angle, pitch angle, and tracking performance.


## 4. Conclusion

In this paper, a novel human-like control framework is implemented to control a mobile service robot using a Kinect sensor and DMP with GMM. It aims to bridge the human activity recognition techniques and assist the mobile medical service robot and allows the robot to cooperate with the medical staff. The Kinect sensor is used to detect human activities to generate a set of movement points, and then the teaching method including dynamic movement primitives with the Gaussian mixture model can generate the desired trajectory. To achieve stable tracking, a model predictive tracking control scheme based on neural networks is implemented to follow the teaching trajectory. Finally, some demonstrations are carried out in a medical room to validate the effectiveness and superiority of the developed framework.Human-machine collaborative control based on the Internet of Things (IoT) is the future research direction. In our lasted work [43], we have successfully used IoT technology to exploit the best action in human-robot interaction for the surgical KUKA robot. Instead of utilizing compliant swivel motion, HTC VIVE PRO controllers, used as the Internet of Things technology, are adopted to detect the collision, and a virtual force is applied on the elbow of the robot, enabling a smooth rotation for human-robot interaction. Future work, combined with the IoT technology and multisensors, the concept of the intelligent medical room, will be considered to strengthen the human-robot cooperation.



---

*Source: 2905841-2020-10-27.xml*