A Scaled Bilateral Teleoperation System for Robotic-Assisted Surgery with Time Delay

The master-slave teleoperated robotic systems have advanced the surgeries in the past decades. Time delay is usually caused due to the data transmission between communication channel connecting the master and slave in bilateral teleoperation, which is crucial because even small time delay could destabilize the whole teleoperation system. Motivated to solve the instability caused by time delay in bilateral teleoperation, wave variable transformation (WVT) structure has been proposed to passivate the delayed communication channel. However, conventional WVT structure provides poor velocity, position and force tracking performances which are not sufficient for surgical applications. In this paper, a new wave variable compensation (WVC) structure is proposed to improve the tracking performances with less conservative condition and comprehensive analysis to keep stable and improved tracking performance is also provided. In order to better facilitate certain surgical procedures with special requirements, e.g. robotic-assisted neurosurgery, velocity/position and force scalings are designed in the proposed structure with guaranteed system passivity, and transparency of the scaled WVC structure is also analyzed. Simulation and experimental studies were carried out to verify the performance of the proposed structure with time delay. System performance comparisons with several existing wave based bilateral teleoperation structures are also provided through simulation studies to show the improvements brought by the proposed teleoperation structure.

Related papers:

title: Pre‐clinical remote telesurgery trial of a da Vinci telesurgery prototype
abstract: The objective of this study was to perform a pre‐clinical remote telesurgery trial of a da Vinci® telesurgery prototype on a surgical‐grade virtual private network.

title: The role of haptic feedback in laparoscopic simulation training.
abstract: Laparoscopic virtual reality simulators are becoming a ubiquitous tool in resident training and assessment. These devices provide the operator with various levels of realism, including haptic (or force) feedback. However, this feature adds significantly to the cost of the devices, and limited data exist assessing the value of haptics in skill acquisition and development. Utilizing the Laparoscopy VR (Immersion Medical, Gaithersburg, MD), we hypothesized that the incorporation of force feedback in the simulated operative environment would allow superior trainee performance compared with performance of the same basic skills tasks in a non-haptic model.



Ten medical students with minimal laparoscopic experience and similar baseline skill levels as proven by performance of two fundamentals of laparoscopic surgery (FLS) tasks (peg transfer and cutting drills) voluntarily participated in the study. Each performed two tasks, analogous to the FLS drills, on the Laparoscopy VR at 3 levels of difficulty, based on the established settings of the manufacturer. After achieving familiarity with the device and tasks, the students completed the drills both with and without force feedback. Data on completion time, instrument path length, right and left hand errors, and grasping tension were analyzed. The scores in the haptic-enhanced simulation environment were compared with the scores in the non-haptic model and analyzed utilizing Student's t-test.



The peg transfer drill showed no difference in performance between the haptic and non-haptic simulations for all metrics at all three levels of difficulty. For the more complex cutting exercise, the time to complete the tasks was significantly shorter when force feedback was provided, at all levels of difficulty (158+/-56 versus 187+/-51 s, 176+/-49 versus 222+/-68 s, and 275+/-76 versus 422+/-220 s, at levels 1, 2, and 3, respectively, P<0.05). Data on instrument path length, grasping tension, and errors showed a trend toward a benefit from haptics at all difficulty levels, but this difference did not achieve statistical significance.



In the more advanced tasks, haptics allowed superior precision, resulting in faster completion of tasks and a trend toward fewer technical errors. In the more basic tasks, haptic-enhanced simulation did not demonstrate an appreciable performance improvement among our trainees. These data suggest that the additional expense of haptic-enhanced laparoscopic simulators may be justified for advanced skill development in surgical trainees as simulator technology continues to improve.

title: A Framework for the Design of a Novel Haptic-Based Medical Training Simulator
abstract: This paper presents a framework for the design of a haptic-based medical ultrasound training simulator. The proposed simulator is composed of a PHANToM haptic device and a modular software package that allows for visual feedback and kinesthetic interactions between an operator and multimodality image databases. The system provides real-time ultrasound images in the same fashion as a typical ultrasound machine, enhanced with corresponding augmented computerized tomographic (CT) and/or MRI images. The proposed training system allows trainees to develop radiology techniques and knowledge of the patient's anatomy with minimum practice on live patients, or in places or at times when radiology devices or patients with rare cases may not be available. Low-level details of the software structure that can be migrated to other similar medical simulators are described. A preliminary human factors study, conducted on the prototype of the developed simulator, demonstrates the potential usage of the system for clinical training.

title: Towards advanced robotic manipulation for nuclear decommissioning: A pilot study on tele-operation and autonomy
abstract: We present early pilot-studies of a new international project, developing advanced robotics to handle nuclear waste. Despite enormous remote handling requirements, there has been remarkably little use of robots by the nuclear industry. The few robots deployed have been directly teleoperated in rudimentary ways, with no advanced control methods or autonomy. Most remote handling is still done by an aging workforce of highly skilled experts, using 1960s style mechanical Master-Slave devices. In contrast, this paper explores how novice human operators can rapidly learn to control modern robots to perform basic manipulation tasks; also how autonomous robotics techniques can be used for operator assistance, to increase throughput rates, decrease errors, and enhance safety. We compare humans directly teleoperating a robot arm, against human-supervised semi-autonomous control exploiting computer vision, visual servoing and autonomous grasping algorithms. We show how novice operators rapidly improve their performance with training; suggest how training needs might scale with task complexity; and demonstrate how advanced autonomous robotics techniques can help human operators improve their overall task performance. An additional contribution of this paper is to show how rigorous experimental and analytical methods from human factors research, can be applied to perform principled scientific evaluations of human test-subjects controlling robots to perform practical manipulative tasks.

title: Use of remotely operated marine vehicles at Minamisanriku and Rikuzentakata Japan for disaster recovery
abstract: Three underwater remotely operated vehicles (ROVs) were used over a five day period to inspect critical infrastructure and to assist with victim search and recovery at six sites in the Iwate Prefecture following the Tohoku Earthquake and Tsunami. Unmanned marine vehicles have been used since 2004 for disaster recovery operations but in limited applications, in single areas, and in short deployment durations. The joint IRS-CRASAR deployment matched robots for missions specified by civilian municipalities and the Japanese Coast Guard. The ROVs successfully allowed a fishing port to be re-opened and searched for victims trapped underwater in five different locations in varying areas (marinas, bridge debris, and waterfront residential areas) that could not be searched by manual divers. From a scientific perspective, the deployment provides a corpus of 15 hours of data of how rescue robots can be used. It illustrates that rescue robots are i) valuable for both economic and victim recovery, not just response, ii) that disaster robots need to be optimized for the unique missions and stakeholder needs, and iii) that human-robot interaction remains a challenge. This paper also identifies new areas for research: computer vision and cognitive engineering, cyber-physical systems, heterogeneous multi-robot coordination, human-robot interaction, simulation and geographical information systems.

title: Bilateral teleoperation: An historical survey
abstract: This survey addresses the subject of bilateral teleoperation, a research stream with more than 50 years of history and one that continues to be a fertile ground for theoretical exploration and many applications. We focus on the control theoretic approaches that have been developed to address inherent control problems such as delays and information loss. Exposure to several concurrent applications is provided, and possible future trends are outlined.

title: From Passive Tool Holders to Microsurgeons: Safer, Smaller, Smarter Surgical Robots
abstract: Within only a few decades from its initial introduction, the field of surgical robotics has evolved into a dynamic and rapidly growing research area with increasing clinical uptake worldwide. Initially introduced for stereotaxic neurosurgery, surgical robots are now involved in an increasing number of procedures, demonstrating their practical clinical potential while propelling further advances in surgical innovations. Emerging platforms are also able to perform complex interventions through only a single-entry incision, and navigate through natural anatomical pathways in a tethered or wireless fashion. New devices facilitate superhuman dexterity and enable the performance of surgical steps that are otherwise impossible. They also allow seamless integration of microimaging techniques at the cellular level, significantly expanding the capabilities of surgeons. This paper provides an overview of the significant achievements in surgical robotics and identifies the current trends and future research directions of the field in making surgical robots safer, smaller, and smarter.

title: Emerging Robotic Platforms for Minimally Invasive Surgery
abstract: Recent technological advances in surgery have resulted in the development of a range of new techniques that have reduced patient trauma, shortened hospitalization, and improved diagnostic accuracy and therapeutic outcome. Despite the many appreciated benefits of minimally invasive surgery (MIS) compared to traditional approaches, there are still significant drawbacks associated with conventional MIS including poor instrument control and ergonomics caused by rigid instrumentation and its associated fulcrum effect. The use of robot assistance has helped to realize the full potential of MIS with improved consistency, safety and accuracy. The development of articulated, precision tools to enhance the surgeon's dexterity has evolved in parallel with advances in imaging and human-robot interaction. This has improved hand-eye coordination and manual precision down to micron scales, with the capability of navigating through complex anatomical pathways. In this review paper, clinical requirements and technical challenges related to the design of robotic platforms for flexible access surgery are discussed. Allied technical approaches and engineering challenges related to instrument design, intraoperative guidance, and intelligent human-robot interaction are reviewed. We also highlight emerging designs and research opportunities in the field by assessing the current limitations and open technical challenges for the wider clinical uptake of robotic platforms in MIS.

title: Medical Robotics and Computer-Integrated Surgery
abstract: The growth of medical robotics since the mid1980s has been striking. From a few initial efforts in stereotactic brain surgery, orthopaedics, endoscopic surgery, microsurgery, and other areas, the field has expanded to include commercially marketed, clinically deployed systems, and a robust and exponentially expanding research community. This chapter will discuss some major themes and illustrate them with examples from current and past research. Further reading providing a more comprehensive review of this rapidly expanding field is suggested in Sect. 52.4. Medical robots may be classified in many ways: by manipulator design (e.g., kinematics, actuation); by level of autonomy (e.g., preprogrammed versus teleoperation versus constrained cooperative control), by targeted anatomy or technique (e.g., cardiac, intravascular, percutaneous, laparoscopic, microsurgical); or intended operating environment (e.g., in-scanner, conventional operating room). In this chapter, we have chosen to focus on the role of medical robots within the context of larger computer-integrated systems including presurgical planning, intraoperative execution, and postoperative assessment and follow-up. First, we introduce basic concepts of computerintegrated surgery, discuss critical factors affecting the eventual deployment and acceptance of medical robots, and introduce the basic system paradigms of surgical computer-assisted planning, registration, execution, monitoring, and assessment (CAD/CAM) and surgical assistance. In subsequent sections, we provide an overview of the technology of medical robot systems and discuss examples of our basic system paradigms, with brief additional discussion topics of remote telesurgery and robotic surgical simulators. We conclude with some thoughts on future research directions and provide suggested further reading. 52.1 Core Concepts ....................................... 1200 52.1.1 Medical Robotics, Computer-Integrated Surgery, and Closed-Loop Interventions ...... 1200 52.1.2 Factors Affecting the Acceptance of Medical Robots......................... 1200 52.1.3 Medical Robotics System Paradigms: Surgical CAD/CAM and Surgical Assistance ................. 1202

title: Neurosurgical robotics: a review of brain and spine applications
abstract: Neurosurgery has traditionally been at the forefront of advancing technologies, adapting new techniques and devices successfully in an effort to increase the safety and efficacy of brain and spine surgery. Among these adaptations are surgical robotics. This paper reviews some of the more promising systems in neurosurgical robotics, including brain and spine applications in use and in development. The purpose of the discussion is twofold—to discuss the most promising models for neurosurgical applications, and to discuss some of the pitfalls of robotic neurosurgery given the unique anatomy of the brain and spine.

title: A Telerobotic System for Transnasal Surgery
abstract: Mechanics-based models of concentric tube continuum robots have recently achieved a level of sophistication that makes it possible to begin to apply these robots to a variety of real-world clinical scenarios. Endonasal skull base surgery is one such application, where their small diameter and tentacle-like dexterity are particularly advantageous. In this paper, we provide the medical motivation for an endonasal surgical robot featuring concentric tube manipulators, and describe our model-based design and teleoperation methods, as well as a complete system incorporating image guidance. Experimental demonstrations using a laparoscopic training task, a cadaver reachability study, and a phantom tumor resection experiment illustrate that both novice and expert users can effectively teleoperate the system, and that skull base surgeons can use the robot to achieve their objectives in a realistic surgical scenario.

title: Computer-enhanced "robotic" cardiac surgery: experience in 148 patients.
abstract: A computer-enhanced instrumentation system was used in 148 patients to minimize access in cardiac surgical procedures.



The da Vinci telemanipulation system (Intuitive Surgical, Mountain View, Calif) provides a high-resolution 3-dimensional videoscopic image and allows remote, tremor-free, and scaled control of endoscopic surgical instruments with 6 degrees of freedom. By April 2000, the system had been used in 131 patients for coronary artery bypass grafting and 17 patients for mitral valve repair. In the coronary bypass group, the system was used in one of three ways: (1) to take down the internal thoracic artery followed by a minimally invasive direct coronary bypass procedure (n = 81); (2) to perform the anastomosis between the internal thoracic artery and the left anterior descending coronary artery in standard-sternotomy coronary bypass (n = 15); or (3) for total endoscopic coronary artery bypass grafting to anastomose the left internal thoracic artery to the left anterior descending on the arrested heart (n = 27) or the beating heart (n = 8). In 17 patients with nonischemic mitral valve insufficiency the mitral valve was repaired. Closed-chest cardiopulmonary bypass with cardioplegic arrest (Port-Access technique; Heartport, Inc, Redwood City, Calif) was used for arrested-heart total endoscopic coronary bypass and mitral valve repair.



The da Vinci system allows for precise tissue handling and enables the endoscopic performance of cardiac surgical tasks that require a high degree of dexterity (coronary anastomosis, mitral valve repair). No technical mishaps have occurred. The internal thoracic artery was successfully taken down in 79 of 81 patients in the group undergoing minimally invasive coronary bypass and, after a steep learning curve, is currently performed in less than 40 minutes. The postoperative patency rate is 96.3%. Total endoscopic coronary bypass was completed in 22 of 27 cases with 95.4% patency as demonstrated by angiography at 3 months' follow-up. Closed-chest endoscopic beating-heart bypass grafting was successfully performed in 2 out of 8 patients with the use of a new endoscopic stabilizer. In the group having mitral valve repair, primary endoscopic computer-enhanced repair was successfully completed in 14 of 17 patients; three others had to be changed to a standard endoscopic technique, including 1 who required valve replacement. At 3 months' follow-up, 1 additional patient underwent early reoperation for recurrent mitral insufficiency. Overall early and late mortality in this cohort of 148 patients was 2.0% and was not related to the use of the system.



In conclusion, computer-enhanced endoscopic cardiac surgery can be performed safely in selected patients. Internal thoracic artery takedown is now routinely performed with good results. Total endoscopic coronary bypass is feasible on the arrested heart but does not offer a major benefit over the minimally invasive direct approach because cardiopulmonary bypass is still required. The early clinical experience with closed-chest beating-heart bypass grafting outlines the limitations of this approach despite some procedural success.

title: Methods for haptic feedback in teleoperated robot-assisted surgery
abstract: Teleoperated minimally invasive surgical robots can significantly enhance a surgeon's accuracy, dexterity and visualization. However, current commercially available systems do not include significant haptic (force and tactile) feedback to the operator. This paper describes experiments to characterize this problem, as well as several methods to provide haptic feedback in order to improve surgeon's performance. There exist a variety of sensing and control methods that enable haptic feedback, although a number of practical considerations, e.g. cost, complexity and biocompatibility, present significant challenges. The ability of teleoperated robot-assisted surgical systems to measure and display haptic information leads to a number of additional exciting clinical and scientific opportunities, such as active operator assistance through "virtual fixtures" and the automatic acquisition of tissue properties.

title: Haptics for Teleoperated Surgical Robotic Systems
abstract: An important obstacle in Minimally Invasive Surgery (MIS) is the significant degradation of haptic feedback (sensation of touch) to the surgeon about surgical instrument s interaction with tissue. This monograph is concerned with devices and methods required for incorporating haptic feedback in master-slave robotic MIS systems. In terms of devices, novel mechanisms are designed including a surgical end-effector (slave) with full force sensing capabilities and a surgeon-robot interface (master) with full force feedback capabilities. Using the master-slave system, various haptic teleoperation control schemes are compared in terms of stability and performance, and passivity-based time delay compensation for haptic teleoperation over a long distance is investigated. The monograph also compares haptic feedback with visual feedback and with substitution for haptic feedback by other sensory cues in terms of surgical task performance.  Sensorized Surgical Effector (Slave); Haptic User Interface (Master); Unilateral Teleoperation Control; Bilateral Teleoperation Control; Substitution for Haptic Feedback: Bilateral Teleoperation Control Under Time Delay.

title: Sensory Subtraction in Robot-Assisted Surgery: Fingertip Skin Deformation Feedback to Ensure Safety and Improve Transparency in Bimanual Haptic Interaction
abstract: This study presents a novel approach to force feedback in robot-assisted surgery. It consists of substituting haptic stimuli, composed of a kinesthetic component and a skin deformation, with cutaneous stimuli only. The force generated can then be thought as a subtraction between the complete haptic interaction, cutaneous, and kinesthetic, and the kinesthetic part of it. For this reason, we refer to this approach as sensory subtraction. Sensory subtraction aims at outperforming other nonkinesthetic feedback techniques in teleoperation (e.g., sensory substitution) while guaranteeing the stability and safety of the system. We tested the proposed approach in a challenging 7-DoF bimanual teleoperation task, similar to the Pegboard experiment of the da Vinci Skills Simulator. Sensory subtraction showed improved performance in terms of completion time, force exerted, and total displacement of the rings with respect to two popular sensory substitution techniques. Moreover, it guaranteed a stable interaction in the presence of a communication delay in the haptic loop.

title: Implementation of Tactile Sensing for Palpation in Robot-Assisted Minimally Invasive Surgery: A Review
abstract: Robot-assisted minimally invasive surgery (RMIS) made it possible to perform a number of medical manipulations with reduced patient trauma and better accuracy. Various devices, including tactile sensors, have been developed in recent years to enhance the quality of this procedure. The objective of this paper is to review the latest advancements and challenges in the development of tactile sensing devices designed for surgical applications. In particular, the focus is on palpation and probing devices that can be potentially used in RMIS. In addition, we explore the aspects that should be taken into account when designing tactile sensors for RMIS, incorporating biological inspiration of tactile sensing, features of manual palpation, requirements of RMIS. We provide an overview of recommendations for the development of tactile sensing devices, especially in the context of RMIS.

title: Adaptive Fuzzy Control for Multilateral Cooperative Teleoperation of Multiple Robotic Manipulators Under Random Network-Induced Delays
abstract: In this paper, an adaptive fuzzy control is investigated for multilateral teleoperation of two cooperating robotic manipulators that manipulate an object with constrained trajectory/force in the presence of dynamics uncertainties and random network-induced delays. First, the interconnected dynamics that consist of two master robots and cooperating slave robots are formulated. To consider multiple stochastic delays in communication channels, Markov processes are used to model these random network-induced delays. The interconnected dynamics of the teleoperation are divided into a local master/slave position/force subsystem and a stochastic-delayed motion synchronization subsystem. Then, an adaptive fuzzy control strategy, which is based on linear matrix inequalities (LMIs) that combine adaptive update techniques, is proposed to suppress the dynamics uncertainties, the external disturbances, and the multiple stochastic delays in communication channels. The control approach ensures that the defined synchronization errors converge to zero. The stochastic stability in mean square of the closed-loop system is proved using LMIs based on Lyapunov-Krasovskii functional synthesis. The proposed controls are validated using extensive simulation studies.

title: Neural Network-Based Control of Networked Trilateral Teleoperation With Geometrically Unknown Constraints
abstract: Most studies on bilateral teleoperation assume known system kinematics and only consider dynamical uncertainties. However, many practical applications involve tasks with both kinematics and dynamics uncertainties. In this paper, trilateral teleoperation systems with dual-master-single-slave framework are investigated, where a single robotic manipulator constrained by an unknown geometrical environment is controlled by dual masters. The network delay in the teleoperation system is modeled as Markov chain-based stochastic delay, then asymmetric stochastic time-varying delays, kinematics and dynamics uncertainties are all considered in the force-motion control design. First, a unified dynamical model is introduced by incorporating unknown environmental constraints. Then, by exact identification of constraint Jacobian matrix, adaptive neural network approximation method is employed, and the motion/force synchronization with time delays are achieved without persistency of excitation condition. The neural networks and parameter adaptive mechanism are combined to deal with the system uncertainties and unknown kinematics. It is shown that the system is stable with the strict linear matrix inequality-based controllers. Finally, the extensive simulation experiment studies are provided to demonstrate the performance of the proposed approach.

title: Perception-Based Data Reduction and Transmission of Haptic Data in Telepresence and Teleaction Systems
abstract: We present a novel approach for the transmission of haptic data in telepresence and teleaction systems. The goal of this work is to reduce the packet rate between an operator and a teleoperator without impairing the immersiveness of the system. Our approach exploits the properties of human haptic perception and is, more specifically, based on the concept of just noticeable differences. In our scheme, updates of the haptic amplitude values are signaled across the network only if the change of a haptic stimulus is detectable by the human operator. We investigate haptic data communication for a 1 degree-of-freedom (DoF) and a 3 DoF teleaction system. Our experimental results show that the presented approach is able to reduce the packet rate between the operator and teleoperator by up to 90% of the original rate without affecting the performance of the system.

title: Passive Haptic Data-Compression Methods With Perceptual Coding for Bilateral Presence Systems
abstract: In this paper, lossy compression methods for haptic (velocity and force) data as exchanged in telepresence or virtual reality systems are introduced. Based on the proposed interpolative and extrapolative compression strategies, arbitrary passive compression algorithms can be implemented. The derived algorithms do not affect the stability of the presence system. Two algorithms were implemented and experimentally evaluated. The results show that constant data rate savings of 89% still lead to a perceptually transparent presence system.

title: A Survey of Transport Protocols for Haptic Applications
abstract: The term Hap tic refers to the sense of touch. The feeling has the ability to increase the sense of reality, to excite the user and improve the quality of experience. To carry out this sense through the Internet was, until recently, impracticable due to processing inefficiencies and/or protocol performance in capabilities, such as throughput and jitter constraints. This paper describes a Hap tic system architecture. Moreover it presents a survey of transport protocols for hap tic applications. It also performs a classification of related protocol capabilities and outlines the flow requirements that should be met by protocols designed to carry such data.

title: Bilateral control of teleoperators with time delay
abstract: When a robot is operated remotely by use of a teleoperator, it is desirable to communicate contact force information from the slave to the master, in order to kinesthetically couple the operator to the environment and increase the sense of telepresence. One problem, however, recognized as early as 1965, has remained unsolved until now: How to maintain stability in a force-reflecting bilateral teleoperator in the presence of substantial time delay? In this paper, we present a solution to this problem.

title: Stable Adaptive Teleoperation
abstract: Telerobotics, the body of science and technology which bridges human control and purely autonomous machines, is expected to be a merging point of modern developments in robotics, control theory, cognitive science, machine design, and computer science. Besides traditional applications in space, subsea, and handling of hazardous material, many new potential uses of advanced telerobotic systems have recently been suggested or explored, such as safety applications or microsurgery. This paper is a preliminary study of what role recent results in adaptive robot control and the understanding of time-delays may play in the development of effective telerobotics systems, which would best exploit the presence of the human operator while making full use of available robot control technology and computing power. The key paradigm it explores is that of simplifying, transforming, or enhancing the remote dynamics perceived by the operator by proper use of adaptive robot control techniques and tools from passivity theory.

title: Transparency and Stability of Bilateral Kinesthetic Teleoperation with Time-Delayed Communication
abstract: This paper presents a novel control approach that simultaneously ensures stability and transparency of bilateral kinesthetic teleoperation in case of communication delay. The approach rests upon a straight-forward combination of two-port lossless line theory, proper adaptation of line terminating impedance functions and situation dependent switching of adaptation filters. Application of a resulting transparency oriented control strategy to 1- or multi-DoF telemanipultion systems leads to improvements in human operator's presence feeling and task performance.

title: Passivity-based control for bilateral teleoperation: A tutorial
abstract: This tutorial revisits several of the most recent passivity-based controllers for nonlinear bilateral teleoperators with guaranteed stability properties. These schemes, which include scattering-based, damping injection and adaptive controllers, ensure asymptotic stability in multiple situations that range from constant to variable time-delays, with or without scattering transformation and with or without position tracking capabilities. Although all controllers exploit the basic property of passivity of the teleoperators, they have been developed invoking various analysis and design tools, which complicates their comparison and relative performance assessment. The objective of this paper is to present a unified theoretical framework-based on a general Lyapunov-like function-that, upon slight modification, allows to analyze the stability of all the schemes.

title: Analysis of absolute stability for time-delay teleoperation systems
abstract: In this paper, a new bilateral control algorithm based on absolute stability theory is put forward, which aims at the time-delay teleoperation system with force feedback from the slave directly. In the new control algorithm, the delay-dependent stability, instead of delay-independent stability, is taken as the aim of control design. It improves the transparency of the system at the price of unnecessary stability. With this algorithm, the time-delay teleoperation systems have good transparency and stability. A simulation system is established to verify the effect of this algorithm.

title: Bilateral Telemanipulation With Time Delays: A Two-Layer Approach Combining Passivity and Transparency
abstract: In this paper, a two-layer approach is presented to guarantee the stable behavior of bilateral telemanipulation systems in the presence of time-varying destabilizing factors such as hard contacts, relaxed user grasps, stiff control settings, and/or communication delays. The approach splits the control architecture into two separate layers. The hierarchical top layer is used to implement a strategy that addresses the desired transparency, and the lower layer ensures that no “virtual” energy is generated. This means that any bilateral controller can be implemented in a passive manner. Separate communication channels connect the layers at the slave and master sides so that information related to exchanged energy is completely separated from information about the desired behavior. Furthermore, the proposed implementation does not depend on any type of assumption about the time delay in the communication channel. By complete separation of the properties of passivity and transparency, each layer can accommodate any number of different implementations that allow for almost independent optimization. Experimental results are presented, which highlight the benefit of the proposed framework.

title: Teleoperation Control Based on Combination of Wave Variable and Neural Networks
abstract: In this paper, a novel control scheme is developed for a teleoperation system, combining the radial basis function (RBF) neural networks (NNs) and wave variable technique to simultaneously compensate for the effects caused by communication delays and dynamics uncertainties. The teleoperation system is set up with a TouchX joystick as the master device and a simulated Baxter robot arm as the slave robot. The haptic feedback is provided to the human operator to sense the interaction force between the slave robot and the environment when manipulating the stylus of the joystick. To utilize the workspace of the telerobot as much as possible, a matching process is carried out between the master and the slave based on their kinematics models. The closed loop inverse kinematics (CLIK) method and RBF NN approximation technique are seamlessly integrated in the control design. To overcome the potential instability problem in the presence of delayed communication channels, wave variables and their corrections are effectively embedded into the control system, and Lyapunov-based analysis is performed to theoretically establish the closed-loop stability. Comparative experiments have been conducted for a trajectory tracking task, under the different conditions of various communication delays. Experimental results show that in terms of tracking performance and force reflection, the proposed control approach shows superior performance over the conventional methods.

title: Bilateral teleoperation with time delay using modified wave variable based controller
abstract: Force-reflecting teleoperators in which the remote environment is kinesthetically coupled to the operator can considerably increase task performance. Wave-variable-based controllers can support the stable operation of force-reflecting teleoperators under arbitrary communication delays. Transparency in such systems is compromised in order to maintain stability. We had previously proposed a modified wave variable controller that implemented additional wave impedance in the wave variable transformations in order to focus more closely on force tracking. In this paper, we present a new controller for bilateral teleoperators based on the modified wave variable control method which provides superior position and force tracking performance compared to the traditional wave-variable-based method. Moreover, the method has high stability. Theoretical investigation and experimental results confirm the performance of this new controller.

title: Reducing Wave-Based Teleoperator Reflections for Unknown Environments
abstract: Wave variables guarantee stability for delayed teleoperation. They also introduce reflections which can prove very disorientating for an operator. This paper presents a new method for reducing the wave reflections when operating in unknown environments. The new method does not only guarantee stability but also laboratory experiments show it also reduces wave reflections allowing for improved velocity tracking and force feedback in complex and unknown operating environments.

title: Improving Haptic Feedback Fidelity in Wave-Variable-Based Teleoperation Orientated to Telemedical Applications
abstract: In wave-variable-based teleoperation systems, under the condition of time delays, the perceived haptic feedback by the user is deteriorated by a bias term due to the communication nature of wave variables. Orientated to telemedical applications, this paper proposes a new method that can partially cancel the bias portion, so that the fidelity of haptic feedback is improved. Specifically, the returning wave is augmented by the difference between the velocity of the master and the desired velocity of the slave. The steady-state position tracking is not affected by the modification. The passivity of the augmented teleoperation system can be maintained by tuning the bandwidth of a low-pass filter. Hence, stability is always achievable. The experiment results demonstrate the effectiveness of the scheme.

title: Improving Trajectory Tracking in Wave-Variable-Based Teleoperation
abstract: For wave-variable-based teleoperation systems, the performance of trajectory tracking is not ensured due to a bias term introduced by the wave communication channel. This paper proposes a new method that can improve the trajectory tracking performance. The key idea is that the wave in the forward path is augmented by the user-perceived force and slave control force. The passivity of the augmented teleoperation system can be maintained by tuning the bandwidth of the low-pass filter, as demonstrated via an example. Therefore, the system stability can be always guaranteed. Both the simulation and experimental results verify the effectiveness of the scheme.

title: Stable and enhanced position-force tracking for bilateral teleoperation with time delay
abstract: Time delay in bilateral teleoperation is a challenging problem from the point of view of both system stability and transparency. Traditional wave variable based teleoperation method can theoretically guarantee system passivity with arbitrary large time delay but cannot provide good position and force tracking performance. In this paper, a novel wave variable teleoperation structure is firstly proposed which can achieve position and force tracking with only one time delay, then a predictor is utilized to further reduce the force tracking difference between master and slave side. In order to retain passivity of the whole teleoperation system, energy reservoir technique is used to constraint injected energy. Simulation studies show that the proposed structure provides stable and enhanced position and force tracking performance compared to traditional wave variable method with position tracking difference of only one time delay and even smaller force tracking delay thanks to the predictor used.

title: Trends in the Control Schemes for Bilateral Teleoperation with Time Delay
abstract: In the last two decades, an increasing interest has been observed in the development of effective bilateral teleoperation systems. One of the biggest challenges encountered however is the presence of time delay, which can cause instability and poor transparency to the system. Researchers have been trying to tackle this problem from different angles and various control schemes have been proposed and tested. This paper aims to do an overview of those schemes and to compare them based on their assessment in order to give new researchers a summary of the current state of art. The schemes are divided into four main categories: passivity based, prediction based, sliding-mode control schemes and others. The comparison is based on transparency level, stability requirement, delay type, and delay range.

title: A Review of Bilateral Teleoperation Algorithms
abstract: Bilateral Teleoperation is a key technology to allow humans to interact with remote environments by providing the operator with haptic feedback. Haptic feedback from the one hand improves human perception and therefore the quality of the human-robot interaction, on the other hand it can tamper with the stability of the system when the communication between the master side (where the operator is) and the slave side (where the remote robot interacts with the environment) is not instantaneous but affected by delay and packet drops. In the last 40 years many algorithms have been developed to guarantee the stability of haptic teleoperation in the presence of time delay, many of them based on passivity theory. In this paper we review and compare a few algorithms that are representative of the tools in the frequency or in the time domains that have been used to develop a safe and transparent physical human-robot interaction with unknown environments.

title: Discrete-time control of bilateral teleoperation systems: a review
abstract: The possibility of operating in remote environments using teleoperation systems has been considered widely in the control literature. This paper presents a review on the discrete-time teleoperation systems, including issues such as stability, passivity and time delays. Using discrete-time methods for a master-slave teleoperation system can simplify control implementation. Varieties of control schemes have been proposed for these systems and major concerns such as passivity, stability and transparency have been studied. Recently, unreliable communication networks affected by packet loss and variable transmission delays have been received much attention. Thus, it is worth considering discrete-time theories for bilateral teleoperation architectures, which are formulated on the same lines as the continuous-time systems. Despite the extensive amount of researches concerning continuous-time teleoperation systems, only a few papers have been published on the analysis and controller design for discrete bilateral forms. This paper takes into account the challenges for the discrete structure of bilateral teleoperation systems and notifies the recent contributions in this area. The effect of sampling time on the stability-transparency trade-off and the task performance is taken into consideration in this review. These studies can help to design guidelines to have better transparency and stable teleoperation systems.

title: Scaled position-force tracking for wireless teleoperation of miniaturized surgical robotic system
abstract: Miniaturized surgical robotic system presents promising trend for reducing invasiveness during operation. However, cables used for power and communication may affect its performance. In this paper we chose Zigbee wireless communication as a means to replace communication cables for miniaturized surgical robot. Nevertheless, time delay caused by wireless communication presents a new challenge to performance and stability of the teleoperation system. We proposed a bilateral wireless teleoperation architecture taking into consideration of the effect of position-force scaling between operator and slave. Optimal position-force tracking performance is obtained and the overall system is shown to be passive with a simple condition on the scaling factors satisfied. Simulation studies verify the efficiency of the proposed scaled wireless teleoperation scheme.

title: Stability and transparency in bilateral teleoperation
abstract: Tools for quantifying teleoperation system performance and stability when communication delays are present are provided. A general multivariable system architecture is utilized which includes all four-types of data transmission between master and slave: force and velocity in both directions. It is shown that a proper use of an four channels is of critical importance in achieving high performance telepresence in the sense of accurate transmission of task impedances to the operator. It is also shown that transparency and robust stability (passivity) are conflicting design goals in teleoperation systems. The analysis is illustrated by comparing transparency and stability in two common architectures, as well as a recent passivated approach and a new transparency-optimized architecture, using simplified one-degree-of-freedom examples.

title: Wave-based teleoperation with prediction
abstract: Wave variables based on passivity and scattering theory provide a good tool for establishing bilateral teleoperation in the presence of a constant time delay. Recently, these techniques have been extended to be used for a varying time delay, as in the case of Internet-based teleoperation. Although stability is guaranteed for virtually any time delay, performance rapidly degrades for larger delays. In this paper, a predictor derived from a modified Smith predictor along with a Kalman estimator and an energy regulator is used to enhance the performance of a wave-based teleoperator in the presence of a constant delay. Also, the current wave transformation equations are extended to a more general case (better suited for systems with multiple degrees of freedom).

title: Using wave variables in time delayed force reflecting teleoperation
abstract: Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Aeronautics and Astronautics, 1996.

title: Internet-Based Bilateral Teleoperation Based on Wave Variable With Adaptive Predictor and Direct Drift Control
abstract: In a conventional bilateral teleoperation, transmission delay over the Internet can potentially cause instability. A wave variable algorithm guarantees teleoperation stability under varying transmission delay at the cost of poor transient performance. Adding a predictor on the master side can reduce this undesirable side effect, but that would require a slave model. An inaccurate slave model used in the predictor as well as variations in transmission delay, both of which are likely under realistic situations, can result in steady-state errors. A direct drift control algorithm is used to drive this error to zero, regardless of the source of the error. A semi-adaptive predictor that can distinguish between free space and a rigid contact environment is used to provide a more accurate force feedback on the master side. A full adaptive predictor is also used that estimates the environmental force using recursive least squares with a forgetting factor. This research presents the experimental results and evaluations of the previously mentioned wavevariable-based methods under a realistic operation environment using a real master and slave. The algorithm proposed is innovative in that it takes advantage of the strengths of several control methods to build a promising bilateral teleoperation setup that can function under varying transmission delay, modeling error, and changing environment. Success could lead to practical applications in various fields, such as space-based remote control, and telesurgey.

title: Array of Robots Augmenting the Kinematics of Endocavitary Surgery
abstract: Minimally invasive surgery (MIS) has been introduced in the last decades with the goal of making scarless surgery feasible. In general, an MIS approach allows concrete benefits in terms of reduced trauma, quicker recovery times, and improved cosmetics. On the other hand, in its current state, MIS introduces more difficulties for surgeons, due to its intrinsic complexity. This issue has inspired the major technological challenge of designing miniaturized robots able to completely enter the body and to perform surgical procedures under intuitive teleoperation. The dream of achieving a completely minimally invasive therapeutic procedure, while offering the typical advantages of traditional open surgery, has brought to the complete elimination of external incisions by gaining access to the peritoneal cavity through a natural orifice. These scarless procedures are known as Natural Orifice Transluminal Endoscopic Surgery (NOTES) interventions. In this paper, novel approaches to NOTES instruments and platforms are presented, in which modular robots measuring 12 mm in diameter with basic functionalities (manipulation, cutting, vision, and retraction) and multiple degrees of freedom are deployed inside a human phantom and anchored on a supporting frame for the stable execution of tasks. This paper illustrates the general concept, novel design guidelines for the modular robots, and two robotic units successfully assembled and tested with ten users, in order to assess the capabilities of the system in pick and place experiments and cutting tasks. Experiments for the assessing force and accuracy are described as well.

title: Viscoelastic model based force control for soft tissue interaction and its application in physiological motion compensation
abstract: Controlling the interaction between robots and living soft tissues has become an important issue as the number of robotic systems inside the operating room increases. Many researches have been done on force control to help surgeons during medical procedures, such as physiological motion compensation and tele-operation systems with haptic feedback. In order to increase the performance of such controllers, this work presents a novel force control scheme using Active Observer (AOB) based on a viscoelastic interaction model. The control scheme has shown to be stable through theoretical analysis and its performance was evaluated by in vitro experiments. In order to evaluate how the force control scheme behaves under the presence of physiological motion, experiments considering breathing and beating heart disturbances are presented. The proposed control scheme presented a stable behavior in both static and moving environment. The viscoelastic AOB presented a compensation ratio of 87% for the breathing motion and 79% for the beating heart motion.

title: The impact of interaction model on stability and transparency in bilateral teleoperation for medical applications
abstract: An analysis of stability and transparency of a force feedback teleoperation system for cutting-edge robotic surgery is presented. Previous works in teleoperated robotic surgery do not consider the real behavior of the environment, which was supposed to be only elastic. However, new surgical procedures in which the environment dynamics plays a crucial role start emerging as a result of technological progress. In robotic assisted beating-heart surgery, for instance, the dynamics of the contact between surgical tools and soft tissues has an impact not only in the performance of the force control task but also in the performance of the teleoperation control scheme in terms of transparency and stability. Therefore, a more realistic description of the environment has to be adopted in order to safely operate during robot-patient interaction. For this purpose, a viscoelastic contact model is introduced into the bilateral teleoperation scheme, and a performance study is provided. The obtained results show the advantages of the selected approach when targeting teleoperated surgical interventions in which the interaction dynamics has become a significant issue.

title: Needle steering in biological tissue using ultrasound-based online curvature estimation
abstract: Percutaneous needle insertions are commonly performed for diagnostic and therapeutic purposes. Accurate placement of the needle tip is important to the success of many needle procedures. The current needle steering systems depend on needle-tissue-specific data, such as maximum curvature, that is unavailable prior to an interventional procedure. In this paper, we present a novel three-dimensional adaptive steering method for flexible bevel-tipped needles that is capable of performing accurate tip placement without previous knowledge about needle curvature. The method steers the needle by integrating duty-cycled needle steering, online curvature estimation, ultrasound-based needle tracking, and sampling-based motion planning. The needle curvature estimation is performed online and used to adapt the path and duty cycling. We evaluated the method using experiments in a homogenous gelatin phantom, a two-layer gelatin phantom, and a biological tissue phantom composed of a gelatin layer and in vitro chicken tissue. In all experiments, virtual obstacles and targets move in order to represent the disturbances that might occur due to tissue deformation and physiological processes. The average targeting error using our new adaptive method is 40% lower than using the conventional non-adaptive duty-cycled needle steering method.

title: Kalman filter-based EM-optical sensor fusion for needle deflection estimation
abstract: PurposeIn many clinical procedures such as cryoablation that involves needle insertion, accurate placement of the needle’s tip at the desired target is the major issue for optimizing the treatment and minimizing damage to the neighboring anatomy. However, due to the interaction force between the needle and tissue, considerable error in intraoperative tracking of the needle tip can be observed as needle deflects.MethodsIn this paper, measurements data from an optical sensor at the needle base and a magnetic resonance (MR) gradient field-driven electromagnetic (EM) sensor placed 10 cm from the needle tip are used within a model-integrated Kalman filter-based sensor fusion scheme. Bending model-based estimations and EM-based direct estimation are used as the measurement vectors in the Kalman filter, thus establishing an online estimation approach.ResultsStatic tip bending experiments show that the fusion method can reduce the mean error of the tip position estimation from 29.23 mm of the optical sensor-based approach to 3.15 mm of the fusion-based approach and from 39.96 to 6.90 mm, at the MRI isocenter and the MRI entrance, respectively.ConclusionThis work established a novel sensor fusion scheme that incorporates model information, which enables real-time tracking of needle deflection with MRI compatibility, in a free-hand operating setup.

title: Beating-heart mitral valve surgery: preliminary model and methodology.
abstract: It is our hypothesis that image-guided mitral valve repair can be performed on the beating heart without cardiopulmonary bypass. As a first phase, we tested the feasibility of suturing the anterior and posterior mitral valve leaflets under image guidance.



In a water bath model, imaging approaches and suturing techniques were developed. Then, in 6 pigs, the left atrium was cannulated with a custom-made 15-mm valved port through a left thoracotomy. Atrial pressure was elevated by shunting of arterial blood to minimize air induction. A multiplane transesophageal echocardiographic probe was evaluated in the intraesophageal and epicardial positions. With a commercial endoscopic suturing device, sutures were placed through the anterior and posterior mitral leaflets under echocardiographic guidance. The animals were killed, and suture accuracy was evaluated by measuring the distance from the intended target areas on the anterior and posterior mitral leaflets. Air induction was monitored by echocardiography and graded as minimal to severe.



There were no cases of hemodynamic instability or significant arrhythmia. The most effective imaging plane was a short-axis view that used the transesophageal echocardiographic probe epicardially at the heart base. Air induction was minimal in 2 animals, mild in 3, and moderate in 1. Sutures were successfully placed 9 of 12 times (mean error 0.8 +/- 0.5 cm).



With these methods, off-pump, image-guided suturing of the beating-heart mitral valve was possible. This model may be a useful starting point for developing off-pump mitral valve repair procedures.

title: Medical Robot and Master Slave System for Minimally Invasive Surgery
abstract: A robotic surgical system reduces stress on the patient, equalizes differences in the quality of local medical service, and enhances emergency medical care and medical education. Robotic surgery is particularly appropriate for minimally invasive surgery (MIS) because it achieves high operational accuracy even in concealed regions. The minimally invasive surgical system introduced in this paper comprises master manipulators operated by a surgeon and slave manipulators that conduct the surgery. The system transmits the total environment of the operating room and reduces the load applied to soft tissues by approximately 30% by implementing an augmented force presentation capability for the surgeon. In the system developed for the total knee arthroplasty (TKA), minimal invasiveness was achieved by generating a tool path employing multiple degrees of freedom of the surgical robot to avoid injury to the soft tissues. The bone-cutting force and resulting temperature rise were limited by employing cutting conditions compatible with bone viability.

title: Techniques and accuracy of thoracolumbar pedicle screw placement.
abstract: Pedicle screw instrumentation has been used to stabilize the thoracolumbar spine for several decades. Although pedicle screws were originally placed via a free-hand technique, there has been a movement in favor of pedicle screw placement with the aid of imaging. Such assistive techniques include fluoroscopy guidance and stereotactic navigation. Imaging has the benefit of increased visualization of a pedicle's trajectory, but can result in increased morbidity associated with radiation exposure, increased time expenditure, and possible workflow interruption. Many institutions have reported high accuracies with each of these three core techniques. However, due to differing definitions of accuracy and varying radiographic analyses, it is extremely difficult to compare studies side-by-side to determine which techniques are superior. From the literature, it can be concluded that pedicles of vertebrae within the mid-thoracic spine and vertebrae that have altered morphology due to scoliosis or other deformities are the most difficult to cannulate. Thus, spine surgeons would benefit the most from using assistive technologies in these circumstances. All other pedicles in the thoracolumbar spine should theoretically be cannulated with ease via a free-hand technique, given appropriate training and experience. Despite these global recommendations, appropriate techniques must be chosen at the surgeon's discretion. Such determinations should be based on the surgeon's experience and the specific pathology that will be treated.

title: Bilateral control of a teleoperator for soft tissue palpation: design and experiments
abstract: In robot-assisted interventions, providing a surgeon with haptic information regarding contacts made between surgical instruments and tissue can improve task performance and reliability. In this paper, a force-reflective user interface is used with a sensorized surgical instrument to form a master-slave test-bed for studying haptic interaction in a soft-tissue endoscopic surgery environment. After modeling and parametric identification of the master and the slave, bilateral controllers are designed and teleoperation experiments involving a single degree of freedom surgical task on soft tissue (palpation) are conducted. The transparency of the teleoperator in terms of transmitting the critical task-related information to the user in the context of soft-tissue surgical applications is investigated

title: The Low-Stiffness Teleoperator Slave — a Trade-off between Stability and Performance
abstract: Stability is essential for teleoperation and a prerequisite for performance. This paper analyzes the the stability/performance trade-off of a teleoperator where the slave device has a built-in passive intrinsic stiffness. Stability is quantified as time delay robustness and performance is expressed using teleoperator damping and teleoperator stiffness, the boundaries of the Colgate Z-width Two classic control schemes, position error and Lawrence 4-channel, are used along with a novel 5-channel scheme where the slave stiffness deflection is measured, and compensated for, to improve the performance. The teleoperator system was analyzed theoretically using a linear model and the findings were experimentally validated on a one degree of freedom teleoperation setup. It was found that: • A lower slave stiffness improves stability for all three teleoperator architectures. • The stability boundary of the three controllers is similar. • The performance of the controllers increases from: (poor) position error, 4-Channel to (excellent) 5-channel. • A classical linear analysis method can accurately predict the stability characteristics of the teleoperator system. Therefore it can be concluded that a compliant slave device offers a stability advantage for a range of teleoperation situations and that the loss of performance can be partly compensated.

title: Analysis of effective impedance transmitted to the operator in position-exchange bilateral teleoperation
abstract: In this paper, we analyze the impedance transmitted to the operator in position-exchange bilateral teleoperation including the effects of master and slave dynamics, local and communication time delay, low-pass filtering of the velocity estimate, and controller stiffness and damping, for three different environment dynamics: free, clamped, and a mass-damper-spring. We show the impedance transmitted the operator numerically using effective impedances, a conceptual tool that decomposes the impedance into components with physical analogs, and also present symbolic expressions for the effective stiffness and damping transmitted to the operator at low frequencies. Our results show that local and communication time delay do not significantly affect the effective stiffness, strongly affect effective damping, and weakly affect effective mass transmitted to the operator. The low-pass filter cut-off frequency affects the impedance transmitted to the operator differently depending on the environment impedance. For many cases, increasing the cutoff frequency primarily increases the effective damping bandwidth. Increasing controller stiffness increases the effective stiffness transmitted to the operator at low frequencies, but can significantly change the effective stiffness, damping, and mass, at higher frequencies. Experimental data gathered using a bilateral telemanipulator consisting of two Phantom Premium 1.5 robots interacting with a custom-made environment validates the theoretical analysis.