Surgical Robot Transformer (SRT): Imitation Learning for Surgical Tasks

Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 11:54:28

Abstract: We explore whether surgical manipulation tasks can be learned on the da Vinci robot via imitation learning.
However, the da Vinci system presents unique challenges which hinder straight-forward implementation of imitation learning. Notably, its forward kinematics is inconsistent due to imprecise joint measurements, and naively training a policy using such approximate kinematics data often leads to task failure. To overcome this limitation, we introduce a relative action formulation 
which enables successful policy training and deployment using its approximate kinematics data. A promising outcome of this approach is that the large repository of clinical data, which contains approximate kinematics, may be directly utilized for robot learning without further corrections. We demonstrate our findings through successful execution of three fundamental surgical tasks, including tissue manipulation, needle handling, and knot-tying.

Main paper: 
 introduction: 
 Recently, large-scale imitation learning has shown great promise in creating generalist systems for manipulation tasks [1] . Prior research in this area has mostly focused on learning day-to-day household activities. However, an under-explored area with high potential is the surgical domain, particularly with the use of Intuitive Surgical's da Vinci robot. These robots are deployed globally and possess immense scaling potential: as of 2021, over 10 million surgeries have been performed using 6,500 da Vinci systems in 67 countries, with 55,000 surgeons trained on the system [2] . Often, the video and kinematics data are recorded for post-operative analysis, resulting in a large repository ofRelative pose trajectory (pos, rot, gripper∠)L/R stereo cam L/R wrist cam demonstration data. Utilizing such large scale data holds significant potential for building generalist systems for autonomous surgery [3] .However, robot learning on the da Vinci presents unique challenges. The hardware suffers from inaccurate forward kinematics due to potentiometer-based joint measurements, hysteresis, and overall flexibility and slack in its mechanism [4] . These limitations result in the robot's failure to perform simple visual-servoing tasks [5] . As we discover in this work, naively training a policy using such approximate kinematics data almost always leads to task failure. For instance, a policy trained to output absolute end-effector poses, which is a common strategy to train robot policies, achieves near-zero success rates across all tasks explored in this work, including tissue lift, needle pickup and handover, and knot-tying (Fig. 1 ). To achieve robot learning at scale, we must devise a strategy that leverages such approximate kinematics data effectively.Towards this end, we present an approach for robot learning on the da Vinci using its approximate kinematics data. Intuitively, our approach is based on the observation that the relative motion of the robot is much more consistent than its absolute forward kinematics. We thus model policy actions as differential motion and further explore its variants to design the most effective action representation for the da Vinci. We find that training an imitation learning algorithm using such relative formulation shows robustness to various configuration changes to the robot, even those known to significantly disrupt the robot's forward kinematics. Specifically, the da Vinci tools can be removed and reinstalled and all the robot joints can be freely moved, including the notoriously inaccurate set-up joints [4] , without significantly impacting policy performance.Additionally, we explore the use of wrist cameras in the surgical workflow. While not commonly employed in clinical settings, wrist cameras have demonstrated effectiveness in improving policy performance and facilitating generalization to out-of-distribution scenarios, such as varying workspace heights or unfamiliar visual distractions [6] . We thus evaluate their impact on performance and practical potential by designing removable brackets that enable easy sharing across various surgical instruments.Overall, our results indicate that the relative motion on the da Vinci is more consistent than its absolute motion. Following this result, we further observe that a carefully chosen relative action representation can sufficiently train policies that achieve high success rates in surgical manipulation tasks. Additionally, using wrist cameras significantly improves policy performance, especially during phases of the procedure when precise depth estimation is crucial. In robustness tests, our model demonstrates the ability to generalize to novel scenarios, such as in the presence of unseen 3D suture pads and animal tissues, showing promise for future extensions into pre-clinical research.Our main contributions are: (i) a successful demonstration of imitation learning on the da Vinci while using its approximate kinematics data and without requiring further kinematics corrections, while drastically outperforming the baseline approach; (ii) experiments showing that imitation learning can effectively learn complex surgical tasks and generalize to novel scenarios such as in the presence of unseen realistic tissue; (iii) ablative experiments demonstrating the importance of wrist cameras for learning surgical manipulation tasks. 
related work: 
 Manipulation and Imitation Learning Imitation learning enables robots to learn from expert demonstrations [7] . Behavioral cloning (BC) is a simple instantiation of imitation learning that directly predicts actions from observations. Early works tackle this problem through the lens of motor primitives [8, 9, 10, 11] . With the development of deep learning and generative modeling, different architectures and training objectives have been proposed to model the demonstrations end-to-end. This includes the use of ConvNets or ViT [12] for image processing [13, 14, 15] , RNN or transformers for fusing history of observations [16, 17, 18] , tokenization of the action space [19] , generative modeling techniques such as energy-based models [20] , diffusion [21] and VAEs [22, 23] . Prior works also focus on the few-shot aspect of imitation learning, [24, 25, 26, 27] , language conditioning [15, 18, 19, 28] , co-training [19, 29, 28] , retrieval [30, 31, 32] , using play data [33, 34, 35, 36] , using human videos [37, 38, 39, 40, 41, 42] , and exploiting task-specific structures [43, 44, 45] .However, most of these prior works focus on table-top manipulation in home settings. Surgical tasks, on the other hand, pose a unique set of challenges. They require precise manipulation of deformable objects, involve hard perception problems with inconsistent lighting and occlusions, and surgical robots may often have inaccurate proprioception and hysteresis [4] that are less pronounced in industrial arms. While in principle end-to-end imitation learning could capture these variations implicitly, it is unclear what design choices are important to enable effective learning in this regime.We also note that in the dVRK community, the inaccuracies of the robot have been addressed via hand-eye calibration [46, 47, 48] . However, we assume that the demonstration dataset has already been collected without hand-eye calibration e.g., the large-scale clinical data, which means precise ground-truth kinematics is not available during training. Therefore, hand-eye calibration is not a fundamental solution to our problem. Autonomous Surgery Prior works in autonomous surgery primarily focus on designing taskspecific policies for specific tasks, such as for suturing [49, 50, 51, 52] , endoscope control [53, 54] , navigation [55, 56, 57] , and tissue manipulation [58, 59] . However, these methods do not typically scale well across various tasks or generalize well to varying environmental conditions. In contrast, end-to-end imitation learning offers a relatively simple solution to these shortcoming by only requiring good robot demonstrations. While prior works have also explored the use of imitation learning for surgical tasks [60, 61, 62, 55] , its application to complex manipulation tasks like knot-tying remains unexplored, and practical design choices for implementing on the da Vinci have not been addressed.We model tool-centric actions as relative motion w.r.t the current end-effector frame, which is a moving body frame. Tool-centric actions can be defined as:EQUATIONIntuitively, the desired poses ĝi s are subtracted by the current end-effector poses g i t using the SE(3) subtraction rule, for each time up to horizon C and for each corresponding left and right grippers. There are largely two benefits to adopting this action representation. A relative motion formulation is used, which we show later in Section 5, is more consistent compared to the absolute forward kinematics of the arms. Also, the subtraction cancels out the endoscope forward kinematics terms and the actions can be expressed in terms of the PSMs forward kinematics only. This effectively reduces the margin for errors since less joints are involved in representing actions. However, one caveat of this approach is that the delta motion is defined w.r.t a moving reference frame. This requires the policy to implicitly localize the current end-effector orientation from image observations, and output translations and rotations along the localized principal axes, which can be a challenging task. The policy objective can be defined as π : o t → A tool t,C . This formulation is visually shown in Fig. 4 . defined w.r.t endoscope tip frame and delta rotations are defined w.r.t the current end-effector frame. This formulation can be defined as follows:EQUATIONWhere the subtraction operation ⊖ defined as:EQUATIONIntuitively, the subtraction is performed between the corresponding translation and rotation elements i.e. vector subtraction for positions and SO(3) subtraction for rotations. A key distinction of this approach from the tool-centric approach lies in modeling the delta translation with respect to the fixed frame of the endoscope-tip, rather than the moving frame of the end-effector. This approach removes the burden for the policy to localize the end-effector's orientation to generate delta translations along the localized axes, thereby improving the quality of translation motion. The policy can be defined as π :o t → A hybrid t,C. This formulation is visually shown in Fig. 4 . 
technical approach: 
 Consider the dVRK system, as illustrated in Fig. 3 , which includes both the robot and a teleoperation console for user interaction. The dVRK features an endoscopic camera maipulator (ECM) and two patient side manipulators (PSM1, PSM2) sharing the same robot base. Each arm is a sequential combination of set-up joints (SUJ) which are passive, followed by active joints which are motorized (Fig. 3 ). The passive joints are notoriously inaccurate due to using only potentiometers for joint measurements. The active joints use both potentiometers and motor encoders, providing improved precision. However, in general, the use of potentiometers throughout all the joints causes the forward kinematics of the arms to be inaccurate, even up to 5cm error [4] .Using the dVRK console, the user collects many demonstrations of a task, acquiring a dataset D = {τ 1 , ..., τ N }, where each trajectory τ i = {(o 1 , x 1 , a 1 ), ..., (o T , x T , a T )} is a collection of observation o t , proprioception x t , and actions a t , collected at time step t. Specifically, the observation o t includes left/right surgical endoscope images and left/right wrist camera images, totaling four images (Fig. 2 ), proprioception is the current pose of the PSMs w.r.t surgical endoscope tip frame denoted as x t = {g l t , g r t }, where g = (p, R) ∈ SE(3) and {l, r} denotes the left and right grippers respectively, and actions are the commanded desired waypoints to reach specified via teleoperation control, denoted as a t = {ĝ l t , ĝr t }. ). Unfortunately, the dVRK arms are notorious for providing inconsistent forward kinematics. This is due to the setup joints (blue) only using potentiometers for joint measurements, which can be unrelible. The active joints (pink) use both potentiometers and motor encoders, improving precision.Our objective is to learn surgical manipulation tasks via imitation learning. Given the robot's inaccurate forward kinematics, choosing the appropriate action representation is crucial. To illustrate this, we investigate three action representations: camera-centric, tool-centric, and hybrid-relative as shown in Fig. 4 . The camera-centric approach serves as a baseline, highlighting the limitations of modeling actions as absolute poses of the end-effectors. The tool-centric approach offer an improved formulation by modeling actions as relative motion, leading to higher success rates. The hybridrelative approach further improves beyond tool-centric approach by modeling translation actions with respect to a fixed reference frame, further improving accuracy in translation movements. These approaches are detailed below:1. Camera-centric actions: We model camera-centric actions as absolute poses of the end-effectors w.r.t the endoscope tip frame. The setup is similar to how position-based visual servoing applications (PBVS) are implemented and is a natural choice on the dVRK. Specifically, the objective is to learn a policy π that, given an observation o t at time t, predicts an action sequence A t,C = (a t , ..., a t+C ), where C denotes the action prediction horizon. The policy can thus be defined as π : o t → A t,C . This formulation is visually shown in Fig. 4 . 
implementation details: 
 To train our policies, we use action chunking with transformers (ACT) [23] and diffusion policy [63] . The policies were trained using the endoscope and wrist cameras images as input, which are all downsized to image size of 224 × 224 × 3. The original input size of the surgical endoscope images were 1024 × 1280 × 3 and the wrist images were 480 × 640 × 3. Kinematics data is not provided as input as commonly done in other imitation learning approaches because it is generally inconsistent due to the design limitations of the dVRK. The policy outputs include the end-effector (delta) position, (delta) orientation, and jaw angle for both arms. We leave further specific implementation details in Appendix A. 
experiments: 
 In our experiments, we aim to understand the following key questions: (1) is imitation learning sufficient to learn challenging surgical manipulation tasks? (2) Is the dVRK's relative motion more consistent than its absolute forward kinematics? (3) Are using wrist cameras critical to achieving high success rates? (4) How well does our proposed model generalize to unseen novel scenarios?To answer these questions, we compare the task success rates of the policies trained using various action representations. We also directly compare the consistency of relative versus absolute motion by tracking a reference trajectory using the various action representations and comparing their tracking Tool-centric 0.9 0.9 0.7Hybrid-relative 0.9 0.8 0.8 errors. We also explore the importance of wrist cameras by comparing the policy performance with and without them. Finally, we consider whether the proposed models can generalize to novel unseen scenarios, such as in the presence of animal tissues. These experiments are explored in the context of three tasks: lift tissue, needle pickup and handover, and knot-tying.Experiment Setup During data collection, the robot is set up in a reference configuration as shown in Fig. 5 . In this configuration, 224 trials were collected for tissue lift, 250 trials for needle pickup and handover, and 500 trials for knot-tying, all collected by a single user across multiple days. During all experiments, a dome simulating the human abdomen (Fig. 5 ) was used to roughly place the arms and the endoscope in an approximately similar location using the same holes. The placement is only approximate because the holes are much larger than the endoscope and tool shaft size, and the tools have to be manually placed into the holes by moving the set-up joints.Evaluating the Consistency of Relative Motion vs. Absolute Forward Kinematics In this section we seek to understand whether relative motion on the dVRK is more consistent than its absolute forward kinematics. To test our hypothesis, we teleoperate a reference trajectory e.g., an infinity sign as shown in Fig. 5 . This trajectory is then represented in various action representations using the formulas presented in Section 3. Then, we place the end-effector in the same initial pose and replay the trajectories using the various action representations under different robot configurations. These different configurations include shifting the robot workspace to the left and to the right side (Fig. 5 ). These workspace shifts cause the robot set-up joints to move, which are the joints prone to cause large joint measurement errors due to using only potentiometers for joint measurements. To compare their tracking errors, the replayed trajectories are annotated at the end-effector (i.e. control point) in image coordinates and plotted, as shown in the bottom row of Fig. 5 .The plots in Fig. 5 and the numeric RMSE results in Table 1 show that in the reference configuration, all action representation precisely reconstruct the reference trajectory, since the set-up joints have not yet moved. However, when the robot configuration is changed by moving the set-up joints and new erroneous joint measurments are obtained, the camera-centric action representation fails to reconstruct the reference trajectory. Also, this error is not consistent for different robot configurations as shown in the trajectory plots in Fig. 5 . For relative action representations, which include tool-centric and hybrid-relative action formulations, the reference trajectory is repeated more consistently, and their numeric errors do not vary significantly as observed in Table 1 . In summary, this experiment shows that relative motion on the dVRK is more consistent compared to its absolute forward kinematics in the presence of inconsistent joint measurement errors.Policy Performance Using Various Action Representations We evaluate the policy performance using the various action representations on tissue lift, needle pickup and handover, and knot-tying as shown in Table 2 . The camera-centric action representation performed poorly across all three tasks.Because the joint measurements of the dVRK are inconsistent, the end-effectors almost always failed to reach the target objects (e.g., tissue corner, needle, and string) and often dangerously collided with the underlying rubber pads. The policy trained using tool-centric action representation showed improved performance across all three tasks. However, during needle pickup + handover when large rotations were involved, the handover phase of the task often failed. In particular, after picking up the needle, the left gripper had to make a ∼ 90 degree rotation to transfer the needle to the opposing arm (Fig. 6 ). During this phase of the motion, the orientations of the grippers appeared correct, however, the translation motion appeared incorrect and seemed to be the cause of task failure. We conjecture this reason was due to grounding the policy actions to a moving end-effector frame. The policy is required to localize the moving end-effector orientation using image observations and generate delta translations along the localized principal axes of direction, which can be a challenging task.To fix this issue, the hybrid relative motion action was used, which grounds the translation motion to a fixed frame of the endoscope-tip. This formulation improved the translation errors during the aforementioned needle handover phase and ultimately achieved the highest success rates across all three tasks. This best performing action representation was also implemented on Diffusion Policy, however, the performance was not as high as ACT. We also observed that Diffusion Policy often generated unsmooth motions, potentially due to not being able to adapt to the inconsistent kinematics of the robot.Evaluating the Importance of Wrist Camera We also evaluate the importance of adding wrist cameras and their contribution to task success. To demonstrate this, we trained policies without wrist cameras on the needle pickup and handover and knot-tying tasks using the hybrid relative action formulation (Table 2 ). Overall, we observed that omitting wrist cameras lead to significant drop in performance. We conjecture that wrist cameras aid in scenarios where precise depth estimation is necessary. For instance, during the needle pickup and handover task, specifically during the latter phase transferring the needle, the wrist views clearly showed whether the needle was being navigated into the opposing grippers from afar. This additional view may have provided better context for task success. However, we observed that this level of information was difficult to discern from the third-person endoscopic view alone.Evaluating Generalization We also evaluate the ability of our models to generalize to novel scenarios, such as under more clinically relevant background using animal tissues (pork and chicken) and an unseen 3D suture pad. Most of this evaluation remains qualitative and greater details are elaborated in Appendix B. In terms of quantitative results, we evaluate the hybrid-relative action formulation on the needle pick-up and handover task on a pig loin background. We observe that its overall success rate is quite high (Table 2 ), however, the quality of the motion and the accuracy of the needle grasps were much lower compared to those observed in the core experiments. In terms of qualitative results, we observe multiple instances of successful knot-tying achieved on pork tissue, and needle grasps on chicken background and on an unseen 3D pad (Appendix B). 
limitations and conclusion: 
 In this work, we opt for using off-the-shelf large wrist cameras which are not clinically relevant. However, the cameras may be replaced with much smaller ones (1-2mm diameter) and its mount can be further optimized by integrating quick-release mechanisms for swift transfer between surgical tools. Also, our model is limited as it can only act based on current observations and does not have the ability to modulate different behavior based on human instruction. We hope to address these issues in future work to further advance the autonomy of surgical robots.In summary, we demonstrated an approach for imitation learning on the dVRK using its approximate kinematics data, without providing further post-processing corrections. The key idea of our approach was to rely on the more consistent relative motion of the robot, achieved by modeling policy actions as relative motion such as tool-centric and hybrid-relative actions. As mentioned in the introduction, we believe that our work is a step towards leveraging the large repository of approximate surgical data for robot learning at scale, without providing further kinematics corrections. We believe more research in this direction can further guide the path towards building general-purpose systems towards autonomous surgery. 
 
 Appendix: For ACT, main modifications include changing the input layers to accept four images, which include left/right surgical endoscope views and left/right wrist camera views. The output dimensions are also revised to generate end-effector poses, which amounts to a 10-dim vector for each arm (position [3] + orientation [6] + jaw angle [1] = 10), thus amounting to a 20-dim vector total for both arms. The orientation was modeled using a 6D rotation representation following [21] , where the 6 elements corrrespond to the first two columns of the rotation matrix. Since the network predictions may not generate orthonormal vectors, Gram-Schmidt process is performed to convert them to orthonormal vectors, and a cross product of the two vectors are performed to generate the remaining third column of the rotation matrix. For diffusion policy, similar modifications are made such as changing the input and the output dimensions of the network appropriately. The specific hyperparameters for training are shown in Table 3  


Decision: The reviewers all agree that the manipulation capabilities of the presented work in real-world surgical experiments are impressive. However, there are concerns regarding the technical novelty of this approach as it seems to be limited to the action representation.
A clarification on the novelty of the presented approach and the action representation would benefit the paper.

Additional discussion on the potential risk of the approach would further strengthen the paper.
One potential risk is the smoothness of the action sequences produced by diffusion policies (even when combined with action chunking).
This issue has been extensively studied in previous work[1]. The paper would be further strengthened by discussing and comparing to this approach.

[1]: Scheikl et al, “Movement Primitive Diffusion: Learning Gentle Robotic Manipulation of Deformable Objects”, 2023

## Post Rebuttal
The rebuttal addressed all remaining concerns and further improved the already strong submission. The reviewers unanimously agree that this work should be published at CoRL 2024.

Reviews: review: title: title: Official Review
paper_summary: paper_summary: The paper proposes relative pose as an effective action representation to learn complex surgical tasks can be from demonstrations that have imperfect/noisy kinematics data.
main_review: main_review: **Strengths**
- The proposed approach is simple, intuitive, and well-described
- The proposed approach could enable further research in learning complex surgical tasks from existing datasets despite containing approximate kinematics.
- Experiments are convincing and demonstrate that the proposed method is effective and provides justification for the design choices based on the inferior performance of ablative baselines.
- Clinical demonstrations are very impressive to see. It would have been further improved by reporting even minimal statistics (e.g., success rate out of five attempts).

**Weaknesses and questions**
- The performance improvement over tool-centric design seems to be minimal, except in one stage of one task.
- The diffusion policy was not trained using the baseline representations, leaving out the possibility that the claimed benefits might diminish when a diffusion policy is used. Any justifications for why this is unlikely would be helpful.
- Why was the ablation without wrist camera only evaluated on the needle handover task?
questions: questions: Please address the questions and concerns from above.
limitations: limitations: 2
review_summary: review_summary: The paper presents a simple yet effective action representation to enable learning complex surgical skills from existing datasets containing approximate kinematics. The paper is well-written, and the experimental results sufficiently demonstrate its effectiveness and justify the core design choices.
title: title: Surgical Robot Transformer (SRT): Imitation Learning for Surgical Subtasks
paper_summary: paper_summary: The authors propose an imitation learning method to solve tasks that are relevant to robotic surgery such as suturing, tissue lift, and needle pickup. One of the major challenges with imitation learning on the da Vinci robot is dealing with the noisy forward kinematics. The authors hypothesize that modeling the action representation as relative motion of the robot might have better success across all tasks. To this end, the author compare camera-centric, tool-centric, and hybrid-relative actions to see which performs best.
main_review: main_review: Strengths
* This paper is clearly written and proposes a straightforward method for an important class of manipulation problems.
* The hypothesis is clearly stated, notably that action representation is important given the noisy kinematic data of the robot
* The authors perform hardware experiments on the da Vinci platform. Aside from the speed of the videos, the demonstrations of dexterous manipulation are impressive.
* The authors' approach seems to generalize to unseen tasks and tissue types.

Weaknesses
* The only novel technical contribution appears to be changing the action representation to account for relative motion of the robot
* Although the method appears to work, the videos don’t demonstrate much diversity in the task. It would be much more informative if it were possible to discern the differences in the initial conditions of a particular task.
* The videos are sped up 4-12x, suggesting that the robot is moving quite slowly.
* It's unclear how much the performance of the robot could be improved by performing system identification and learning a better predictive model of the robot.
questions: questions: * Could the authors clarify if there is any novelty in their method beyond comparing the action representations?
* It would be interesting to know how much data is required to get good policies. The authors report ~250 expert demonstrations. Could they get away with less?
* The authors use ACT and Diffusion Policy but don’t mention how long either policy takes to train. They also don’t discuss inference times. This would be helpful in evaluating the method.
limitations: limitations: 3
review_summary: review_summary: I suggest a weak accept because while I interpret a lack of novelty in the proposed approach, the authors have succesfully demonstrated their approach working on real hardware. I believe this can be a very important step in doing imitation learning for surgical robotics.
title: title: Great work! Small step in IL giant leap on robot.
paper_summary: paper_summary: The paper studies the imitation learning setting (i.e. the best action representation) of Da Vinci Research Kit on surgical tasks.
main_review: main_review: # Originality

Though there is not too much novelty in the learning method as well as the data processing pipeline, the paper makes great contribution to the community studying several important setup in the camera and contributing an imitation learning dataset.

As the "large robotics models" are promising and prevailing in the manipulation tasks, we are looking forward for the migration of the idea to, I would say, more important task. And this paper makes that step out.

# Clarity

The paper is well-written.

# Strengths

1. The paper experiments different settings (action representation and model architecture) and conduct real world experiments.
2. Appreciate the "single user" who collected the dataset (Line 183). A great contribution to the community.
3. The paper conducts real-world evaluation of the trained model.


# Weaknesses

1. I would expect more discussion on the potential risks and solutions of current method. Connecting to Question 1, do we have any way to estimate the risk and safe capability of a trained model?
2. The technical part of the paper might not be novel. That is, the learning algorithm as well as the model architecture are not the focus of this paper.
questions: questions: 1. Do we have fine-grained metrics in addition to the success rate? For example the "progress" the robot achieves so far before failure. 
2. Have you studied the scalability of the model? That is, have you try to increase / decrease the scale of the model and see if it works? I think that would be very useful to the community to see, given current amount of data, what's the largest possible size of the model.
limitations: limitations: 2
review_summary: review_summary: The paper is well structured and studied several settings in the surgical tasks. An important and bold attempt to apply large model toward surgical domain.