Surgical Robot Transformer (SRT): Imitation Learning for Surgical Tasks
Authors: Ji Woong Kim
Tony Z. Zhao
Samuel Schmidgall
Anton Deguet
Marin Kobilarov
Chelsea Finn
Axel Krieger
Venue: CoRL 2024 (2024)
Keywords: Imitation Learning
Manipulation
Medical Robotics
TL;DR: Imitation learning on the da Vinci robot without needing hand-eye calibration

Abstract: We explore whether surgical manipulation tasks can be learned on the da Vinci robot via imitation learning.
However, the da Vinci system presents unique challenges which hinder straight-forward implementation of imitation learning. Notably, its forward kinematics is inconsistent due to imprecise joint measurements, and naively training a policy using such approximate kinematics data often leads to task failure. To overcome this limitation, we introduce a relative action formulation 
which enables successful policy training and deployment using its approximate kinematics data. A promising outcome of this approach is that the large repository of clinical data, which contains approximate kinematics, may be directly utilized for robot learning without further corrections. We demonstrate our findings through successful execution of three fundamental surgical tasks, including tissue manipulation, needle handling, and knot-tying.

Surgical Robot Transformer (SRT): Imitation Learning for Surgical Tasks
Ji Woong Kim-
Tony Z. Zhao
Samuel Schmidgall
Anton Deguet
Marin Kobilarov
Chelsea Finn
Axel Krieger
Johns Hopkins University'
Stanford University? hwtpssforrrrr..ttu...... github. io/
Needle pickup + handover
Lift tissue
Knot tying
Da Vinci Research Kit (dVRK)
Suture needle
Suture string
Stereo camera
Clinical setup
Lifted tissue
Suture needle
Suture string
Our setup
Phantom tissue
Wrist cameras
Figure 1: (Left): The da Vinci Surgical Research Kit (dVRK) system is equipped with surgical endoscope and wrist cameras. (Right): Three fundamental surgical tasks learned, including lift tissue (i.e. tissue retraction), needle-picke pickup handover. knot-tying which are among most common surgical tasks.
Abstract: We explore whether surgical manipulation tasks be learned on the da Vinci robot via imitation learning. However, the da Vinci system presents unique challenges which hinder straight-forward implementation of imitation learning. Notably, its forward kinematics is inconsistent due to imprecise joint measurements, and naively training policy using such approximate kinematics data often leads to task failure. overcome this limitation, we introduce a relative action formulation which enables successful policy training and deployment using its approximate kinematics data. A promising outcome of this approach is that the large repos- itory of clinical data, which contains approximate kinematics, may be directly utilized for robot learning without further corrections. We demonstrate our findings through successful execution of three fundamental surgical tasks, including tissue manipulation, needle handling, and knot-tying.
Keywords: Imitation Learning, Manipulation, Medical Robotics
1 Introduction
Recently, large-scale imitation learning has shown great promise in creating generalist systems for manipulation tasks [1]. Prior research in this area has mostly focused on learning day-to-day household activities. However, an under-explored area with high potential is the surgical domain, particularly with the use of Intuitive Surgical's 's da Vinci robot. These robots are deployed globally and possess immense scaling potential: of 2021, over 10 million surgeries have been performed using 6,500 da Vinci systems in 67 countries, with 55,000 surgeons trained on the system video and kinematics data recorded for post-operative analysis, resulting in a large repository of
[2]. Often, the
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Relative pose trajectory (pos, rot, gripper<)
Input
Policy
L/R stereo cam
L/R wrist cam
Figure 2: We propose policy design which only takes images as input and outputs relative pose trajectories for both arms. Modeling policy actions relative motion is key ingredient that makes robot learning work on the dVRK.
demonstration data. Utilizing such large scale data holds significant potential for building generalist systems for autonomous surgery [3].
However, robot learning on the da Vinci presents unique challenges. The hardware suffers from inaccurate forward kinematics due to potentiometer-based joint measurements, hysteresis, and overall flexibility and slack in its mechanism [4]. These limitations result in the robot's failure to perform simple visual-servoing tasks [5]. As discover in this work, naively training a policy using such approximate kinematics data almost always leads to task failure. For instance, policy trained to output absolute end-effector poses, which is a common strategy to train robot policies, achieves near-zero success rates across all tasks explored in this work, including tissue lift, needle pickup and handover, and knot-tying (Fig. 1). To achieve robot learning at scale, we must devise a strategy that leverages such approximate kinematics data effectively.
Towards this end, we present approach for robot learning the da Vinci using its approximate kinematics data. Intuitively, our approach is based the observation that the relative motion of the robot is much more consistent than its absolute forward kinematics. We thus model policy actions as differential motion and further explore its variants to design the most effective action representation for the da Vinci. We find that training an imitation learning algorithm using such relative formulation shows robustness to various configuration changes to the robot, even those known to significantly disrupt the robot's forward kinematics. Specifically, the da Vinci tools can be removed and reinstalled and all the robot joints can be freely moved, including the notoriously inaccurate set-up joints [4], without significantly impacting policy performance.
Additionally, we explore the use of wrist cameras in the surgical workflow. While not commonly employed clinical settings, wrist cameras have demonstrated effectiveness in improving policy performance and facilitating generalization to out-of-distribution scenarios, such as varying workspace heights unfamiliar visual distractions [6]. We thus evaluate their impact on performance and practical potential by designing removable brackets that enable easy sharing across various surgical instruments.
Overall, our results indicate that the relative motion on the da Vinci is more consistent than its absolute motion. Following this result, we further observe that carefully chosen relative action representation can sufficiently train policies that achieve high success rates surgical manipulation tasks. Additionally, using wrist cameras significantly improves policy performance, especially during phases of the procedure when precise depth estimation is crucial. In robustness tests, our model demonstrates the ability to generalize to novel scenarios, such as in the presence of unseen 3D suture pads and animal tissues, showing promise for future extensions into pre-clinical research.
Our main contributions are: (i) a successful demonstration of imitation learning on the Vinei while using its approximate kinematies data and without requiring further kinematics corrections, while drastically outperforming the baseline approach; (ii) experiments showing that imitation learning can effectively learn complex surgical tasks and generalize to novel scenarios such in the presence of

---

unseen realistic tissue; (iii) ablative experiments demonstrating the importance of wrist cameras for learning surgical manipulation tasks.
2 Related Work
Manipulation and Imitation Learning Imitation learning enables robots to learn from expert demonstrations [7]. Behavioral cloning (BC) is a simple instantiation of imitation learning that directly predicts actions from observations. Early works tackle this problem through the lens of motor primitives [8, 9, 10, 11]. With the development of deep learning and generative modeling. different architectures and training objectives have been proposed model the demonstrations end-to-end. This includes the use of ConvNets or VIT [12] for image processing [13, 14, 15], RNN or transformers for fusing history of observations [16, 17, 18], tokenization of the action space [19], generative modeling techniques such as energy -based models [20], diffusion [21] and VAEs [22, 23 Prior works also focus on the few-shot aspect of imitation learning, [24, 25, 26, 27], language conditioning [15, 18, 19, 28], co-training [19, 29, 28], retrieval [30, 31, 32], using play data [33, 34, 35, 36], using human videos [37, 38, 39, 40, 41, 42], and exploiting tpec-specific structures [43, 44, 45].
However, most of these prior works focus on table-top manipulation in home settings. Surgical tasks, on the other hand, pose unique set of challenges. They require precise manipulation of deformable objects, involve hard perception problems with inconsistent lighting and occlusions, and surgical robots may often have inaccurate proprioception and hysteresis [4] that are less pronounced in industrial arms. While in principle end-to-end imitation learning could capture these variations implicitly, it is unclear what design choices are important to enable effective learning in this regime.
We also note that in the dVRK community, the inaccuracies of the robot have been addressed via hand-eye calibration [46, 47, 483. However, we assume that the demonstration dataset has already been collected without hand-eye calibration c.g., the large-scale clinical data, which means precise ground-truth kinematics is not available during training. Therefore, hand-eye calibration is not a fundamental solution to our problem.
Autonomous Surgery Prior works in autonomous surgery primarily focus designing task- specific policies for specific tasks, such for suturing [49, 50, 51, 52], endoscope control [53, 54], navigation [55, 56, 57], and tissue manipulation [58, 591. However, these methods do not typically scale well across various tasks or generalize well to varying environmental conditions. In contrast, end-to-end imitation learning offers relatively simple solution to these shortcoming by only requiring good robot demonstrations. While prior works have also explored the use of imitation learning for surgical tasks [60, 61, 62, 55], its application to complex manipulation tasks like knot-tying remains unexplored, and practical design choices for implementing on the da Vinei have not been addressed.
3 Technical Approach
Consider the dVRK system, as illustrated in Fig. 3, which includes both the robot and teleoperation console for user interaction. The dVRK features endoscopic camera maipulator (ECM) and two patient side manipulators (PSM1, PSM2) sharing the same robot base. Each arm is sequential combination of set-up joints (SUJ) which are passive, followed by active joints which are motorized (Fig. 3). The passive joints are notoriously inaccurate due to using only potentiometers for joint measurements. The active joints use both potentiometers and motor encoders, providing improved precision. However, in general, the use of potentiometers throughout all the joints causes the forward kinematics of the arms to be inaccurate, even up 5cm error [4].
Using the dVRK console, the user collects many demonstrations of task, acquiring dataset D = [ J. where each trajectory = {(01, 11, 41), (oT, ær, aT)} is collection of observation or, proprioception F,, and actions a,, collected at time step f. Specifically, the observation or includes left/right surgical endoscope images and left/right wrist camera images, totaling four images (Fig. 2), proprioception the current pose of the PSMs w.r.t surgical endoscope tip frame denoted as I; = (gi, g where g = (p, R) E SE(3) and r denotes the left and right grippers respectively, and actions are the commanded desired waypoints reach specified via teleoperation control, denoted a, (ol, }.
3

---

ECM base (D)
ECM
Teleoperation console
PSM
PSM2 Tip (C)
PSM2
ECM
PSM1 Base (F)
Base (B)
Tip
PSM1 Tip (G)
Robot Base (A)
PSM: patient side manipulator
Set-up joints (imprecise joint readings)
ECM: endoscopic camera manipulator
Active joints (more precise joint readings)
Figure 3: The dVRK system consists of an endoscopic camera manipulator (ECM) and two patient side manipulators (PSMI, PSM2). Unfortunately. the dVRK arms are notorious for providing inconsistent forward kinematics. This is due to the setup joints (blue) only using potentiometers for joint measurements, which can be unrelible. The active joints (pink) use both potentiometers and motor encoders, improving precision.
Our objective is to learn surgical manipulation tasks via imitation learning. Given the robot's inaccurate forward kinematics, choosing the appropriate action representation is crucial. To illustrate this, investigate three action representations: camera-centric, tool-centric, and hybrid-relatins as shown in Fig. 4. The camera-centrr approach serves as a baseline, highlighting the limitations of modeling actions as absolute poses of the end-effectors. The tool-centric approach offer an improved formulation by modeling actions as relative motion, leading to higher success rates. The hybrid- relative approach further improves beyond tool-centric approach by modeling translation actions with respect to a fixed reference frame, further improving accuracy in translation movements. These approaches are detailed below:
1. Camera-centric actions:
We model camera-centric actions as absolute poses of the end-effectors w.r.t the endoscope tip frame. The setup is similar how position- based visual servoing ap- plications (PBVS) are implemented and is a natural choice on the dVRK. Specifically, the objective is to learn a policy ı that, given observation or at time predicts action sequence Ar, (at, .... Cc+c), where C denotes the action prediction horizon. The policy can thus be defined as ग. or + crc. This formulation is visually shown in Fig. 4.
2. Tool-centric actions: We model tool-centric actions as relative motion w.r.t the current end-effector frame, which is a moving body frame. Tool-centric actions can be defined as:
Alool
( of E [t, L C]; i E (1,r)}
Intuitively, the desired poses 9 are subtracted by the current end-effector poses of using the SE) subtraction rule, for each time up to horizon C' and for each corresponding left and right grippers. There are largely two benefits to adopting this action representation. A relative motion formulation is used, which we show later in Section 5, is more consistent compared to the absolute forward kinematics of the arms. Also, the subtraction cancels out the endoscope forward kinematics terms and the actions can be expressed in terms of the PSMs forward kinematics only. This effectively reduces the margin for errors since less joints are involved in representing actions. However, one caveat of this approach is that the delta motion defined w.r.t a moving reference frame. This requires the policy to implicitly localize the current end-effectee orientation from image observations, and output translations and rotations along the localized principal axes, which can be a challenging task. The policy objective can be defined as 0. Alool This formulation is visually shown in Fig. 4.
3. Hybrid Relative Actions: Similar to tool-centric actions, hybrid relative actions are modeled as relative motion but w.r.t two different reference frames. Specifically, the delta translations are

---

Camera-centric action
Tool-centric action
Hybrid-relative action
Endoscope
{
{E}
PSM1
PSM2
(G)
(G)
PoSg
Aposc
Aposg ArotG
rotg
ArotG
Policy action
Figure 4: We consider three options for modeling policy actions. (Left): Camera- centric approach models actions absolute end-effector poses w.r.t the endoscope tip frame. (Middle): Tool-centric approach models actions as delta positions and delta rotations defined w.r.t the current end-effector frame. (Righr): Hybrid relative approach models actions as delta positions defined w.r.t the endoscope tip frame and delta rotations defined w.r.t the current end-effector frame.
defined w.r.t endoscope tip frame and delta rotations are defined w.r.t the current end-effector frame. This formulation can be defined as follows:
kybrid A
[g e gi s ie {(,r}}
(2)
Where the subtraction operation e defined as:
e (f-- pi, (R)) Rc.)
(3)
Intuitively, the subtraction is performed between the corresponding translation and rotation elements i.c. vector subtraction for positions and SO(3) subtraction for rotations. A key distinction of this approach from the tool-centric approach lies in modeling the delta translation with respect to the fixed frame of the endoscope-tip. rather than the moving frame of the end-effector. This approach removes the burden for the policy to localize the end-effector's orientation to generate delta translations along the localized axes, thereby improving the quality of translation motion. The policy can be defined as क : o2 + Alybrid This formulation is visually shown in Fig. 4.
Implementation Details
To train our policies, [63]. The policies were trained using the endoscope and wrist cameras images as input, which are all downsized to image size of 224 x 224 x 3. The original input size of the surgical endoscope images were 1024 x 1280 x and the wrist images were 480 x 640 3. Kinematics data is not provided as input as commonly done in other imitation learning approaches because it is generally inconsistent due to the design limitations of the dVRK. The policy outputs include the end-effector (delta) position, (delta) orientation, and jaw angle for both arms. We leave further specific implementation details in Appendix A.
action chunking with transformers (ACT) [23] and diffusion policy
Experiments
In our experiments, we aim to understand the following key questions: (1) is imitation learning sufficient to learn challenging surgical manipulation tasks? (2) Is the dVRK's relative motion more consistent than its absolute forward kinematics? (3) Are using wrist cameras critical to achieving high success rates? (4) How well does our proposed model generalize to unseen novel scenarios? To answer these questions, we compare the task success rates of the policies trained using various action representations. We also directly compare the consistency of relative versus absolute motion by tracking reference trajectory using the various action representations and comparing their tracking
5

---

Reference configuration
Eval config 1 (robot shifted left)
Eval config 2 (robot shifted right)
Endoscope axis
PSM axis
Abdomen axis
Reference
Camera-centric
Tool-centric
Hybrid
The repeatability of all action representations are tested by repeating recorded reference trajectory under various robot configurations. (Leff): The first column shows perfect reconstruction trajectory for all action representations since the robot joints have not moved since when the reference trajectory was collected. (Middle, Right) When the robot shifted the left or to the right, the camera-centric action representation fails to track the reference trajectory while the relative action representations track them quite closely. This primarily errors. This experiment proves that in the presence of inconsistent joint measurements, relative motion can be more consistent.
Figure 5:
reference
set-up joints being moved, which causes significant joint measurement
Table 1: Trajectory tracking RMSE (mm) under various robot configurations
Ref config
Eval config
Eval config 2
Camera-centric
0.6
1.9
2.8
Tool-centric
0.9
0,9
0.7
Hybrid-relative 0.9
0.8
0.8
errors. We also explore the importance of wrist cameras by comparing the policy performance with and without them. Finally, we consider whether the proposed models can generalize to novel unseen scenarios, such as in the presence of animal tissues. These experiments are explored in the context of three tasks: lift tissue, needle pickup and handover, and knot-tying.
Experiment Setup During data collection, the robot is set up in a reference configuration as shown Fig. 5. In this configuration, 224 trials were collected for tissue lift, 250 trials needle pickup and handover, and 500 trials for knot-tying, all collected by single user across multiple days. During all experiments, dome simulating the human abdomen (Fig. 5) was used to roughly place the arms and the endoscope in approximately similar location using the same holes. The placement is only approximate because the holes are much larger than the endoscope and tool shaft size, and the tools have to be manually placed into the holes by moving the set-up joints.
Evaluating the Consistency of Relative Motion Absolute Forward Kinematies In this section we seek to understand whether relative motion on the dVRK is more consistent than its absolute forward kinematics. To test our hypothesis, we teleoperate a reference trajectory c.g., an infinity sign as shown Fig. 5. This trajectory is then represented in various action representations using the formulas presented Section 3. Then, we place the end-effector in the same initial pose and replay the trajectories using the various action representations under different robot configurations. These different configurations include shifting the robot workspace the left and the right side (Fig. 5). These workspace shifts cause the robot set-up joints to move, which the joints prone to cause large joint measurement erтоrs due to using only potentiometers for joint measurements. To

---

Lift tissue
Iriik
Test
Needle pickup handover
Init. / Test
Knot tying
Init. / test
Figure 6: (Top): Tissue lift task requires grabbing the corner of the rubber pad (i.e. tissue) and lifting it upwards. During training the corner kept within the red box and the configuration of corners test is shown. (Middle): Needle pickup and handover is selffcxplanatory. The needle was placed randomly inside the red box during training. At test time, the center hump of the needle was placed at nine locations shown, to enforce consistent setup during evaluation. (Bottom): Knot-tying requires creating loop using the left string, grabbing the terminal end of the string through the loop, and pulling the grippers away from each other. During training, location strings originating from the pads were randomly placed inside was centered in the red box as shown.
box, and at time,
Table 2: Success rates on three surgical tasks using various action representations
Tissue
Needle pick
Knot
lift
handover
tying
Grasp String
Whole Task
Test 1
Test 2
Grasp
Handover
Loop
Camera-centric
0/5
0/5
0/9
0/9
0/20
0/20
0/20
Tool-centric
5/5
5/5
9/9
5/9
20/20
20/20
18/20
ACT [23]
Hybrid-relative
5/5
5/5
9/9
9/9
20/20
20/20
18/20
Hybrid-relative (no wrist cam)
9/9
6/9
8/20
4/20
4/20
Hybrid-relative (pork backgrd)
9/9
9/9
Diffusion
Hybrid-relative
5/5
5/5
8/9
4/9
10/20
7/20
4/20
Policy [21]
compare their tracking errors, the replayed trajectories are annotated the end-effector (i.e. control point) in image coordinates and plotted, as shown in the bottom row of Fig. 5.
The plots in Fig. 5 and the numeric RMSE results in Table show that in the reference configuration, all action representation precisely reconstruct the reference trajectory, since the set-up joints have not yet moved. However, when the robot configuration is changed by moving the set-up joints and new erroneous joint measurments are obtained, the camera-centric action representation fails reconstruct the reference trajectory. Also, this error is not consistent for different robot configurations as shown in the trajectory plots in Fig. 5. For relative action representations, which include tool-centric and hybrid-relative action formulations, the reference trajectory is repeated more consistently, and their numeric errors do not vary significantly observed in Table 1. In summary, this experiment shows that relative motion on the dVRK is more consistent compared to its absolute forward kinematics in the presence of inconsistent joint measurement errors.
Policy Performance Using Various Action Representations We evaluate the policy performance using the various action representations tissue lift. needle pickup and handover, and knot-tying as shown in Table 2. The camera-centrr action representation performed poorly across all three tasks.
7

---

Because the joint measurements of the dVRK are inconsistent, the end-effectors almost always failed to reach the target objects (e.g., tissue corner, needle, and string) and often dangerously collided with the underlying rubber pads. The policy trained using tool-centric action representation showed improved performance across all three tasks. However, during needle pickup + handover when large rotations were involved. the handover phase of the task often failed. In particular, after picking up the needle, the left gripper had make a 90 degree rotation to transfer the needle to the opposing arm (Fig. 6). During this phase of the motion, the orientations of the grippers appeared correct, however, the translation motion appeared incorrect and seemed to be the cause of task failure. We conjecture this reason was due to grounding the policy actions to moving end-effector frame. The policy is required to localize the moving end-effector orientation using image observations and generate delta translations along the localized principal axes of direction, which can be a challenging task. To fix this issue, the hybrid relative motion action was used, which grounds the translation motion fixed frame of the endoscope-tip. This formulation improved the translation errors during the aforementioned needle handover phase and ultimately achieved the highest success rates across all three tasks. This best performing action representation was also implemented on Diffusion Policy, however, the performance was not as high as ACT. We also observed that Diffusion Policy often generated unsmooth motions, potentially due to not being able to adapt to the inconsistent kinematics of the robot.
Evaluating the Importance of Wrist Camera We also evaluate the importance of adding wrist cameras and their contribution to task success. To demonstrate this, we trained policies without wrist cameras on the needle pickup and handover and knot-tying tasks using the hybrid relative action formulation (Table 2). Overall, we observed that omitting wrist cameras lead to significant drop in performance. We conjecture that wrist cameras aid in scenarios where precise depth estimation is necessary. For instance, during the needle pickup and handover task, specifically during the latter phase transferring the needle, the wrist views clearly showed whether the needle was being navigated into the opposing grippers from afar. This additional view may have provided better context for task success. However, we observed that this level of information was difficult to discern from the third-person endoscopic view alone.
Evaluating Generalization We also evaluate the ability of our models to generalize to novel scenarios, such as under more clinically relevant background using animal tissues (pork and chicken) and an unseen 3D suture pad. Most of this evaluation remains qualitative and greater details are elaborated in Appendix B. In terms of quantitative results, we evaluate the hybrid-relative action formulation on the needle pick-up and handover task on a pig loin background. We observe that its overall success rate is quite high (Table 2), however, the quality of the motion and the accuracy of the needle grasps were much lower compared to those observed in the core experiments. In terms of qualitative results, we observe multiple instances of successful knot-tying achieved on pork tissue, and needle grasps on chicken background and on an unseen 3D pad (Appendix B).
Limitations and Conclusion
In this work, opt for using off-the-shelf large wrist cameras which not clinically relevant. However, the cameras may be replaced with much smaller ones (1-2mm diameter) and its mount can be further optimized by integrating quick-release mechanisms for swift transfer between surgical tools. Also, our model is limited as it can only act based on current observations and does not have the ability to modulate different behavior based on human instruction. We hope to address these issues in future work to further advance the autonomy of surgical robots.
In summary, we demonstrated an approach for imitation learning on the dVRK using its approximate kinematics data, without providing further post-procsssing corrections. The key idea of our approach was to rely on the more consistent relative motion of the robot, achieved by modeling policy actions as relative motion such as tool-centric and hybrid hybrid-relative actions. As mentioned in the introduction, we believe that our work is step towards leveraging the large repository of approximate surgical data for robot learning scale, without providing further kinematies corrections. We believe more research in this direction can further guide the path towards building general purposs systems towards autonomous surgery.
8

---

7 Acknowledgement
This material is based upon work supported by NSF/ERR 2144348, NIH R56EB033807, NSF DGE 2139757, and ARPA-H AYIAX000023.
References
[1] E. Collaboration, A. Padalkar, A. Pooley, A. Mandlekar, A. Jain, A. Tung, A. Bewley, A. Herzog,
A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Garg, A. Brohan, A. Rafin, A. Wahid, B. Burgess- Limerick, B. Kim, B. Schölkopf, B. Ichter, C. Lu, C. Xu, C. Finn, C. Xu, Chi, C. Huang, Shah, D. Büchler, D. Kalashnikov, D. Sadigh, E. Johns, F. Ceola, F. Xia, F. Stulp, G. Zhou, G. S. Sukhatme, G. Salhotra, G. Yan, G. Schiavi, G. Kahn, H. Su, H.-S. Fang, H. Shi, H. B. Amor, H. I. Christensen, H. Furuta, H. Walke, H. Fang. I. Mordatch, I. Radosavovic, I. Leal, J. Liang, J. Abou-Chakra, J. Kim,
C. Chan, C. Pan, C. Fu, C. Devin, Driess, D. Pathak,
J. Peters, J. Schneider, J. Hsu, J. Bohg, and J. Bingham. Open x-embodiment: Robotic learning datasets and rt-x models, 2023.
[2] Intuitive Surgical.
2021 intuitive sustainability report.
https:
/ // //www. wtwp:i/ntuliiiee..c//eeeeaaaa/// /ttt:////////////t///ttt///////
2021- 2022-nntultive--uuttaaaaii11111:
pdÉ, 202.
[3] Schmidgall, J. W. Kim, A. Kuntz, A. E. Ghazi, and Krieger. General-purpose foundation models for increased autonomy in robot-assisted surgery. arXiv preprint arXiv:2401.00672 2024.
[4] Z. Cui, J. Cartucho, S. Giannarou, and F. Rodriguez Baena. Caveats on the first-generation da vinci research kit: Latent technical constraints and essential calibrations. IEEE Robotics &amp; Automation Magazine, page 2-17, 2023. ISSN 155--223X. 10010...10///ma.22233330301
URL http: /x doi org/ /00 11.0..1099 110111191100 MRRA. 2. 3310863.
[5] M. Hwang, B. Thananjeyan, S. Paradis, D. Seita, J. Ichnowski, D. Fer, T. Low, and K. Goldberg. Efficiently calibrating cable-driven surgical robots with rgbd fiducial sensing and recurrent neural networks. IEEE Robotics and Automation Letters, 5(4):5937-5944, 2020. doi:10.1109/ L.R..2020.011076..
[6] K. Hsu, M. J. Kim, R. Rafailov, J. Wu, and C. Finn. Vision-based manipulators need to also see from their hands. In The Tenth International Conference on Learning Rep- resentations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https://openreria
net www.ooodddnoraaaao.ooooo
[7] D. A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. In NIPS, 1988.
[8] A. J. Ijspeert, J. Nakanishi, H. Hoffmann, P. Pastor, and S. Schaal. Dynamical movement primitives: learning attractor models for motor behaviors. Neural computation, 2013.
[9] Pastor, H. Hoffmann, T. Asfour, and S. Schaal. Learning and generalization motor skills by learning from demonstration. 2009 IEEE International Conference on Robotics and Automation, pages 763-768, 2009.
[10] J. Kober and J. Peters. Learning motor primitives for robotics. In 2009 IEEE International Conference on Robotics and Automation, 2009.
[11] A. Paraschos, C. Daniel, J. Peters, and G. Neumann. Using probabilistic movement primitives in robotics. Autonomous Robots, 42:529-551,, 2018.
[12] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words:
Transformers for image recognition at scale. 2021.

---

[13] R. Rahmatizadeh, P. Abolghasemi, L. Bölöni, and S. Levine. Vision-based multi-task ma- nipulation for inexpensive robots using end-to-end learning from demonstration. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3758-3765, 2017.
[14] Zhang, Z. McCarthy, O. Jow, D. Lee, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1-8, 2017.
[15] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-z: Zero-shot task generalization with robotic imitation learning. In Conference Robot Learning, 2022.
[16] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Mart Bart'in-Mart'in in. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning, 2021.
[17] N. M. M. Shafiullah, J. Cui, A. Altanzaya, and L. Pinto. Behavior transformers: Cloning modes with one stone. ArXiv, abs/2206. 11251, 2022.
[18] A. Brohan, N. Brown, Carbajal, Y. Chebotar, Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, J. Ibarz, Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, Leal, K.-H. Lec, S. Levine, Y. Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong. F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-1: Robotics transformer for real-world control at scale. In arXiv preprint arXiv:2212.06811 2022.
[19] A. Brohan, N. Brown, Carbajal, Chebotar, Chen, K. Choromanski, Ding. D. Driess, A. Dubey, Finn, P. Florence, Fu, M. Arenas, K. Gopalakrishnan, K. Han, Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T.-W. E. Lee, S. Levine, Y. Lu, H. Michalewski, I. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, G. Salazar, P. Sanketi, P. Sermanet, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In arXiv preprint vrXiv:2307.18816 2023.
[20] R. Florence, C. Lynch, A. Zeng, O. Ramirez, A. Wahid, L. Downs, A. S. Wong, J. Lee, I. Mordatch, and Tompson. Implicit behavioral cloning. ArXiv, abs/2109.00137. 2021.
[21] C. Chi, S. Feng, Du, Z. Xu, E. Cousineau, B. Burchfiel, and Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Syslems (RSS), 2023.
[22] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In Conference on Robot Learning, 2019.
[23] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. RSS, 2023.
[24] Y. Duan, M. Andrychowiez, C. Stadie, Ho, Schneider, Sutskever, P. Abbeel, and
W. Zaremba. One-shot imitation learning. ArXiv, abs/1703.07320 2017.
[25] S. James, M. Bloesch, and A. J. Davison. Task-embedded control networks for few-shot imitation learning. ArXiv, abs/1810.03237, 2018.
[26] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine. One-shot visual imitation learning via meta-learning. In Conference robot learning, 2017.
10

---

[27] P. Englert and M. Toussaint. Learning manipulation skills from single demonstration. The International Journal of Robotics Research, 37(1):137-154, 2018.
[28) Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna,
C. Xu, J. Luo. T. Kreiman, Y. Tan, D. Sadigh, C. Finn, and S. Levine. Octo: An open-source generalist robot policy. https: http::://cctooo http:///ccotoomo/eeeooo git.ub........ io, 2023.
[29] 7. Fu, T. Z. Zhao, and C. Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation, 2024.
[30] J. Pari, N. M. Shafiulaa,, S. P. Arunachalam, and L. Pinto. The surprising effectiveness of representation learning for visual imitation. arXiv preprint arXiv:2112.0151,, 2021.
[31] S. Nasiriany, T. Gao, A. Mandlekar, and Y. Zhu. Learning and retrieval from prior data for skill-based imitation learning, 2022.
[32] M. Du, S. Nair, D. Sadigh, and C. Finn. Behavior retrieval: Few-shot imitation learning by querying unlabeled datasets, 2023.
[33] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In Conference robot learning, pages
PMLR, 2020.
[34] C. Wang, L. Fan, J. Sun, Zhang, L. Fei-Fei, Xu, Zhu, and A. Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprint arX1v:2302.12222, 2023.
[35] Z. Cui, Y. Wang, N. M. M. Shafiullah, and L. Pinto. From play policy: Conditional behavior generation from uncurated robot data. arXiv preprint arXiv:2210. 10047, 2022.
[36] E. Rosete-Beas, O. Mees, G. Kalweit, Boedecker, and W. Burgard. Latent plans for task- agnostic offline reinforcement learning. In Conference Robot Learning, pages 1838-1849. PMLR, 2023.
[37] H. Xiong, Q. Li, Y.-C. Chen, H. Bharadhwaj. S. Sinha, and A. Garg. Learning by watching: Physical imitation of manipulation skills from human videos. In 202/ JEEE/RS/ International Conference Intelligent Robots and Systems (IROS), pages 7827-7834. IEEE, 2021.
[38] N. Das, S. Bechtle, T. Davchev, D. Jayaraman, A. Rai, and F. Meier. Model-based inverse reinforcement learning from visual demonstrations. In Conference Robot Learning, pages 1930-1942, PMIR, 2021.
[39] L. Smith, N. Dhawan, M. Zhang, P. Abbeel, and S. Levine. Avid: Learning multi-stage tasks via pixel-level translation of human videos. arXiv preprint arXiv:1912.04443. 2019.
[40] L. Shao, T. Migimatsu, Q Zhang, K. Yang, and Bohg. Concept2robot: Learning manipulation concepts from instructions and human demonstrations. The International Journal of Roboties Research, 40112-14):419--4443 2021.
[41] A. S. Chen, S. Nair, and C. Finn. Learning generalizable robotic reward functions from" in-the-wild" human videos. arXiv preprint arXiv:2103.16817 2021.
[42] A. D. Edwards and C. L. Isbell. Perceptual values from observation. arXiv preprint 1905.0900..07880 2019.
[43] A. Zeng, R. Florence, J. Tompson, Welker, J. Chien, M. Attarian, T. Armstrong, Krasin, D. Duong, V. Sindhwani, and Lee. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference Robot Learning. 2020.
[44] E. Johns. Coarse-to-fine imitation learning: Robot manipulation from single demonstration. 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 4613 46119, 2021.
11

---

[45] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. ArXiv, abs/2209.05451, 2022.
[46] O. Özgüner, T. Shkurti, S. Huang, R. Hao, R. C. Jackson, W. S. Newman, and M. C. Çav Cabera-robot calibration for the da vinci robotic surgery system. IEEE T Transactions on Au- tomation Science and Engineering, 171422115, 11 10..1109/AAAE.220022200000
[47] K. Pachtrachai, M. Allan, V. Pawar, S. Hailes, and D. Stoyanov. Hand-eye calibration for robotic assisted minimally invasive surgery without a calibration object. In 2016 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 2485-2492 2016. doi: 10w.11.111/RRRSS. 201.110052211.7753333
[48] Z. Wang, Z. Liu, Q. Ma, A. Cheng, Y.-h. Liu, S. Kim, A. Deguet, A. Reiter, P. Kazanzides, and R. H. Taylor. Vision-based calibration of dual rem-based robot arms in human-robot collaborative minimally invasive surgery. IEEE Robotics and Automation Letters, 3(2):672-679, 2018. doi:
[49] A. Shademan, R. S. Decker, J. D. Opfermann, S. Leonard, A. Krieger, and P. C. Kim. Su- pervised autonomous robotic soft tissue surgery. Science translational medicine, 8(337): 337ra64-337ra64, 2016.
[50] H. Sacidi, J. D. Opfermann, M. Kam, S. Wei, S. Léonard, M. H. Hsich, J. U. Kang, and A. Krieger. Autonomous robotic laparoscopic surgery for intestinal anastomosis. Science robotics, 7(62):eabj2908, 2022.
[51] F. Zhong, Y. Wang, Z. Wang, and Y.-H. Liu. Dual-arm robotic needle insertion with active tissue deformation for autonomous suturing, IEEE Robotics and Automation Letters, 4(3):2669-28) 2676, 2019.
[52] S. A. Pedram, P. Ferguson, J. Ma, F. Dutson, and J. Rosen. Autonomous suturing via surgical robot: An algorithm for optimal selection of needle diameter, shape, and path. In 20/7 IEEE International conference robotics and automation (ICRA), pages 2391-2399 IEEE, 2017.
[53] C. Gruijthuijsen, L. C. Gareia-Peraza-Herrene G. Borghesan, D. Reynaerts, J. Deprest, S. Ourselin, T. Vercauteren, and E. Vander Poorten. Robotic endoscope control via autonomous instrument tracking. Frontiers in Robotics and AI, 9:832208, 2022.
[54] X. Ma, Song. P. W. Chiu, and Z. Li. Autonomous flexible endoscope for minimally invasive surgery with enhanced safety. IEEE Robotics and Automation Letters, 4(3):2607-2613, 2019.
[55] J. W. Kim, P. Zhang, P. Gehlbach, lordachita, and M. Kobilarov. Towards autonomous eye surgery by combining deep imitation learning with optimal control. In J. Kober, F. Ramos, and C. Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedingy of Machine Learning Research, pages 2347-2358. PMLR, 16-18 Nov 2021. URL https: / wwwpppoeeeinng.. mlr. wwwppppsspee..p..pp... / vtl.vv15./.../1 htt.///mmmaa.....m html.
[56] W. Kim, C. He, M. Urias, Gehlbach, G. D. Hager, lordachita, and M. Kobilarov. Autonomously navigating surgical tool inside the eye by learning from demonstration. In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 7351-7357, 2020. doi: 199/CCA449445 1099ICRAA40445222229999333 200009999220...55
[57] W. Kim, S. Wei, P. Zhang, Gehlbach, J. Kang, I. Iordachita, and M. Kobilarov. Towards autonomous retinal microsurgery using rgb-d images. IEEE Robotics and Automation Letters, (4):3807-3814, 2024. doi:
[58] E. Tagliabue, A. Pore, D. Dall' Alba, E. Magnabosco, M. Piccinelli, and P. Fiorini. Soft tissue simulation environment to learn manipulation tasks in autonomous robotic surgery. In 2020 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 3261-3266
IEEE, 2020.
12

---

[59] C. Shin, P. W. Ferguson, S. A. Pedram, J. Ma, E. P. Dutson, and J. Rosen. Autonomous tissue manipulation via surgical robot using learning based model predictive control. In 2019 International conference on robotics and automation (ICRA), pages 2875-3881. IEEE, 2019.
[60] P. M. Scheikl, N. Schreiber, C. Haas, N. Freymuth, G. Neumann, R. Lioutikov, and F. Mathis- Ullrich. Movement primitive diffusion: Learning gentle robotic manipulation of deformable objects. IEEE Robotics and Automation Letters, 96):5338-5345, June 2024. ISSN 2377- 1101111.111111111 20.../.......1111.22............ 2024.
3774. 200:01011000/1222223333933999999 UtLhttp: /dxtt:://dx.ee /wtp:///xxodooo.oodo//oo 3382529.
[61] K. Kawaharazuka, K. Okada, and M. Inaba. Robotic constrained imitation learning for the peg transfer task in fundamentals of laparoscopic surgery, 2024. URL https://axxx. org/ abs/ 240.. 43440.
[62] B. Li, R. Wei, J. Xu, B. Lu, C.-H. Yee, C.-.. Ng. P.-A. Heng, Q. Dou, and Y.-H. Liu. 3d perception based imitation learning under limited demonstration for laparoscope control in robotic surgery, 2022. URL https://aaxaa
org/ abs/ 03195.
[63] C. Chi, S. Feng, Du, Z. Xu, E. Cousineau, B. Burchfiel, and Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[64] Song, C. Meng, and Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[65] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2015.
[66] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. ArXiv, abs/1505.04597, 2015. URL https: /appi. semant icscholar. brg/CorrpusID::719288
13

---

A Implementation Details
For ACT, main modifications include changing the input layers to accept four images, which include left/right surgical endoscope views and left/right wrist camera views. The output dimensions are also revised to generate end-effector poses, which amounts to a 10-dim vector for each arm (position [3] + orientation [6] + jaw angle [1] 10), thus amounting 20-dim vector total for both arms. The orientation modeled using 6D rotation representation following [21], where the 6 elements corrrespond to the first two columns of the rotation matrix. Since the network predictions may not generate orthonormal vectors, Gram-Schmice process is performed to convert them to orthonormal vectors, and a cross product of the two vectors are performed to generate the remaining third column of the rotation matrix. For diffusion policy, similar modifications are made such as changing the input and the output dimensions of the network appropriately. The specific hyperparameters for training shown in Table 3 and 4, Additionally, ACT was trained for approximately 12 15 hours and Diffusion Policy was trained for 24 hours for all tasks RTX 3090.
learning rate batch size # encoder layers
le-5
8
# decoder layers
7
feedforward dimension hidden dimension heads
3200
512
chunk size
100
beta dropout
10
0.1
Table 3: Hyperparameters of ACT.
learning rate
le-4
batch size
64
chunk size
32
scheduler
DDIM[64]
train and test diffusion steps
100, 100
ema power
0.75
backbone
ResNet18[65]
noise predictor
UNet[66]
RandomCCroptratio-00555 &
image augmentation
ColorJitterbbrihhtnes==333333,,1 contrast=0.4, saturation=0.5) &
KandonnRoattondegre==5555,,, 5.0])
Table Hyperparameters of Diffusion Policy.
14

---

B Generalization to Novel Settings
Needle pickup and handover
Chicken
leg
Pork
loin
3D pad
Knot tying
Pork
loin
Figure 7: We show qualitative examples of our model generalizing to novel scenarios beyond training settings. (Top): Successful zero-sho shot needle pickup and handover on chicken leg, pork loin, and on a 3D pad. (Bottom): Successful shot knot-tyng pork loin.
15


Reviews:
[{"review_id": "wDjRxqJc6Y", "reviewer": "Reviewer_XUen", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes relative pose as an effective action representation to learn complex surgical tasks can be from demonstrations that have imperfect/noisy kinematics data.", "summary_of_recommendation": "The paper presents a simple yet effective action representation to enable learning complex surgical skills from existing datasets containing approximate kinematics. The paper is well-written, and the experimental results sufficiently demonstrate its effectiveness and justify the core design choices.", "technical_quality": 5, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Please address the questions and concerns from above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KHfURa4qmv", "reviewer": "Reviewer_q2qZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose an imitation learning method to solve tasks that are relevant to robotic surgery such as suturing, tissue lift, and needle pickup. One of the major challenges with imitation learning on the da Vinci robot is dealing with the noisy forward kinematics. The authors hypothesize that modeling the action representation as relative motion of the robot might have better success across all tasks. To this end, the author compare camera-centric, tool-centric, and hybrid-relative actions to see which performs best.", "summary_of_recommendation": "I suggest a weak accept because while I interpret a lack of novelty in the proposed approach, the authors have succesfully demonstrated their approach working on real hardware. I believe this can be a very important step in doing imitation learning for surgical robotics.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "* Could the authors clarify if there is any novelty in their method beyond comparing the action representations?\n* It would be interesting to know how much data is required to get good policies. The authors report ~250 expert demonstrations. Could they get away with less?\n* The authors use ACT and Diffusion Policy but don’t mention how long either policy takes to train. They also don’t discuss inference times. This would be helpful in evaluating the method.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2K0nGHB264", "reviewer": "Reviewer_kUC3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 5, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper studies the imitation learning setting (i.e. the best action representation) of Da Vinci Research Kit on surgical tasks.", "summary_of_recommendation": "The paper is well structured and studied several settings in the surgical tasks. An important and bold attempt to apply large model toward surgical domain.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. Do we have fine-grained metrics in addition to the success rate? For example the \"progress\" the robot achieves so far before failure. \n2. Have you studied the scalability of the model? That is, have you try to increase / decrease the scale of the model and see if it works? I think that would be very useful to the community to see, given current amount of data, what's the largest possible size of the model.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers all agree that the manipulation capabilities of the presented work in real-world surgical experiments are impressive. However, there are concerns regarding the technical novelty of this approach as it seems to be limited to the action representation.
A clarification on the novelty of the presented approach and the action representation would benefit the paper.

Additional discussion on the potential risk of the approach would further strengthen the paper.
One potential risk is the smoothness of the action sequences produced by diffusion policies (even when combined with action chunking).
This issue has been extensively studied in previous work[1]. The paper would be further strengthened by discussing and comparing to this approach.

[1]: Scheikl et al, “Movement Primitive Diffusion: Learning Gentle Robotic Manipulation of Deformable Objects”, 2023

## Post Rebuttal
The rebuttal addressed all remaining concerns and further improved the already strong submission. The reviewers unanimously agree that this work should be published at CoRL 2024.
Decision: Accept