SHADOW: Leveraging Segmentation Masks for Cross-Embodiment Policy Transfer
Authors: Marion Lepert
Ria Doshi
Jeannette Bohg
Venue: CoRL 2024 (2024)
Keywords: Cross-embodiment learning
Imitation Learning
Manipulation
TL;DR: We introduce Shadow, an efficient data editing scheme for robust cross-embodiment learning from a source to a target robot. Shadow overlays composite segmentation masks on the input images to train and evaluate this policy.

Abstract: Data collection in robotics is spread across diverse hardware, and this variation will increase as new hardware is developed. Effective use of this growing body of data requires methods capable of learning from diverse robot embodiments. We consider the setting of training a policy using expert trajectories from a single robot arm (the source), and evaluating on a different robot arm for which no data was collected (the target). We present a data editing scheme termed Shadow, in which the robot during training and evaluation is replaced with a composite segmentation mask of the source and target robots. In this way, the input data distribution at train and test time match closely, enabling robust policy transfer to the new unseen robot while being far more data efficient than approaches that require co-training on large amounts of data from diverse embodiments. We demonstrate that an approach as simple as Shadow is effective both in simulation on varying tasks and robots, and on real robot hardware, where Shadow demonstrates over 2x improvement in success rate compared to the strongest baseline.

SHADOW: Leveraging Segmentation Masks for Cross-Embodiment Policy Transfer
Marion Lepert'
Ria Doshi
Jeannette Bohg'
Stanford University
UC Berkeley
Abstract: Data collection in robotics spread across diverse hardware, and this variation will increase as new hardware is developed. Effective use of this grow- ing body of data requires methods capable of learning from diverse robot embodi- ments. We consider the setting of training policy using expert trajectories from a single robot arm (the source), and evaluating on a different robot arm for which no data was collected (the target). We present a data editing scheme termed Shadow, in which the robot during training and evaluation is replaced with a composite segmentation mask of the source and target robots. this way, the input data dis- tribution at train and test time match closely, enabling robust policy transfer to the unseen robot while being far more data efficient than approaches that require co-training large amounts of data from diverse embodiments. We demonstrate that an approach as simple as Shadow is effective both in simulation varying tasks and robots, and on real robot hardware, where Shadow demonstrates an av- erage of over 2x improvement in success rate compared to the strongest baseline. Videos are available at attpss///shaeee cross- embodiment. github. io.
Introduction
S.Sgmmentscurea robet
3) Overlay
Train SOURCE
Genennte tegmentttion mark target mbot
Trein policy
Lots data
11 Segment target robet
Evaluate on TARGET
30 Overay monks
tegmentation
Run policy
data
Figure 1: Schematic of Shadow. Policy trained source robot and evaluated the target robot. No data is collected for the target robot. Observation images are overlaid with the segmentation masks of the source and the target robots. During training, the target robot rendered to be the same end-effector pose, Tee, as the source robot and vice-versa during evaluation.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Increasing the size of open-sourced robotics data is crucial to advancing robotics research. The last several years have seen exciting large-scale collaborative robot data collection efforts in an effort to address this need [1, 2, 3, 4]. However, many of these efforts are spread across various robot hardware, and this variation likely to increase as new hardware is developed and more contributors share their data. Therefore, effective use of this limited but growing body of data requires methods capable of learning from diverse robot embodiments.
As hardware improves, it is unrealistic to require collecting large amounts of data on every new emhodiment. Therefore, we need methods that can generalize to new embodiments without the need for large-scale ambodiment-speciff data collection.
One of the main reasons why policy learning across different embodiments is challenging is that robots look visually quite different (Figure 2) and vision based policies suffer dramatically when visual observations are disturbed. The number of different robot embodiments for manipulation in use by research labs today remains quite low - 18 as seen in OXE [3] and given current algorithms, far too little to provide enough visual diversity to generalize policies to unseen embodiments.
We consider the setting of training policy using expert trajectories from single robot arm, and evaluating different robot arm for which no data was collected. We present a data editing scheme termed Shadow (Figure 1). At train time, the source robot is replaced by composite segmentation mask: a mask of the source robot overlaid with a mask of the target robot such that its end effector pose matches that of the source robot. At test time, the converse data edit is performed: the target robot is replaced by a composite segmentation mask of the target robot overlaid with a mask of the source robot such that its end effector pose matches that of the target robot. The segmentation mask of either robot in the target pose can be easily generated since current joint angles, the kinematics, and CAD models of each robot are known.
The principal advantage of Shadow is that the input data distribution at train time and test time match closely. In either case, the "active" robot is replaced with a composite segmentation mask of the source and target robots. In contrast to other image augmentation and in-painting methods, Shadow deliberately discards the RGB details of the robots (which we show be unimportant see A.3). Instead, Shadow preserves the crucial gripper/object interaction, and the RGB details of the surroundings (which can be distorted with prior in in-painting methods [5]). Additionally, compared to methods that co-train on large datasets with diverse embodiments, our method is significantly more data efficient: it does not require any data to be collected on the target robot; and is able to generalize to an unseen target robot from data collected on only a single source robot. Finally, our method does not require an image of the scene with no robot, an assumption required by methods such as keypoint tracking [6] and affordance prediction [7].
To summarize: our main contribution is Shadow, efficient data editing scheme for robust cross-embodiment learning that uses composite segmentation masks to train policy using data collected on a source robot to transfer to an unseen target robot.
SOURCE
TARGET
Franka Gripper
Panda Robotiq 2F85
URSe
Sawyar
IIWA
Robotiq 2F85
Figure 2: In simulation, we evaluate the effectiveness of Shadow in transferring policy trained on Panda with Robotiq 2F85 gripper the target robot/gripper combinations shown here. On real-world hardware, we transfer to the Panda robot with the Franka gripper and the UR5e robot with the Robotiq 2F85 gripper.

---

2 Related Works
2.1 Cross-Embodiment Transfer
To improve policy transfer, prior work has utilized wrist cameras to minimize visual differences in the input observation space [8]. However, these works limit tasks to those that do not require a thorough view of the environment and make impossible to leverage the valuable amount of existing robot data collected from third-person viewpoints. Some works [9] mask robot pixels, train video prediction model that predicts non-robot pixels, and use visual foresight when rolling the policy. Such video prediction models are compute-intensive. On the architecture side, Furuta et. al. [10] trains task and morphology specific policy, distilling each combination into transformer. Similarly, Devin et. al [11] train a separate head for each robot and task. Other work [10, 12, 13, 14] follows this line of work assuming a known distribution of robots at eval-time. Another subset of multi-robot works condition on carphology-ppeciffe information [15, 16, 17, 18], or train latent vector for new robot from few shot trajectories. These works all require data collected from the target robot prior to evaluation, whereas our method aims to perform on an entirely new robot without any collected trajectories.
2.2 Lcrge-Scale talli-Emmbodieett Datasets
number recent works robot learning pre-train policies on large, diverse robotic datasets [19, 20, 21, 22, 23, 24] as a means of generalizing to new embodiments. Some works use Cartesian space actions [3, 25], but cannot be deployed zero-shot any prior knowledge of the robot's dynamics, such as a robot's URDF, nor do they align action spaces. Our method allows us to take policy trained on single robot and generalize to any single- arm robot. Other prior work [26] uses human demonstrations to extract object-centric trajectories that the target robot can use to collect its own demonstrations. However, we do not require the collection of target robot demonstrations nor do we impose object-centric constraints on our tasks.
robot because they do not leverage
2.3 Image Augmentation and In- painting
Image editing as form of domain randomization to build more robust policies is another approach to generalization [27, 28, 29]. SuSIE [30] queries a diffusion model for subgoal generation for goal-conditioned policies. Another work [31] simulates different robot viewpoints by augmenting existing data. Video in-painting [32] has been used to mask out human hands and replace them with robot grippers. These works rely on large diffusion models which are computationally expensive to train or fall short in instances where the delineation between the gripper and object must be very precise, as in-painting models can suffer from blurriness. Probably most similar our work is Mirage [5], which trains policy a source robot and inpaints the target, unseen robot over the source robot during evaluation time. In this way, no data is required for the downstream, target robot. However, this method also inherits the short-comings of in-iinninn painting models, limiting generalizability to more visually-complex backgrounds or settings where the source and target robot are drastically different (e.g., inpainting a WidowX over Franka robot).
3 Approach
Our method, Shadow, enables training a policy on one robot embodiment and deploying it on an- other robot embodiment without needing to collect any new data on the second robot embodiment. Our key insight is that generating segmentation masks of robot embodiments at a desired end- effector pose is easy and fast given access to robot's geometry, joint angles and cell-calibrated camera.
3

---

3.1 Problem Setup
We consider the scenario where we have access to dataset D ai........ "H ( i=1 of N trajectories on given robot embodiment, referred to as the source robot. We use the term embodiment to mean a rype of robot such as the Franka, Kinova, IIWA, etc. Each observation or == (!.- It) is a stack of RGB images from the T previous steps and each state 8t corresponds the joint positions of the robot. Each action a, is the absolute Cartesian pose of the end-effector. We evaluate our method on a different robot embodiment, referred to as the target robot, for which trajectories are collected. The pose of the camera is calibrated relative to the base of both robots, resulting in known transformations T and To T. where So and To are the base frames of the source and target robots, respectively, While setup is limited by the need for camera calibration parameters, this information is frequently collected in open-sources robot datasets (46% of Open-X Embodiment [3] data).
N
Our method does not require that the pose of the bases of the source and target robots be the same, but we assume that the distribution of locations of the objects to be manipulated relative to the camera stays the same. We assume that the background, lighting, and pose of the camera in the scene stays fixed for the two robots, robustness scene variations is outside this paper's scope.
Different embodiments cause significant distribution shift in the image observation space. We focus on how to train vision-based policies to be robust to this distribution shift and therefore only consider tasks that both the source and target robot embodiments can complete using similar strategies.
3.2 Shadow data editing
3.2.1 Training
For each image in our training dataset, It, we use robot proprioception, known kinematics and geometry of the robot, and camera parameters segment out the pixels corresponding to the source robot and set them to a fixed color, c. While can be any color, we use solid black for simplicity. We refer to this modified image as It,..
In order to generate a segmentation mask of the target robot, first we model a virtual rendering of the target robot with its end-effector in the same Cartesian pose as the end-effector of the source robot. The joint positions of the target robot can be solved using any robot controller; in our work, use operational space control [33]. We render the target robot image using a simulated camera with the same intrinsics and extrinsies, C,Camera, as the real world camera. We then extract the segmentation mask of the target robot from this image, and overlay the mask (with all its pixels set to c) onto It,, to obtain T.... If the target robot has a different gripper embodiment, we also overlay the target gripper segmentation mask set to c. For clarity, 4t, is L, modified to have the source robot "blacked out" with its segmentation mask, and the target robot mask further super-imposed on the image. We then N
train policy with imitation learning on the modified dataset D. = { ai...., Op a )
3.2.2 Evaluation
To evaluate our policy on the target robot, we follow a similar process as done during training. We take the current image J, of the scene and segment out the pixels corresponding to the target robot and set them to the fixed color c, obtaining It,.- We generate a virtual rendering of the source robot with its end-effector in the same Cartesian pose that of the target robot. We render the source robot image using a simulated camera with the same intrinsics and extrinsics, Camera as the real world camera. We then extract the segmentation mask of the target robot from this image, and overlay the mask onto Jt,, to obtain I,... This gives us ot, which forms the input to our policy at test time to get the next action. We execute the action using a blocking controller for accurate trajectory tracking.
Because we have access to an exact CAD model of the robot, the segmentation masks of the robots generated from virtual renderings have no loss of fidelity beyond noise introduced from camera calibration error. Therefore, in properly camera calibrated setup, we expect minimal distribution

---

shift between the images f. in our training set and those obtained during evaluation on either the source or target robot, enabling robust cross-embodiment policy transfer.
Results
We hypothesize that Shadow enables policy transfer to unseen target robots with minimal degrada- tion compared to policy trained and evaluated on raw source robot images. To test this, we evaluate method in simulation and real hardware, using Diffusion Policy for imitation learning [34).
4.1 Baselines
We compare our method to naive baseline: training the unmodified source robot data and evaluating on unmodified images of the target robot. We also compare to the closest method to ours, Mirage [5], a cross- emboddment method that leverages in in-painting to transfer between source and target robots. We re-implement Mirage use Diffusion Policy instead of the LSTM behavior cloning policy used in the original paper for a fair comparison to our method. We do not explicitly compare large model pre-trained diverse embodiments (e.g. Octo model [25], which is pre- trained a mixture of 25 datasets from OXE), because Mirage was already reported to outperform Octo by 20-50 percentage points for cross emboddmmbdi learning similar tasks [5].
We also compare to a model that is an upper-bound of performance on a given task: a model trained using raw images of the source robot and evaluated on the source robot (i.e., no data editing train test time, A.7). In each results figure, the dashed line denotes performance of this model. Asterisks denote methods that statistically inferior Shadow swwo-proportion z-test, 0.05).
4.2 Simulation experiments
Stack
Can
Stack Three
Coffee
Kitchen
Mug Cleanup
Figure The six tasks evaluated in simulation, shown from the viewpoint used during training.
We evaluate our method in the Mujoco [35] simulator using the Mimicgen [36] and Robomimic [37] datasets and the robot models from [38]. We use the Panda with the Robotiq 2F-85 gripper source robot and evaluate performance on the Sawyer, IIWA, and UR5e target robots with two grippers across six tasks (Figure 3). For each task, we 190-950 demos the source robot, depending on the amount of available data in Mimicgen and Robomimic for the source robot (A.5).
We compare the success rates between each method, task, and embodiment combination over 100 trials each (Figure 4, 5). The performance of the naive baseline is close 0% on all tasks and embodiments, indicating that the distribution shift caused by the change in embodiment drastically decreases performance even the simplest stack task. The Mirage baseline performs well on the easiest tasks Stack and Can but its performance degrades tasks that require higher precision such as the Stack Three task which requires balancing three objects on top of each other or the Coffee task which requires inserting a coffee pod into narrow slot. Mirage relies on in in-painting which causes distortion in the image and generates erroncous artifacts which change the image distribution. In contrast, Shadow achieves performance on the target robot comparable to the performance on the source robot in all tasks except Mug cleanup, in which it still significantly outperforms both baselines. The Mug Cleanup task is especially difficult for the Franka gripper because the gripper can jam against the side of the drawer due to its large width, whereas the source gripper does not.
5

---

Note that Mirage struggles even when the robot stays the same but only the gripper changes (Figure Panda). The is because the distances from the flange the robot to the end-effector control point are different between the Robotiq 2F-85 gripper and the Franka gripper. As a result, moving different grippers to same control point requires moving the flange the robots to differ- locations. Mirage still requires the use of in-painting this scenario, and Shadow still needs overlay two segmentation masks. Therefore, in simulation, changing only the gripper is challenging.
Stack
Can
Stack three
1.0
25 96
1.0
95
1.0
Success Rate
0.4
0.4
0.4
0.2
0.2
0.2
Panda
Sawyer
IMA
URSe
Panda
Sawyer
IMA
URSe
Panda
Sawyer
IMA
URSe
Coffee
Kitchen
Mug cleanup
59
1.0
Success Rate
0.6
0.6
0.6
46
45
0.4
0.4
0.4
0.0
0.0
Panda
Sawyer
IMA
URSe
Panda
Sawyer
URSe
Panda
Sawyer
IMA
URSe
Source robot
Target robot
Naive
Mirage
Shadow
Figure 4: Different robot, different gripper: Evaluation in simulation over six tasks and four target robots (Panda, Sawyer, IIWA, URSe; each with the Franka gripper) (100 roll-outs per evaluation). The source robot Panda+Robotiq 2F85 gripper. Dashed line: policy trained and evaluated on source robot. * denotes statistically inferior to Shadow (p < 0.05). Shadow outperforms Mirage on all tasks, and shows no performance degradation compared source robot in 5/6 tasks.
Stack
Can
Stack three
01---
Success Rate
0.8
0.0
0.0
0.6
04
0.4
0.2
0.2
0,2
Sawyer
IWA
URSe
Sawyer
IWA
URSe
Sawyer
IWA
URSe
Coffee
Kitchen
Mug cleanup
1.0
27
1.0
Success Rate
0.8
0.8
0.8
0.6
0.4
0.4
0.4
0.
0.2
0.2
0.0
Sawyer
IWA
URse
Sawyer
IWA
URSe
Sawyer
IWA
URSe
robot
robot
Mirege
Shadow
Figure 5: Different robot, same gripper: Evaluation in simulation six tasks and three target robots (Sawyer, IIWA, UR5e; each with the Robotiq 2F85 gripper) (100 roll-outs per evaluation). The source robot is Panda+Robotiq 2F85 gripper. Dashed line: policy trained and evaluated on source robot. * denotes statistically inferior to Shadow (p < 0.05). Shadow outperforms Mirage on all tasks, and shows no performance degradation compared the source robot in 5/6 tasks.

---

Panda (Source)
UR5e (Target)
IIWA (Target)
Black-only Shadow
0.94
0.02
0
0.98
0.97
0.97
Table 1: Ablation of target mask in sim on the Stack task. (100 roll-outs per evaluation)
The Shadow data editing procedure during training is the union of two changes to the raw image: (I) the source robot is "blacked out", (2) the mask of the target robot mask is overlaid. To evaluate the respective contribution these two separable edits, we ablate (2) and consider a data editing procedure that only blacks out the source robot (Table 5). "Black-only" does not enable policy transfer to target robots URSe or IIA all, while Shadow results in near-perfect performance. This demonstrates that the overlay of the target robot mask is crucial for the efficacy of Shadow.
4.3 Real world experiments
Mug
Cups
Hexagon
Blocks
Figure 6: The tasks evaluated in real-world experiments. Mug and Hexagon require placing the respective objects in the wooden container. Cups and Blocks require stacking the respective objects.
In the real world, we use the Panda robot with the Robotiq gripper as the source robot and collect 300 demos per task. We consider two cross-embodiment scenarios. First, keep the robot, but change the Robotiq gripper Franka gripper. As explained above, this scenario is challenging because the distance the end-effector control point is different between the grippers, meaning that the configuration of the Panda robot will be different between the source and target scenarios for the same action. Second, we keep the same gripper, but change the Panda robot to the UR5e robot. We compare Mirage and Shadow on each task and embodiment combination using 50 trials each. For hardware safety, we not evaluate the naive baseline since it performs poorly in simulation.
Source robot policy images
Target robot observation
0.6
Shadow
Success
0.4
0.2
Mirage
Mug
Cups
Heragon
Blocks
Source
Target
Mirage
Shadow
Figure 7: Different robot, same gripper: Real-world evaluation over four tasks. The target robot is UR5e Robotiq 2F85 gripper, and the source robot is Panda Robotig 2F85 gripper. (50 roll-outs per evaluation) denotes statistically inferior to Shadow (p < 0.05). Shadow outperforms Mirage baseline on all tasks, and shows no performance degradation compared to source robot in 2/4 tasks.

---

Our results demonstrate that Shadow significantly outperforms Mirage in all real world tasks. We notice that the primary failure mode Mirage that it sometimes executes seemingly random actions that are irrelevant to the task, which we hypothesize is due to the distortion of the image caused by in-painting. In contrast, Shadow's primary failure mode lack precision, where sometimes slightly misses grasp and not able recover. Performance is lowest on the Hexagon and Blocks tasks because they require grasping wider objects which means the robots must precisely position themselves above the objects before closing the gripper. In contrast, wider range of gripper positions will result in successful grasp of the narrow rim of the cup mug's edge.
Source robot policy on raw images
Target robot observation
Success rate
Shadow
0.4
0.2
Mirage
0.0
Mug
Cups
Hoxagon
Blocks
Source
Target
Mirage
Shadow
Figure 8: Same robot, different gripper: Real-world evaluation over four tasks. The target robot Panda Franka gripper, and the source robot Panda Robotiq 2F85 gripper. (50 roll-outs evaluation) denotes statistically inferior to Shadow (p < 0.05). Shadow outperforms Mirage baseline on tasks.
5 Limitations
In order to correctly generate segmentation masks of the source and target robots, Shadow relies on accurate camera calibration parameters and accurate proprioception information. Additionally, Shadow requires a policy to be trained for every new target rohot embodiment, although data from a single source sufficient for any number of targets. In situations where the robot is partially occluded by objects in the scene, the overlaid segmentation mask will obfuscate these objects. This can be addressed with depth data, but may be limited by the depth data quality. Lastly, Shadow currently does not generalize to novel scenes, but it can be readily combined with orthogonal work on inducing robustness to scene variations to address this limitation. See A. for more details.
6 Conclusion
We propose Shadow, an effective method for cross-embodiment policy transfer. We show in simula- tion and on real hardware that we can train a policy on a dataset of trajectories collected on one robot and obtain robust performance on another robot for which no data has ever been collected. The main advantages of our approach are its simplicity, its close alignment of the train and test set distributions despite different embodiments, and its data efficiency compared co-training approaches.
There are many avenues for future work. First, while Shadow is theoretically compatible with target robot embodiments placed anywhere in a scene, further investigation needed to assess its robust- ness to different robot placements. Second, although we demonstrated Shadow's robustness across various target robots, each trained policy was specific to a source/target robot pair. Future work could explore Shadow's ability allow single policy generalize across multiple embodiments. Finally, improving robustness to camera calibration errors via data augmentation provides another clear opportunity for improvement.

---

Acknowledgments
This work is supported by the National Science Foundation under Grant Number 2327974. We thank Connor Yako and Ken Salisbury for letting borrow their URSe robot, Lawrence Yunliang Chen for helpful responses to our questions on Mirage, and Claire Chen for help with physical experiments.
References
[1] D. Shah, A. Sridhar, N. Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine. Vint: A foundation model for visual navigation, 2023.
[2] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishma, Dasari, S. Karamcheti, S. Nasiriany,
M. K. Srirama, L. Y. Chen, Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, Y. J. Ma, Miller, Wu, Belkhale, S. Dass, H. Ha, A. Jain, A. Lee. Y. Lee. M. Memmel, Park, I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. Lu, J. Mer- cat, A. Rehman, P. R. Sanketi, A. Sharma, C. Simpson, Q Vuong, H. R. Walke, B. Wulfe, T. Xiao, J. H. Yang, A. Yavary, T. Z. Zhao, C. Agia, R. Baijal, M. G. Castro, D. Chen, Q. Chen, T. Chung, J. Drake, E. P. Foster, J. Gao, D. A. Herrera, M. Heo, K. Hsu, J. Hu, D. Jackson, C. Le. Y. Li, K. Lin, R. Lin, Z. Ma, A. Maddukuri, Mirchandani, D. Morton, T. Nguyen, A. O'Neill, R. Scalise, D. Seale, V. Son, S. Tian, E. Tran, A. E. Wang, Y. Wu, Xie, J. Yang, Yin, Zhang, O. Bastani, G. Berseth, Bohg, K. Goldberg, A. Gupta, A. Gupta, D. Ja- yaraman, J. J. Lim, J. Malik, R. Martin-Martin, S. Ramamoorthy, D. Sadigh, S. Song, J. Wu, M. C. Yip, Y. Zhu, T. Kollar, S. Levine, and C. Finn. Droid: A large-scale in-the-wild robot manipulation dataset. 2024.
[3] E. Collaboration, Open x-embodiment: Robotic learning datasets and rt-x models, 2024.
[4] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manju- nath, Mordatch, Nachum, C. Parada, J. Peralta, Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, Singh, S. Sontakke, A. Stone, Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong. F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-1: Robotics transformer for real-world control at scale, 2023.
[5] L. Y. Chen, Hari, K. Dharmarajan, C. Xu, Q. Vuong, and K. Goldberg. Mirage: Cross- embodiment zero-shot policy transfer with cross-painting, 2024. URL https:///axiii..ooo abs/2402.1922 19249.
[6] H. Bharadhwaj. R. Mottaghi, A. Gupta, and S. Tulsiani. Track2act: Predicting point tracks from internet videos enables diverse zero-shot robot manipulation, 2024.
[7] S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak. Affordances from human videos versatile representation for robotics, 2023.
[8] Yang. D. Sadigh, and C. Finn. Polybot: Training one policy across robots while embracing variability, 2023. URL https://arxiv http://///aaaio.o/////////////// org/abs/2307 037119.
[9] E. S. Hu, K. Huang, O. Rybkin, and D. Jayaraman. Know thyself: Transferable visual control 09047.
policies through abbt--wwrreness, 2021. URL
[10] H. Furuta, Y. Iwasawa, Y. Matsuo, and S. S. Gu. A system for morphology-tssk genneraaattt via unified representation and behavior distillation, 2022. URL 2211.14296. 14296.
orgaass/
[11] C. Devin, A. Gupta, Darrell, Abbeel, and S. Levine. Learning modular neural net- work policies for multi-task and multi-robot transfer, 2016. URL https: https://arxiv. org/abs/ 1609. 07088.

---

[12] P. Jian, E. Lee, Z. Bell, M. M. Zavlanos, and B. Chen. Policy stitching: Learning transferable robot policies, 2023. URL http::ttp/// pesttss///rxvvv htt///avvvor///aaaaaa htsp/bbs/230000 13753.
[13] Z. Xiong, Beck, and S. Whiteson. Universal morphology control via contextual modulation, 2023. URL https: //axxiv. http:///aaa....oo///////2222222 11070.
[14] Y. Zhou, S. Sonawani, M. Phielipp. S. Stepputtis, and H. Amor. Modularity through atten- tion: Efficient training and transfer of language-condiiee conditioned policies for robot manipulation. In K. Liu, D. Kulic, and J. Ichnowski, editors, Proceedings The 6th Conference on Robot Learn- ing, volume 205 of Proceedings of Machine Learning Research, pages 1684-1695. PMLR, 14-18 Dec 2023. URL htttps:/pproceedingg mlr. pptss//v25//hoo222222 html..
[15] W. Huang, I. Mordatch, and D. Pathak. One policy to control them all: Shared modular policies for agent-agnostic control, 2020. URL https: .rgg wwrg/abs/20077 200./
[16) V. Kurin, M. Igl, T. Rocktäschel, W. Boehmer, and S. Whiteson. My body is a cage: the role of morphology in graph-based incompatible control, 2020. URL httpt://a/x/i hwtp:/ivv.rr/..s..../rrsbb/b 2010.01855 01856.
[17] D. Pathak, C. Lu, T. Darrell, P. Isola, and A. A. Efros. Learning to control self-assembling morphologies: A study of generalization via modularity, 2019. URL https://arxiv.orr abs/1902. 05551992.05566.
[18] T. Wang, R. Liao, J. Ba, and S. Fidler. Nervenet: Learning structured policy with graph neural networks. In International conference on learning representations, 2018.
[19] C. Eppner. A. Mousavian, and D. Fox. Acronym: A large-scale grasp dataset based simu- lation, 2020. URL https: / /ttp/v/wxvvvv//a////a//eaaeaaaa/ http//aavvvv..rabbbaaaa httg://bs/001...../
[20] H. Walke, K. Black, A. Lee, M. J. Kim, M. Du, C. Zheng, T. Zhao, Hansen-Estruch, Q. Vuong, A. He, V. Myers, K. Fang, C. Finn, and S. Levine. Bridgedata v2: A dataset for robot learning at scale, 2023. URL https://aaxii: wwtpexiiiorrg/////aeeo orggabs/2300s.oo. 12952.
[21] A. Depierre, E. Dellandrea, and L. Chen. Jacquard: A large scale dataset for robotic grasp detection. In 2018 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS). IEEE, Oct. 2018. doi: 1109/IR0S. 2018. 8593950.
URL http: /ttt:://x... //tp:/xxxxooooooo/////ooo/oo/o.t oroi/.0
[22] F. Ebert, Y. Yang. K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets, 2021. URL https:tt//// /ttp///axxvvv//aa wttaaaavww....a.aaaa..aeeeaaaaee arg/abs/21099 13396.
[23] S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection, 2016. URL org/abs/ /ttbs//163.00000 02199.
[24] H.-S. Fang, H. Fang, Z. Tang. J. Liu, C. Wang, J. Wang, H. Zhu, and C. Lu. Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot, 2023. URL https: / //aaxivv.oo wrg..rgg//...................... /aob/ 2207.005955....
[25] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, J. Luo, Y. L. Tan, L. Y. Chen, P. Sanketi, Q. Vuong, T. Xiao, D. Sadigh, C. Finn, and S. Levine. Octo: An open-soure generalist robot policy, 2024.
[26] A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y. Narang, L. Fan, Y. Zhu, and D. Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations, 2023. URL https::////. //tp:i////iii/////a wrrili...rr//ae org/abs/23100 17596.
[27] Z. Chen, S. Kiami, A. Gupta, and V. Kumar. Genaug: Retargeting behaviors to unseen situa- tions via generative augmentation, 2023. URL https: /wxvvv.ivvvv./////////a////aaava org/abs. 06671.
10

---

[28] Z. Mandi, H. Bharadhwaj, V. Moens. S. Song. A. Rajeswaran, and V. Kumar. Cacti: A framework for scalable multi-task multi-scene visual imitation learning, 2022. URL, https: 1/arxiv.org. org/abs//2123 05711
[29] T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang, J. Singh, C. Tan, D. M, J. Per- alta, B. Ichter, K. Hausman, and F. Xia. Scaling robot learning with semantically imagined experience, 2023. URL https: /ttpe://aaxxvvvii wawpii..rg//bbb abwpaaas...2//////// 11550.
[30] K. Black, M. Nakamoto, P. Atreya, H. Walke, C. Finn, A. Kumar, and S. Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models, 2023. URL https: // //arxiv.od/ org/ abs/aass.0 2310. 10639.
[31] N. Hirose, D. Shah, A. Sridhar, and Levine. Exaug: Robot-conditioned navigation poli- cies via geometric experience augmentation, 2022. URL https://aazii..org////a//////// 07450.
[32] Bahl, A. Gupta, and D. Pathak. Human-to-robot imitation in the wild, 2022. URL https: // //arxiv. org/ 220/ .9450..
[33] O. Khatib. A unified approach for motion and force control of robot manipulators: The opera- tional space formulation. IEEE Journal on Robotics and Automation, 3(1):43-53, 1987.
[34] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[35] E. Todorov, T. Erez, and Tassa. Mujoco: physies engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026- 5033. IEEE, 2012. 111.10.110/11R052222..6666
[36] A. Mandlekar, S. Nasiriany, Wen, Akinola, Narang, L.. Fan, Zhu, and D. Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. In 7th Annual Conference on Robot Learning, 2023.
[37] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang. R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters learning from offline human demonstrations for robot manipulation. In arXiv preprint arXiv:2108.03298, 2021.
[38] K. Zakka, Y. Tassa, and MuloCo Menagerie Contributors. lection of high-quality simulation models for MuloCo, 2022. google-deepmindd/mjjcco moaooooreengei....
MuloCo Menagerie: A col-
URL http::/github.com

---

A Appendix
A.1 Limitations
1. Camera calibration noise: Shadow relies on accurate camera calibration in order to generate segmentation masks of the source and target robots. We investigate the robustness to camera calibration noise in Section A.4 and find Shadow deteriorates proportional to the scale of noise added. In future work, we could attempt to render Shadow more robust to camera calibration error via (modest) noise injection during training.
2. Training is linear with number of embodiments: We demonstrated that Shadow performs well on various target robots, but each individual policy we trained was for a specific source/target robot pair. Note, however, that training an additional policy for a new target does not require any additional data to be collected (i.e., data from a single source is sufficient for any number of targets). future work, we also hope to evaluate Shadow's ability to allow a single policy to generalize over more than embodiments. Specifically, we could try overlaying segmentation masks of two more target robot embodiments and evaluating if Shadow continues to perform well.
3. Occlusions: In situations where the robot is (partially) occluded by objects in the scene, the overlaid segmentation mask will obfuscate these objects. As result, the model may have diffi- culty disambiguating between cases where the robot is in front of an object versus behind it. In simulation, this can be casily addressed by using depth information from the original scene to only mask robot pixels behind occluding objects. In real, this can be addressed with real-time depth data (either by using a stereo camera or monocular depth estimation), but may be limited by the depth data quality. Additionally, the source and target robots cannot be positioned such way that their mask overlay occludes significant part of the scene such that the model can no longer see the task at hand.
4. No generalization new scenes (yet): For each task, data collected on the source robot in single scene, without adding any diversity the background elements of the scene. As result, Shadow cannot generalize novel scenes either the source robot the target robot. However, our method can be used in combination with orthogonal work on inducing scene robustness (e.g. collecting data in different scenes with different distractor objects, lighting conditions and/or doing data augmentation) to enable Shadow to work scenarios where the source and target robot are in different scenes.
5. Accurate proprioception: Shadow relies on accurate measured joint angles and an accurate kinematic model of a robot in order to properly segment out robot pixels an image. This may make adoption of Shadow on low-cost and less precise hardware more challenging.
6. Embodiments rely on the same physical strategies: The goal of Shadow is to train policy to output the same end-effector actions regardless the visual appearance of the robot embod- iment. Therefore, Shadow works across embodiments and tasks where the same set of actions can be used to complete the task. Shadow is well suited to work with commonly used robots that have parallel jaw grippers and full 6-DOF end-effector control. However, the method may struggle in tasks with cluttered environments where embodiments might collide with the envi- ronment in different ways, requiring different physical strategies. Shadow is also not able to transfer between parallel jaw grippers and dexterous hands.
A.2 Does Shadow perform well on the source robot?
The goal of Shadow is to develop a policy that performs well on the target robot despite never having collected data on the target robot. this section, we investigate whether or not this same policy can also perform well on the source robot. simulation, we run evaluations on the most challenging cross-embodiment scenario: different robot and different gripper. Results Figure 9 demonstrate no significant change in performance between Shadow's 's performance the source and target robots across the majority of the tasks. For the Mug Cleanup task, see increased performance the
12

---

source robot, which makes sense given that the target robot uses the Franka gripper which tends to jam against the drawer in the Mug Cleanup task whereas the source robot with the Robotiq gripper does not suffer from the same problem.
three
1.0
1,0
1.0
0.5
0.5
0.5
0.0
Sawyer
Sawyer
Sawyer
Coffee
Kitchan
Mug cleenup
Succems
0.5
0.0
Sawryer
IMWA
URSe
Santer
KWA
URSe
Sawyer
IWA
URSe
Source
Figure 9: Shadow's performance on six simulation tasks. The source robot is the Panda + Robotiq 2F85 gripper and the target robots use the Franka gripper. Shadow performs equally well on the source and target robots across the majority of tasks. (100 roll-outs evaluation).
A.3 Does masking out robot pixels affect performance?
Shadow relies on masking out the robot's pixels in the observation images. Our intuition for the feasibility of this approach stemmed from an experiment we ran in simulation that showed that replacing a source robot with segmentation mask (i.e., discarding all RGB information about the source robot) had no effect on task performance (Table 2, left). In other words, the RGB values the robot itself not important for completing the tasks studied. Note that this section is independent of cross emboddment learning
simply replaced the source robot with its segmentation mask.
We ran a similar experiment on real robot hardware and came to the same conclusion: on all tasks, replacing the robot with its segmentation mask during training and evaluation did not affect task performance (Table 2, right).
Simulation
Real world
Stack
Can
Stack 3
Coffee
Kitchen
Mug
Hex.
Cups
Stack
Mug
Vanilla 0.97
0.96
0.86
0.95
0.95
0.88
0.92
0.92
0.7
0.98
Masked 0.99
0.96
0.86
0.93
1.0
0.84
0.89
0.92
0.76
1.0
Replacing the source robot with its segmentation mask does not affect task performance in either simulation or on real robot hardware. 100 roll-outs per evaluation simulation and 25 roll- outs per evaluation in real. Vanilla: policy trained/evaluated trained/evaluated on images with the source robot replaced by its mask. No comparisons between Vanilla and Masked for a given task are statistically significant.
Table 2:
raw RGB images, Masked: policy
A.4 Sensitivity to camera calibration
Shadow relies accurate camera calibration in order generate segmentation masks of the source and target robots. We investigated the sensitivity of Shadow's performance real hardware to additional noise introduced to camera calibration extrinsics at evaluation time on the target robot (Table 3). As expected, the performance of Shadow deteriorates proportional to the scale of noise added.
Note that did not deliberately add any camera calibration noise during training. In future work, could attempt to render Shadow more robust to camera calibration error via (modest) noise injection during training.
13

---

Ax (m)
A0
Mug
Hexagon
0"
0.98
0.76
0.01
5
0.64
0.16
0.02
10
0.20
Table 3: Sensitivity of Shadow to camera calibration error. We evaluated the success rate on real hardware on the target robot different noise conditions (source: Panda robot with Robotiq grip- per, target: URSe robot with Robotiq gripper). Ax and A denote the standard deviation of O-mean Gaussian linear and angular noise deliberately added to the camera extrinsic parameters during eval- uation, respectively. (25 roll-outs per evaluation).
A.5 Dataset generation details
To build training datasets in simulation, leverage existing datasets from Robomimic [37] and Mimicgen [36]. These datasets contain demonstrations for Panda with Franka gripper. We convert these demonstrations to our source embodiment, Panda with Robotiq gripper, by rolling out each demonstration from the original dataset onto our source embodiment. Due to small differences robot dynamics and collisions with the scene, not all rollouts are successful. We report final dataset sizes for each task in Table 4.
Task
Dataset origin
Number of demos
Stack
Mimicgen
951
Can
Robomimic
190
Stack three
Mimicgen
937
Coffee
Mimicgen
768
Kitchen
Mimicgen
863
Mug Cleanup
Mimicgen
660
Table Source robot data used to train Shadow
For real world experiments, collect 300 demonstrations per task on the source embodiment by teleoperating the robot with Oculus controller.
A.6 Policy implementation details
Table describes the hyperparameters used to train Diffusion Policy. The same hyperparameters were used for all methods, and we used the CNN-based Diffusion Policy architecture. In simulation, all tasks used the DDPM scheduler with 100 training diffusion steps and 100 inference steps. In the real world, all tasks used the DDIM scheduler with 100 training diffusion steps and 10 inference steps. All hyperparameters not listed are unchanged from the original Diffusion Policy paper.
Task
To
Ta
ImgRes
Batch size
Lr
Epochs
Horizon
Mug Cups
2
8
240
256
le44
2000
200
240
256
le-4
2000
200
Real
Hexagon
2
84
256
le-4
2000
200
Blocks
84
256
lc-4
2000
200
Stack
84
256
le-4
5000
200
Can
2
84
256
le-4
5000
200
Sim
Stack Three
2
84
256
le-4
5000
400
Coffee
2
84
256
le-4
5000
300
Kitchen
84
256
le-4
5000
750
Mug Cleanup
240
256
1e-4 5000
400
Table 5: Hyperparameters used for diffusion policy, To: observation horizon, Ta: action horizon,
Epochs: Number of training epochs, Horizon: max number of rollout steps
14

---

Our method can run at up to 20 Hz (for 84x84 images; or 10 Hz for 240x240), but we execute our policy on the real hardware at only 2Hz due to hardware limitations. the set experiments testing out a different gripper, we are forced to reduce our control frequency because the max con- trol frequency of Franka gripper is only 2Hz. In the set of experiments testing out different robot, we prioritized hardware safety, we were borrowing the UR5e, and chose to use the URSc's slow Movel. blocking command to send Cartesian end-effector pose commands because the Movel. command comes with inherent safety features. The MoveL blocking command forced us to slow our policy down to 2Hz.
A.7 Upper bound of performance
upper-bound For our main results, we compare our method to model that an upper- bound of performance on given task: a model trained using raw images of the source robot and evaluated on the source robot (i.e., no data editing at train or test time, no cross-embodiment transfer). We use the source robot instead of the target robot as an upper bound of the target robot's performance because training on the target robot and then evaluating the target robot would require collecting new dataset of demonstrations on the target robot. Because real world data is collected via human teleoperation, it is impossible to guarantee that the quality of the data collected on the source and target robots are the same. Therefore, comparing models trained on two separate datasets would not be a fair comparison. However, simulation, is possible to create a training dataset on the target robot that nearly identical the original training dataset on the source robot. This is achieved by rolling out the actions both the source and robot embodiments in positions. In Table 6 compare the performance between sain-on-sourrc/tsst-oo bound and a train-on- wan---ttetttttoon-tteeeee upper bound and find no significant difference between them.
with the same initial object
upper
Source-on-source success rate
Target robot w/ Robotig
Target-on-tare success rate
Target robot w/ Franka
Target-on-targe success rate
Panda Sawyer
0.97
Panda Sawyer
0.94
0.97
0.97
Stack
0.97
IIWA
0.98
IIWa
0.95
URSe
0.98
URSe
0.96
Panda
0.96
Panda
0.94
Sawyer
0.93
Sawyer
0.97
Can
0.96
IIWA
0.92
IIWA
0.96
URSe
0.96
URSe
0.95
Panda
0.95
Panda
0.93
Sawyer
0,91
Sawyer
0.95
Coffee
0.95
IIWA
0.90
IIWA
0.97
UR5e
0.88
URSe
0.9
Panda
0.95
Panda
0.96
Sawyer
0.99
Sawyer
1.0
Kitchen
0.95
IIWA
0.93
IIWA
0.99
URSe
0.95
URSe
0.98
Table Comparison of two upper-bounds of performance for Shadow. Source-on-Sourr refers to policy that is trained on the source embodiment, Panda robot + Robotiq gripper, and evaluated on the source embodiment. Target-on-taree refers policy that trained on the target embodiment and evaluated that same target embodiment.
15


Reviews:
[{"review_id": "j8ZAyhWlMi", "reviewer": "Reviewer_Hf9g", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a simple idea to show cross-embodiment robot behavior transfer between training and test times. Given training data on a source bot, this proposed method is supposed to transfer to any target robot. At training time, segmentation masks of both the source and target robots (virtual) are overlaid on a 3rd person view of the scene which the policy network is conditioned on. Similarly, at evaluation time, the end effectors are aligned and segmentation masks of the source (virtual) and target robots are overlaid on the 3rd person view. The policy works with end effector control and shows successful transfer across embodiments in simulation and the real world.", "summary_of_recommendation": "The paper presents real world results which outperforms the baseline. However, it's not clear if the method is scalable and actually useful. Notetheless it's a simple and interesting idea that works in the settings presented in the paper.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Please refer to the weaknesses mentioned above", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GnOwZtSuWT", "reviewer": "Reviewer_ASdD", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a method for policy transfer between two robotic embodiments by augmenting an image to include segmentation mask for both the reference and target robot.", "summary_of_recommendation": "Overall the method is well motivated and evaluated. I am unsure how much the strong requirements on the camera position limit the applicability of the method.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "## Questions\n\nHow were the sample demos collected? Teleop, trained RL agent, scripted policy?\n\nAre there requirements in terms of variety in the training data to enable the method to work?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GMN8NO1NNy", "reviewer": "Reviewer_PcxW", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The submission studies the setting where there is abundant demonstration data for one robot solving a task, and there is a need to solve the same task using a different robot embodiment. The proposed solution is incredibly simple: mask out the part of the image corresponding to the robot, and overlay a mask of a (rendered) image of the other (target or source, during training and evaluation respectively) robot. This requires access to a CAD model of the robot, knowledge of the kinematics and geometry of the robot, and known camera calibration. Given this information, the training and test input distributions are identical, and as long as the same strategy can be used to solve the problem with the target robot as with the source robot, performance should be close. The authors demonstrate the validity of their approach on both simulated and real-robot experiments.", "summary_of_recommendation": "I strongly recommend that this paper is accepted for publication. It is overall an excellent paper. It introduces a simple idea, explains it well, and executes it well. Experiments are well thought out, carried out, and explained. The empirical results are strong. And the main limitations that I can think of are carefully discussed and addressed.", "technical_quality": 5, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Summarizing the points above, here are the main questions I have for the authors:\n- Could the proposed work be extended to operate in new scenes?\n- Why choose the source performance as the upper-bound?\n\n######## Additional feedback ########\n\nThe following points are provided as feedback to hopefully help better shape the submitted manuscript, but did not impact my recommendation in a major way.\n\nIntro\n- \"several reasons. First...\" --> only 1 reason is given! What about the ability to execute the same motions with a new robot? What if the workspaces or topologies are radically different?\n\nSec 2\n- Ln 72 is the first mention of conditioning on URDFs or aligning action spaces: it comes out of nowhere. At this point I was wondering if the proposed approach would do these two things, and what exactly they meant.\n- Nice treatment of related work\n\nSec 3.1\n- The clear statement of assumptions and scope is much appreciated\n- The one assumption that I find concerning is that both robots can solve the tasks with similar strategies. It would be good to include a discussion of which tasks do (and do not) fit within this assumption (potentially for a particular pair of robots, or more generally)\n\nSec 3.2\n- I wonder if the authors could instead overlay the robots with some transparency to maintain more of the 3d geometry. I think this would require adding solid backgrounds to both robots where they _don't_ overlap, and a single common solid background to both robots where they do overlap. This should maintain the property that there is no distribution shift.\n    - Oh, I see that this was actually addressed in Appendix A.3 by demonstrating that RGB info is not needed. Please include pointers to this and other Appendices wherever relevant to point the reader to the relevant appendices\n\nSec 4\n- \"Mirage was already reported to outperform Octo...\" --> I recommend giving just a bit more detail on this (e.g., \"by margins of at least XXX and up to YYY\") to make a more compelling argument. Initially, I was unconvinced until I went to [5] to confirm.\n- \"The mug cleanup task is especially difficult for the Panda gripper ... whereas the source gripper does not\" --> I recommend the authors stick with either Franka or Panda to describe the Panda, instead of moving back-and-forth (e.g., the only time the target gripper is mentioned is in Fig 4 as \"Franka gripper\")\n- How many trials were run per task to obtain statistical significance results?\n- The results are strong even on the real robot, but it would've been nice to get a bit of a discussion of the big performance gaps in hexagon and blocks", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Paper summary**

This paper presents a method for reusing demonstration data collected on one robot for training another. The proposed method masks out the source robot in the image frame and overlays a mask of the target robot. The evaluation demonstrates how this approach is successful in both simulation and real-world experiments and results in better training than recent baselines.

**Review summary**

Summary of strengths:
+ The proposed method is both simple and effective.
+ The evaluation demonstrates greatly improved results compared to a very recent baseline (Mirage).
+ The experiments demonstrate generalizability across multiple robot platforms. 

Summary of weaknesses:
- The proposed method makes several assumptions about the availability and accuracy of data (well-calibrated joint data and camera pose). The evaluation would be improved by additional tests demonstrating how sensitive the proposed method is to these calibrations. **[The rebuttal acknowledges this limitation and poses that it is also a limitation of other SOTA  methods.]**
- The limitations section is present in the supplemental materials, but should be integrated into the main paper. **[This has been addressed in the revision.]**
- The evaluations also assume that the target robot is deployed in the same environment as the source data. The paper should address whether this assumption is necessary for the proposed method to work. **[The rebuttal acknowledges this limitation and proposes strategies for mitigating it.]**

**Response to rebuttal**

The authors have revised the paper to include the requested clarifications and justifications. All authors maintain positive ratings, and one reviewer increased theirs from weak to strong accept.
Decision: Accept