SHADOW: Leveraging Segmentation Masks for Cross-Embodiment Policy Transfer

Authors: marion  lepert
ria  doshi
jeannette  bohg
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 08:23:44

Abstract: Data collection in robotics is spread across diverse hardware, and this variation will increase as new hardware is developed. Effective use of this growing body of data requires methods capable of learning from diverse robot embodiments. We consider the setting of training a policy using expert trajectories from a single robot arm (the source), and evaluating on a different robot arm for which no data was collected (the target). We present a data editing scheme termed Shadow, in which the robot during training and evaluation is replaced with a composite segmentation mask of the source and target robots. In this way, the input data distribution at train and test time match closely, enabling robust policy transfer to the new unseen robot while being far more data efficient than approaches that require co-training on large amounts of data from diverse embodiments. We demonstrate that an approach as simple as Shadow is effective both in simulation on varying tasks and robots, and on real robot hardware, where Shadow demonstrates over 2x improvement in success rate compared to the strongest baseline.

Main paper: 
 introduction: 
 Figure 1 : Schematic of Shadow. Policy is trained on source robot and evaluated on the target robot. No data is collected for the target robot. Observation images are overlaid with the segmentation masks of the source and the target robots. During training, the target robot is rendered to be at the same end-effector pose, x ee , as the source robot and vice-versa during evaluation.Increasing the size of open-sourced robotics data is crucial to advancing robotics research. The last several years have seen exciting large-scale collaborative robot data collection efforts in an effort to address this need [1, 2, 3, 4] . However, many of these efforts are spread across various robot hardware, and this variation is likely to increase as new hardware is developed and more contributors share their data. Therefore, effective use of this limited but growing body of data requires methods capable of learning from diverse robot embodiments.As hardware improves, it is unrealistic to require collecting large amounts of data on every new embodiment. Therefore, we need methods that can generalize to new embodiments without the need for large-scale embodiment-specific data collection.One of the main reasons why policy learning across different embodiments is challenging is that robots look visually quite different (Figure 2 ) and vision based policies suffer dramatically when visual observations are disturbed. The number of different robot embodiments for manipulation in use by research labs today remains quite low -18 as seen in OXE [3] -and given current algorithms, is far too little to provide enough visual diversity to generalize policies to unseen embodiments.We consider the setting of training a policy using expert trajectories from a single robot arm, and evaluating on a different robot arm for which no data was collected. We present a data editing scheme termed Shadow (Figure 1 ). At train time, the source robot is replaced by a composite segmentation mask: a mask of the source robot overlaid with a mask of the target robot such that its end effector pose matches that of the source robot. At test time, the converse data edit is performed: the target robot is replaced by a composite segmentation mask of the target robot overlaid with a mask of the source robot such that its end effector pose matches that of the target robot. The segmentation mask of either robot in the target pose can be easily generated since current joint angles, the kinematics, and CAD models of each robot are known.The principal advantage of Shadow is that the input data distribution at train time and test time match closely. In either case, the "active" robot is replaced with a composite segmentation mask of the source and target robots. In contrast to other image augmentation and in-painting methods, Shadow deliberately discards the RGB details of the robots (which we show to be unimportant -see A.3). Instead, Shadow preserves the crucial gripper/object interaction, and the RGB details of the surroundings (which can be distorted with prior in-painting methods [5] ). Additionally, compared to methods that co-train on large datasets with diverse embodiments, our method is significantly more data efficient: it does not require any data to be collected on the target robot; and is able to generalize to an unseen target robot from data collected on only a single source robot. Finally, our method does not require an image of the scene with no robot, an assumption required by methods such as keypoint tracking [6] and affordance prediction [7] .To summarize: our main contribution is Shadow, an efficient data editing scheme for robust cross-embodiment learning that uses composite segmentation masks to train a policy using data collected on a source robot to transfer to an unseen target robot.  
cross-embodiment transfer: 
 To improve policy transfer, prior work has utilized wrist cameras to minimize visual differences in the input observation space [8] . However, these works limit tasks to those that do not require a thorough view of the environment and make it impossible to leverage the valuable amount of existing robot data collected from third-person viewpoints. Some works [9] mask robot pixels, train a video prediction model that predicts non-robot pixels, and use visual foresight when rolling out the policy. Such video prediction models are compute-intensive. On the architecture side, Furuta et. al. [10] trains a task and morphology specific policy, distilling each combination into a transformer. Similarly, Devin et. al [11] train a separate head for each robot and task. Other work [10, 12, 13, 14] follows this line of work assuming a known distribution of robots at eval-time. Another subset of multi-robot works condition on morphology-specific information [15, 16, 17, 18] , or train a latent vector for a new robot from few shot trajectories. These works all require data collected from the target robot prior to evaluation, whereas our method aims to perform on an entirely new robot without any collected trajectories.A number of recent works in robot learning pre-train policies on large, diverse robotic datasets [19, 20, 21, 22, 23, 24] as a means of generalizing to new embodiments. Some works use Cartesian space actions [3, 25] , but cannot be deployed zero-shot to a new robot because they do not leverage any prior knowledge of the robot's dynamics, such as a robot's URDF, nor do they align action spaces. Our method allows us to take a policy trained on a single robot and generalize to any singlearm robot. Other prior work [26] uses human demonstrations to extract object-centric trajectories that the target robot can use to collect its own demonstrations. However, we do not require the collection of target robot demonstrations nor do we impose object-centric constraints on our tasks.Image editing as a form of domain randomization to build more robust policies is another approach to generalization [27, 28, 29] . SuSIE [30] queries a diffusion model for subgoal generation for goal-conditioned policies. Another work [31] simulates different robot viewpoints by augmenting existing data. Video in-painting [32] has been used to mask out human hands and replace them with robot grippers. These works rely on large diffusion models which are computationally expensive to train or fall short in instances where the delineation between the gripper and object must be very precise, as in-painting models can suffer from blurriness. Probably most similar to our work is Mirage [5] , which trains a policy on a source robot and inpaints the target, unseen robot over the source robot during evaluation time. In this way, no data is required for the downstream, target robot. However, this method also inherits the short-comings of in-painting models, limiting generalizability to more visually-complex backgrounds or settings where the source and target robot are drastically different (e.g., inpainting a WidowX over a Franka robot). 
approach: 
 Our method, Shadow, enables training a policy on one robot embodiment and deploying it on another robot embodiment without needing to collect any new data on the second robot embodiment. Our key insight is that generating segmentation masks of robot embodiments at a desired endeffector pose is easy and fast given access to a robot's geometry, joint angles and a well-calibrated camera.We consider the scenario where we have access to a dataset D= o i 1 , s i 1 , a i 1 , . . . , o i H i , s i H i , a i H i N i=1of N trajectories on a given robot embodiment, referred to as the source robot. We use the term embodiment to mean a type of robot such as the Franka, Kinova, IIWA, etc. Each observation o t := (I t-T , .., I t ) is a stack of RGB images from the T previous steps and each state s t corresponds to the joint positions of the robot. Each action a t is the absolute Cartesian pose of the end-effector. We evaluate our method on a different robot embodiment, referred to as the target robot, for which no trajectories are collected. The pose of the camera is calibrated relative to the base of both robots, resulting in known transformations T Camera S0 and T Camera, where S 0 and T 0 are the base frames of the source and target robots, respectively. While our setup is limited by the need for camera calibration parameters, this information is frequently collected in open-sourced robot datasets (46% of Open-X Embodiment [3] data).Our method does not require that the pose of the bases of the source and target robots be the same, but we assume that the distribution of locations of the objects to be manipulated relative to the camera stays the same. We assume that the background, lighting, and pose of the camera in the scene stays fixed for the two robots, as robustness to scene variations is outside this paper's scope.Different embodiments cause significant distribution shift in the image observation space. We focus on how to train vision-based policies to be robust to this distribution shift and therefore only consider tasks that both the source and target robot embodiments can complete using similar strategies.For each image in our training dataset, I t , we use robot proprioception, known kinematics and geometry of the robot, and camera parameters to segment out the pixels corresponding to the source robot and set them to a fixed color, c. While c can be any color, we use solid black for simplicity. We refer to this modified image as I t, * .In order to generate a segmentation mask of the target robot, first we model a virtual rendering of the target robot with its end-effector in the same Cartesian pose as the end-effector of the source robot. The joint positions of the target robot can be solved using any robot controller; in our work, we use operational space control [33] . We render the target robot image using a simulated camera with the same intrinsics and extrinsics, T Camera T0 , as the real world camera. We then extract the segmentation mask of the target robot from this image, and overlay the mask (with all its pixels set to c) onto I t, * to obtain I t,• . If the target robot has a different gripper embodiment, we also overlay the target gripper segmentation mask set to c. For clarity, I t,• is I t modified to have the source robot "blacked out" with its segmentation mask, and the target robot mask further super-imposed on the image. We then train a policy with imitation learning on the modified datasetD • = o i 1,• , a i 1 , . . . , o i H i ,• , a i H i N i=1 .To evaluate our policy on the target robot, we follow a similar process as done during training. We take the current image I t of the scene and segment out the pixels corresponding to the target robot and set them to the fixed color c, obtaining I t,⋆ . We generate a virtual rendering of the source robot with its end-effector in the same Cartesian pose as that of the target robot. We render the source robot image using a simulated camera with the same intrinsics and extrinsics, T Camera, as the real world camera. We then extract the segmentation mask of the target robot from this image, and overlay the mask onto I t,⋆ to obtain I t,• . This gives us o t,• which forms the input to our policy at test time to get the next action. We execute the action using a blocking controller for accurate trajectory tracking.Because we have access to an exact CAD model of the robot, the segmentation masks of the robots generated from virtual renderings have no loss of fidelity beyond noise introduced from camera calibration error. Therefore, in a properly camera calibrated setup, we expect minimal distribution shift between the images I t,• in our training set and those obtained during evaluation on either the source or target robot, enabling robust cross-embodiment policy transfer.In situations where the robot is (partially) occluded by objects in the scene, the overlaid segmentation mask will obfuscate these objects. As a result, the model may have difficulty disambiguating between cases where the robot is in front of an object versus behind it. In simulation, this can be easily addressed by using depth information from the original scene to only mask robot pixels behind occluding objects. In real, this can be addressed with real-time depth data (either by using a stereo camera or monocular depth estimation), but may be limited by the depth data quality. Additionally, the source and target robots cannot be positioned in such a way that their mask overlay occludes a significant part of the scene such that the model can no longer see the task at hand. 
results: 
 We hypothesize that Shadow enables policy transfer to unseen target robots with minimal degradation compared to a policy trained and evaluated on raw source robot images. To test this, we evaluate our method in simulation and on real hardware, using Diffusion Policy for imitation learning [34] .We compare our method to a naive baseline: training on the unmodified source robot data and evaluating on unmodified images of the target robot. We also compare to the closest method to ours, Mirage [5] , a cross-embodiment method that leverages in-painting to transfer between source and target robots. We re-implement Mirage to use Diffusion Policy instead of the LSTM behavior cloning policy used in the original paper for a fair comparison to our method. We do not explicitly compare to a large model pre-trained on diverse embodiments (e.g. Octo model [25] , which is pretrained on a mixture of 25 datasets from OXE), because Mirage was already reported to outperform Octo by 20-50 percentage points for cross-embodiment learning on similar tasks [5] .We also compare to a model that is an upper-bound of performance on a given task: a model trained using raw images of the source robot and evaluated on the source robot (i.e., no data editing at train or test time, see A.7). In each results figure, the dashed line denotes performance of this model. Asterisks denote methods that are statistically inferior to Shadow (two-proportion z-test, α = 0.05). We evaluate our method in the Mujoco [35] simulator using the Mimicgen [36] and Robomimic [37] datasets and the robot models from [38] . We use the Panda with the Robotiq 2F-85 gripper as our source robot and evaluate performance on the Sawyer, IIWA, and UR5e target robots with two grippers across six tasks (Figure 3 ). For each task, we use 190-950 demos on the source robot, depending on the amount of available data in Mimicgen and Robomimic for the source robot (A.5).We compare the success rates between each method, task, and embodiment combination over 100 trials each (Figure 4, 5 ). The performance of the naive baseline is close to 0% on all tasks and embodiments, indicating that the distribution shift caused by the change in embodiment drastically decreases performance even on the simplest stack task. The Mirage baseline performs well on the easiest tasks -Stack and Can -but its performance degrades on tasks that require higher precision such as the Stack Three task which requires balancing three objects on top of each other or the Coffee task which requires inserting a coffee pod into a narrow slot. Mirage relies on in-painting which causes distortion in the image and generates erroneous artifacts which change the image distribution. In contrast, Shadow achieves performance on the target robot comparable to the performance on the source robot in all tasks except Mug cleanup, in which it still significantly outperforms both baselines. The Mug Cleanup task is especially difficult for the Franka gripper because the gripper can jam against the side of the drawer due to its large width, whereas the source gripper does not.Note that Mirage struggles even when the robot stays the same but only the gripper changes (Figure 4 -Panda). The is because the distances from the flange of the robot to the end-effector control point are different between the Robotiq 2F-85 gripper and the Franka gripper. As a result, moving the different grippers to the same control point requires moving the flange of the robots to different locations. Mirage still requires the use of in-painting in this scenario, and Shadow still needs to overlay two segmentation masks. Therefore, even in simulation, changing only the gripper is challenging. The Shadow data editing procedure during training is the union of two changes to the raw image:(1) the source robot is "blacked out", (2) the mask of the target robot mask is overlaid. To evaluate the respective contribution of these two separable edits, we ablate (2) and consider a data editing procedure that only blacks out the source robot (Table 5 ). "Black-only" does not enable policy transfer to target robots UR5e or IIWA at all, while Shadow results in near-perfect performance. This demonstrates that the overlay of the target robot mask is crucial for the efficacy of Shadow. In the real world, we use the Panda robot with the Robotiq gripper as the source robot and collect 300 demos per task. We consider two cross-embodiment scenarios. First, we keep the same robot, but change the Robotiq gripper to a Franka gripper. As explained above, this scenario is challenging because the distance to the end-effector control point is different between the grippers, meaning that the configuration of the Panda robot will be different between the source and target scenarios for the same action. Second, we keep the same gripper, but change the Panda robot to the UR5e robot. We compare Mirage and Shadow on each task and embodiment combination using 50 trials each. For hardware safety, we do not evaluate the naive baseline since it performs poorly in simulation. Our results demonstrate that Shadow significantly outperforms Mirage in all real world tasks. We notice that the primary failure mode of Mirage is that it sometimes executes seemingly random actions that are irrelevant to the task, which we hypothesize is due to the distortion of the image caused by in-painting. In contrast, Shadow's primary failure mode is lack of precision, where it sometimes slightly misses a grasp and is not able to recover. Performance is lowest on the Hexagon and Blocks tasks because they require grasping wider objects which means the robots must precisely position themselves above the objects before closing the gripper. In contrast, a wider range of gripper positions will result in a successful grasp of the narrow rim of the cup or mug's edge.  
limitations: 
 In order to correctly generate segmentation masks of the source and target robots, Shadow relies on accurate camera calibration parameters and accurate proprioception information. Additionally, Shadow requires a policy to be trained for every new target robot embodiment, although data from a single source is sufficient for any number of targets. In situations where the robot is partially occluded by objects in the scene, the overlaid segmentation mask will obfuscate these objects. This can be addressed with depth data, but may be limited by the depth data quality. Lastly, Shadow currently does not generalize to novel scenes, but it can be readily combined with orthogonal work on inducing robustness to scene variations to address this limitation. See A.1 for more details. 
conclusion: 
 We propose Shadow, an effective method for cross-embodiment policy transfer. We show in simulation and on real hardware that we can train a policy on a dataset of trajectories collected on one robot and obtain robust performance on another robot for which no data has ever been collected. The main advantages of our approach are its simplicity, its close alignment of the train and test set distributions despite different embodiments, and its data efficiency compared to co-training approaches.There are many avenues for future work. First, while Shadow is theoretically compatible with target robot embodiments placed anywhere in a scene, further investigation is needed to assess its robustness to different robot placements. Second, although we demonstrated Shadow's robustness across various target robots, each trained policy was specific to a source/target robot pair. Future work could explore Shadow's ability to allow a single policy to generalize across multiple embodiments. Finally, improving robustness to camera calibration errors via data augmentation provides another clear opportunity for improvement. 
 
 Appendix: A.1 Limitations 1. Camera calibration noise: Shadow relies on accurate camera calibration in order to generate segmentation masks of the source and target robots. We investigate the robustness to camera calibration noise in Section A.4 and find Shadow deteriorates proportional to the scale of noise added. In future work, we could attempt to render Shadow more robust to camera calibration error via (modest) noise injection during training.Training is linear with number of embodiments: We demonstrated that Shadow performs well on various target robots, but each individual policy we trained was for a specific source/target robot pair. Note, however, that training an additional policy for a new target does not require any additional data to be collected (i.e., data from a single source is sufficient for any number of targets). In future work, we also hope to evaluate Shadow's ability to allow a single policy to generalize over more than two embodiments. Specifically, we could try overlaying segmentation masks of two or more target robot embodiments and evaluating if Shadow continues to perform well.No generalization to new scenes (yet): For each task, data is collected on the source robot in a single scene, without adding any diversity to the background elements of the scene. As a result, Shadow cannot generalize to novel scenes on either the source robot or the target robot. However, our method can be used in combination with orthogonal work on inducing scene robustness (e.g. collecting data in different scenes with different distractor objects, lighting conditions and/or doing data augmentation) to enable Shadow to work in scenarios where the source and target robot are in different scenes.Accurate proprioception: Shadow relies on accurate measured joint angles and an accurate kinematic model of a robot in order to properly segment out robot pixels in an image. This may make adoption of Shadow on low-cost and less precise hardware more challenging.6. Embodiments rely on the same physical strategies: The goal of Shadow is to train a policy to output the same end-effector actions regardless of the visual appearance of the robot embodiment. Therefore, Shadow works across embodiments and tasks where the same set of actions can be used to complete the task. Shadow is well suited to work with commonly used robots that have parallel jaw grippers and full 6-DOF end-effector control. However, the method may struggle in tasks with cluttered environments where embodiments might collide with the environment in different ways, requiring different physical strategies. Shadow is also not able to transfer between parallel jaw grippers and dexterous hands.A.2 Does Shadow perform well on the source robot?The goal of Shadow is to develop a policy that performs well on the target robot despite never having collected data on the target robot. In this section, we investigate whether or not this same policy can also perform well on the source robot. In simulation, we run evaluations on the most challenging cross-embodiment scenario: different robot and different gripper. Results in Figure 9 demonstrate no significant change in performance between Shadow's performance on the source and target robots across the majority of the tasks. For the Mug Cleanup task, we see increased performance on the source robot, which makes sense given that the target robot uses the Franka gripper which tends to jam against the drawer in the Mug Cleanup task whereas the source robot with the Robotiq gripper does not suffer from the same problem. A.3 Does masking out robot pixels affect performance?Shadow relies on masking out the robot's pixels in the observation images. Our intuition for the feasibility of this approach stemmed from an experiment we ran in simulation that showed that replacing a source robot with its segmentation mask (i.e., discarding all RGB information about the source robot) had no effect on task performance (Table 2 , left). In other words, the RGB values of the robot itself are not important for completing the tasks studied. Note that this section is independent of cross-embodiment learning -we simply replaced the source robot with its segmentation mask.We ran a similar experiment on real robot hardware and came to the same conclusion: on all tasks, replacing the robot with its segmentation mask during training and evaluation did not affect task performance ( We evaluated the success rate on real hardware on the target robot in different noise conditions (source: Panda robot with Robotiq gripper, target: UR5e robot with Robotiq gripper). ∆x and ∆θ denote the standard deviation of 0-mean Gaussian linear and angular noise deliberately added to the camera extrinsic parameters during evaluation, respectively. (25 roll-outs per evaluation).To build our training datasets in simulation, we leverage existing datasets from Robomimic [37] and Mimicgen [36] . These datasets contain demonstrations for a Panda with a Franka gripper.We convert these demonstrations to our source embodiment, a Panda with a Robotiq gripper, by rolling out each demonstration from the original dataset onto our source embodiment. Due to small differences in robot dynamics and collisions with the scene, not all rollouts are successful. We report the final dataset sizes for each task in Table 4 .  


Decision: **Paper summary**

This paper presents a method for reusing demonstration data collected on one robot for training another. The proposed method masks out the source robot in the image frame and overlays a mask of the target robot. The evaluation demonstrates how this approach is successful in both simulation and real-world experiments and results in better training than recent baselines.

**Review summary**

Summary of strengths:
+ The proposed method is both simple and effective.
+ The evaluation demonstrates greatly improved results compared to a very recent baseline (Mirage).
+ The experiments demonstrate generalizability across multiple robot platforms. 

Summary of weaknesses:
- The proposed method makes several assumptions about the availability and accuracy of data (well-calibrated joint data and camera pose). The evaluation would be improved by additional tests demonstrating how sensitive the proposed method is to these calibrations. **[The rebuttal acknowledges this limitation and poses that it is also a limitation of other SOTA  methods.]**
- The limitations section is present in the supplemental materials, but should be integrated into the main paper. **[This has been addressed in the revision.]**
- The evaluations also assume that the target robot is deployed in the same environment as the source data. The paper should address whether this assumption is necessary for the proposed method to work. **[The rebuttal acknowledges this limitation and proposes strategies for mitigating it.]**

**Response to rebuttal**

The authors have revised the paper to include the requested clarifications and justifications. All authors maintain positive ratings, and one reviewer increased theirs from weak to strong accept.

Reviews: review: title: title: Interesting to see cross embodiment transfer with a simple idea
paper_summary: paper_summary: This paper presents a simple idea to show cross-embodiment robot behavior transfer between training and test times. Given training data on a source bot, this proposed method is supposed to transfer to any target robot. At training time, segmentation masks of both the source and target robots (virtual) are overlaid on a 3rd person view of the scene which the policy network is conditioned on. Similarly, at evaluation time, the end effectors are aligned and segmentation masks of the source (virtual) and target robots are overlaid on the 3rd person view. The policy works with end effector control and shows successful transfer across embodiments in simulation and the real world.
main_review: main_review: Strength:

1. A simple method that shows successful transfer between robot embodiments
2. Real-world result outperforming baseline (a more complicated impainting method)

Weakness:
1. The method is prone to occlusions
2. The method is reliant on a well-calibrated camera (which the paper reports). It also relies on the robot joint angles to generate a segmentation mask. Hence, this relies on robot joint angles to be well-calibrated (which isn’t reported in the paper).
3. Missing video policy execution on the simulated settings
4. The Limitations section should be in the main text instead of the Appendix
5. I am curious about what would be the main use case for this approach.
    1. For a practical deployment, the robot embodiment is fixed. We can simply do a one-time data collection on the target embodiment.
    2. If this approach can combine data from multiple embodiments, then potentially the use case could be that the method can leverage large internet-scale data for training a policy that generalizes well. However, there is evidence for transfer from only a single source embodiment and we don't have evidence for transfer from multiple source embodiments.
questions: questions: Please refer to the weaknesses mentioned above
limitations: limitations: 2
review_summary: review_summary: The paper presents real world results which outperforms the baseline. However, it's not clear if the method is scalable and actually useful. Notetheless it's a simple and interesting idea that works in the settings presented in the paper.
title: title: Official review
paper_summary: paper_summary: The paper presents a method for policy transfer between two robotic embodiments by augmenting an image to include segmentation mask for both the reference and target robot.
main_review: main_review: ## Strengths

The method proposed in the paper engages in a meaningful way with the drawbacks of related prior work (mirage) by effectively leveraging a simpler approach which does not rely on difficult video inpainting to achieve a similar objective. The experiments substantiate that this strategy leads to meaningful improvements over the previous method, albeit by requiring some additional privileged information such as the exact calibrated camera position.

Originality) While the method is a relatively simple extension of inpainting methods such as Mirage, I think the idea is still novel and has potential to be expanded substantially in future work.

Clarity) The paper presents the core method well. In general, some additional details should be provided on the concrete details of the pipeline to simplify reproducibility. For example, the data collection method, simulators used (and their errors) etc. should be described in more detail, potentially in the appendix.

The experiments cover a decent range of embodiments (I appreciate the experiment on the deceptively simple gripper change).

Significance) The method sees to perform well and seems simple to set up. However, I discuss some apprehension below.


## Weaknesses

The method requires some privileged ground truth knowledge about the source robot and it's relationship to the target robot, especially the camera position and end-effector position. While the calibration issue is discussed, it is an open question about whether this is a substantial barrier for adoption of the method outside the lab.

Some details on the experiments are unclear, listed below.

## (Undiscussed) Limitations

One discussion I find missing from the paper is the question how different robot embodiments can be before the method starts running into problems. Since the masks of two robots are used, I assume some non trivial similarity in the geometry of the embodiment is required. I would like for the authors to discuss this.

Similarly, if two embodiments need different physical strategies (such as different grips) to achieve a similar task, would the authors still expect their method to generalize well?

Is Shadow robust against other changing details about the scene, such as the ability to transfer from one table top to another? Why or why not? It might be an unrealistic assumption that reference datasets on exactly the same environment are available.
questions: questions: ## Questions

How were the sample demos collected? Teleop, trained RL agent, scripted policy?

Are there requirements in terms of variety in the training data to enable the method to work?
limitations: limitations: 2
review_summary: review_summary: Overall the method is well motivated and evaluated. I am unsure how much the strong requirements on the camera position limit the applicability of the method.
title: title: An excellent paper introducing a simple yet powerful idea
paper_summary: paper_summary: The submission studies the setting where there is abundant demonstration data for one robot solving a task, and there is a need to solve the same task using a different robot embodiment. The proposed solution is incredibly simple: mask out the part of the image corresponding to the robot, and overlay a mask of a (rendered) image of the other (target or source, during training and evaluation respectively) robot. This requires access to a CAD model of the robot, knowledge of the kinematics and geometry of the robot, and known camera calibration. Given this information, the training and test input distributions are identical, and as long as the same strategy can be used to solve the problem with the target robot as with the source robot, performance should be close. The authors demonstrate the validity of their approach on both simulated and real-robot experiments.
main_review: main_review: ######## Strengths ########

1. The main idea of this work, to overlay masks of source and target robots during training end testing, is simple, well explained, and well executed.
2. Connections to prior work are discussed thoroughly and in a way that motivates the proposed changes well
3. Limitations are addressed adequately both throughout the paper and in the Conclusions section
4. The empirical results are quite strong

######## Weaknesses ########

1. The main weakness in this work are well addressed in the manuscript: the need to train a new model for each new embodiment, the assumption that similar strategies can be used for source and target robots to solve the tasks, the need for a calibrated camera
2. One additional strong assumption, which is not really mentioned in the manuscript but is in the video, is that the target robot will be deployed in the same scene as the target robot
3. The choice of "oracle" is perhaps a bit misleading, and could be easily addressed in simulation experiments

######## Arguments ########

This is an excellently executed paper. The idea is very simple: if the main challenge in transferring from one robot to another is their visual appearances confusing vision-based policies, then let's make sure that the appearance during training matches the appearance during evaluation. While other works have also had this insight, this paper also comes up with the cleanest solution among them: masking out the robot pixels and _overlaying the other robot's mask_. This removes the need for any predictive model and simply requires knowledge of the geometry of the robot, which is readily available. And as expected, the results are very strong over a collection of simulated and real-world experiments. I am unaware of any existing approach that has achieved this level of zero-shot cross-embodiment transfer on vision-based policies.

I would appreciate a deeper dive into the assumption, stated in the accompanying video, that the target robot will be deployed in the same scene. If this wasn't the case, would we just require training across multiple scenes + having access to the new scene's camera calibration? Or would the method (and extensions of it) be inapplicable?

In the experimental section, the authors included the performance on the source robot as an "upper-bound". Wouldn't an upper-bound in performance require training no the _target_ robot and evaluating on the same target robot? Because it might in principle be easier to solve tasks with one robot than another. I understand that running this oracle on the real robot might be too expensive because it would require collecting training data on the target robot, but in simulation it should be feasible. Am I missing something?
questions: questions: Summarizing the points above, here are the main questions I have for the authors:
- Could the proposed work be extended to operate in new scenes?
- Why choose the source performance as the upper-bound?

######## Additional feedback ########

The following points are provided as feedback to hopefully help better shape the submitted manuscript, but did not impact my recommendation in a major way.

Intro
- "several reasons. First..." --> only 1 reason is given! What about the ability to execute the same motions with a new robot? What if the workspaces or topologies are radically different?

Sec 2
- Ln 72 is the first mention of conditioning on URDFs or aligning action spaces: it comes out of nowhere. At this point I was wondering if the proposed approach would do these two things, and what exactly they meant.
- Nice treatment of related work

Sec 3.1
- The clear statement of assumptions and scope is much appreciated
- The one assumption that I find concerning is that both robots can solve the tasks with similar strategies. It would be good to include a discussion of which tasks do (and do not) fit within this assumption (potentially for a particular pair of robots, or more generally)

Sec 3.2
- I wonder if the authors could instead overlay the robots with some transparency to maintain more of the 3d geometry. I think this would require adding solid backgrounds to both robots where they _don't_ overlap, and a single common solid background to both robots where they do overlap. This should maintain the property that there is no distribution shift.
    - Oh, I see that this was actually addressed in Appendix A.3 by demonstrating that RGB info is not needed. Please include pointers to this and other Appendices wherever relevant to point the reader to the relevant appendices

Sec 4
- "Mirage was already reported to outperform Octo..." --> I recommend giving just a bit more detail on this (e.g., "by margins of at least XXX and up to YYY") to make a more compelling argument. Initially, I was unconvinced until I went to [5] to confirm.
- "The mug cleanup task is especially difficult for the Panda gripper ... whereas the source gripper does not" --> I recommend the authors stick with either Franka or Panda to describe the Panda, instead of moving back-and-forth (e.g., the only time the target gripper is mentioned is in Fig 4 as "Franka gripper")
- How many trials were run per task to obtain statistical significance results?
- The results are strong even on the real robot, but it would've been nice to get a bit of a discussion of the big performance gaps in hexagon and blocks
limitations: limitations: 3
review_summary: review_summary: I strongly recommend that this paper is accepted for publication. It is overall an excellent paper. It introduces a simple idea, explains it well, and executes it well. Experiments are well thought out, carried out, and explained. The empirical results are strong. And the main limitations that I can think of are carefully discussed and addressed.