Polybot: Training One Policy Across Robots While Embracing Variability
Authors: Jonathan Heewon Yang
Dorsa Sadigh
Chelsea Finn
Venue: CoRL 2023 Poster (2023)
Keywords: vision-based manipulation
multi-robot generalization

Abstract: Reusing large datasets is crucial to scale vision-based robotic manipulators to everyday scenarios due to the high cost of collecting robotic datasets. However, robotic platforms possess varying control schemes, camera viewpoints, kinematic configurations, and end-effector morphologies, posing significant challenges when transferring manipulation skills from one platform to another. To tackle this problem, we propose a set of key design decisions to train a single policy for deployment on multiple robotic platforms. Our framework first aligns the observation and action spaces of our policy across embodiments via utilizing wrist cameras and a unified, but modular codebase. To bridge the remaining domain shift, we align our policy's internal representations across embodiments via contrastive learning. We evaluate our method on a dataset collected over 60 hours spanning 6 tasks and 3 robots with varying joint configurations and sizes: the WidowX 250S, Franka Emika Panda, and Sawyer. Our results demonstrate significant improvements in success rate and sample efficiency for our policy when using new task data collected on a different robot, validating our proposed design decisions. More details and videos can be found on our project website: https://sites.google.com/view/cradle-multirobot

Polybot: Training One Policy Across Robots While Embracing Variability
Jonathan Yang, Dorsa Sadigh, Chelsea Finn
Stanford University
jyang278cs.o httpe//ccssssaf...ooooe stanford.... edu
Abstract: Reusing large datasets is crucial to scale vision-based robotic manip- ulators to everyday scenarios due to the high cost of collecting robotic datasets. However, robotic platforms possess varying control schemes, camera viewpoints, kinematic configurations, and end-effector morphologies, posing significant chal- lenges when transferring manipulation skills from one platform to another. To tackle this problem, we propose set of key design decisions train single pol- icy for deployment on multiple robotic platforms. Our framework first aligns the observation and action spaces of our policy across embodiments via utilizing wrist cameras and a unified, but modular codebase. To bridge the remaining domain shift, we align our policy's internal representations across embodiments through contrastive learning. We evaluate our method on a dataset collected over 60 hours spanning 6 tasks and robots with varying joint configurations and sizes: the Wid- owX 250S, the Franka Emika Panda, and the Sawyer. Our results demonstrate sig- nificant improvements in success rate and sample efficiency for policy when using new task data collected on different robot, validating proposed de- sign decisions. More details and videos can be found on our project website: www.:/ee....ooll..oooooe///////e muttirooot
Keywords: rision-based manipulation, multi-robot generalization
Introduction
Leveraging large datasets is essential learning widely generalizable models in computer vision [1] and natural language processing [2]. In robotic manipulation. promising avenue of research lies in the collection of similar extensive, in-domain datasets with the aspiration that they bestow comparable benefits. However, while past datasets have demonstrated good generalizability within the same hardware setup, applying them to different robotic configurations has proven difficult [3]. This challenge stems from four sources of variation: control scheme, camera viewpoint, kinematic configuration, and end-effector morphology. Each of these factors can vary significantly across robotic setups, leading to large domain shift when transferring data collected one robot platform to another. In this paper, study the problem of how to mitigate this issue and effectively leverage robotic data across different platforms, making stride toward learning widely-applicable policies.
In an effort to bridge the aforementioned domain gap, prior works have made advancements in enabling transfer across subset of the factors of variation. Early works have studied cross- embodiment transfer across kinematic configurations from low-dimensional observations [4, 5]. More recently, efforts have shifted towards utilizing high-dimensional image observations, enabling transfer across robotic hardware with a fixed camera angle for 3-DoF tasks [6] and across end- effectors with a fixed embodiment [7]. Unlike these works, we do not constrain the camera view- point, embodiment, or low-level controller to be fixed. Instead, we propose several new design choices to align the observation and action spaces across robots, such as using wrist-mounted cam- cras and a shared inverse kinematics solver. Each of these choices greatly mitigate the domain shift across embodiments without compromising the generality of the robotic setup.
We integrate these design choices into a single framework that aligns the input, output, and internal representation spaces of our policy across embodiments. Our choice of utilizing front-mounted wrist
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Hobots
Demonstrations
Alignment
Action Aligmment
Internal Representation Aligmment
f(o,)) f(o,)) f(r,))
Irage Server
D
negat
Envion mennt
Robot-Specific Dynamics
Speciic Controllr
Speciti: Controller
ta (o).)
D
D
4
Shared invese
Figure 1: Our framework for generalization across multiple robots. We first standardize our observation space using front-mounted wrist cameras and our action space using a shared higher- level control environment. We then align our policy's ' internal representations using contrastive learning then finetune these representations learn robot-specific dynamics.
cameras aligns the input (or observation) space by naturally reducing the visual variation between embodiments. This allows us to remove assumptions about collecting data only under specific fixed angle. order to align the output space of our policy, we employ a shared inverse kinematics solver, while allowing the lower-level controller vary. Although we would ideally completely unify the output space by using single abstract action space across robots, this is infeasible due to disparities in action interpretation arising from hardware discrepancies. We instead learn a multiheaded policy with robot-specific heads that capture separate dynamics. Finally, we exploit consistent, low- dimensional proprioceptive state signal to align our policy's internal representations. This facilitates generalization across the remaining factors that may cause a domain shift.
Our main contribution is pipeline for learning end-to-end closed-loop policies that benefit from data collected on other robotic platforms. We empirically demonstrate that our method, Polybot: Cross-Robot A Daptation and ALignmEntWWe first show that given shared dataset of tasks, data from other robots for a new similar task can be transferred in a zero-shot fashion. In addition, providing as few as 5 demonstrations allows our system to achieve an average of > 70% success rate on difficult tasks which cannot be learned without data from other robots. We then show that aligning internal representations across robots can help transfer across remaining domain shifts, with our method having average of 19% higher success rate over an ablation without representation alignment. Finally, show that our multiheaded training method achieves high success on 6-DoF tasks outperforming naive blocking controllers that achieve no success at all.
and
2 Related Work
Learning from large datasets. The study of utilizing diverse datasets scale robotic learning currently gaining momentum in the research community. Examples of large-scale real-world robot datasets include single-embodiment datasets [8, 9, 3, 10, 11, 12] and simulation benchmarks [13, 14, 15, 16]. However, these methods have typically focused on transfer with single em- bodiment. To maximally reuse data from various sources, there have additionally been efforts to learn generalizable representations from sources other than robot data. One line of work is learn- ing from human videos, including unstructured egocentric videos [17, 18, 19, 20, 21, 22, 23, 24], egocentric videos collected with parallel gripper device [10], and in-the-wild videos [25, 26, 27]. Apart from human videos, there are works that representations from large-scale image datasets [28, 29, 30, 31], Youtube videos [32], and multi-source data from the Internet [33]. Although these methods can improve sample-efficiency and generalizability. they are often focused on learning vi- sual representations that can be fine-tuned for learning robot policies. Instead, we focus directly on learning visuomotor policies from robot data.
Learning across embodiments. A number of works have studied learning policies that transfer across embodiments. Some works in the past have focused on smaller aspects of robotic transfer,

---

including across 2-dimensional kinematic configurations [4], robotic models in simulation [5, 34], end-effector morphologies [7], dynamics [35, 36, 37, 38], and camera viewpoints [39]. A few works exist that focus on the task of transferring between robotic hardware [6, 40, 41]. General naviga- tion models (GNM) demonstrates cross-embodiment transfer in navigation for wheeled vehicles, quadrupeds, and drones [40]. Hu et al. study transfer to new manipulators via a robot-aware visual foresight model trained observations from fixed exterior camera by computing mask [6]. In contrast to these works, we focus transfer across wide range of high-dimensional observations, diverse robotic control schemes, and more complex 6-DoF tasks in the low-data regime.
Transfer learning. Cross-robot transfer is closely related to the general area of transfer learning, where there is large body of work on task generalization [42, 43, 44, 45, 46, 47, 48, 49, 33] and sim2real generalization [39, 50, 51, 52, 53, 54, 55]. These methods either only use a single embodiment largely ignore cross-embodimen differences. In addition, our work is closely related to the area of domain adaptation. Previous works include using GANs [56, 57, 55, 26], domain- adversarial learning [58, 59, 60], contrastive learning [17, 21]. and learning system system-invariant invariant feature representations [61, 62, 63] to transfer between two related domains. Our method focuses on the domain adaptation problem across robots by exploiting a low-dimnnsional proprioceptive signal.
3 Multi-Robot Generalization Through Domain Alignment
Our goal is to maximally reuse robotic datasets collected from one setup when deploying policies on another. Let D = {{(o, 90: 0n), (P1, 01, a1),..., (Pr. or, {r)}} be a dataset containing demon- strations of robot r completing task n. p E P, denotes the robot's proprioceptive end-effector pose, o E O, denotes the image observation, and a E A, denotes the action. We assume that there exists shared dataset D == un D of experience for N different tasks and different robots. Then, given a dataset for new task on some robotic platform DJ++1 1 1 S R, we would like to learn a policy T N+1 (alo), k A j that completes the same task on robot k.
In order to reduce the domain gap between robots, we align the observation space, action space, and internal representations of our policy. In an ideal world, this would allow to train joint Cotpolicy
multitask policy ((@ 2) on D U D where à E A is a shared action with similar interpre- tation across robots. Prior work [40] demonstrates that training cross-embodiment policies with a unified abstract action space can facilitate learning cross-robot features by coupling similar states with similar signals. However, discrepancies action interpretation across manipulators make this approach infeasible, so we instead train a task-conditioned multiheaded policy with R heads: n, (a) =), < R. E A, where f is a shared encoder and 2 is one-hot task encoding. Fig. and Fig. 6 in the Appendix depict this architecture.
3.1 Aligning the Observation Space
Ideally, an aligned observation space Ô would have the property that E DJ is similar to E DE 81 is similar to 82, where 81 and 82 denote the ground truth states of o₁ and 02. This corresponds to f(oz) ~ f(o2) if 81 ~ 52. To align this space as much as possible while maintaining generality of our robotic setup, we employ 3D-printed wrist camera mounts. The cameras are mounted directly in front of the robot and positioned to capture the end-effector within their field of view. However, because of variability across robots, the wrist camera mounts are not the same across robots, nor do we standardize the camera angle. Appendix A. shows the locations of the wrist and exterior cameras. Utilizing a wrist camera greatly simplifies the range of variation of camera positioning to two dimensions: the height of the camera with respect to the tip of the end-effector, and the angle of the camera. This is in contrast with exterior cameras, which require six dimensions to fully specify. In addition, wrist cameras provide natural robustness to distribution shifts in the environment [64]. One noticeable example of this is invariance visual differences in robotic arms, since only the end-effector is in view shown in the Observation Alignment section of Fig. 1.
3.2 Aligning the Action Space
To ensure consistent action interpretation across robots, we use shared upper-level environment. Fig. 1 depicts control stack under the Action Alignment section. For action 0, E Ar, use shared upper-level environment, responsible for processing a, and converting it into commanded
3

---

positive negative
Figure 2: Internal Representation Alignment. This figure depicts two trajectories across different robots for the same task. Our contrastive pretraining approach maps observations with similar pro- prioceptive state with respect to the grasped book and cabinet together. The green lines represent example pairs of observations mapped together, while the red line represents an example pair of observations whose embeddings are pushed apart.
delta pose target Apf. Then the sum of the robot's current pose and desired target pose, Pt + Apf. is transmitted to robot-specific controller. For modularization, each robot-specific controller Cr, 1 S r < R. exposes a small, shared API to set pose targets and query end-effector poses, which is called by the upper-level environment. addition, we utilize shared inverse kinematics solver, which processes these pose targets into joint commands. Since inverse kinematies is an underspecified problem, different solvers can cause large discrepancies in control. Our choice of shared solver minimizes inconsistencies in interpretation of pose commands by providing target pose-to-joint mapping. In addition, it aligns the coordinate frame of the robots' actions, by using a consistent definition of a robot's 's pose with respect to its base frame. Further details of our implementations of these controllers and inverse kinematics solvers are provided in Appendix A.4.
stne-standardiied
While the above design decisions successfully aligns the poses of the robot Pt E Pri and ensures consistent targets pt + Apī are transmitted each robot controller, should be noted that the result- ing achieved pose the next timestep Pc+1 is not necessarily aligned. This discrepancy arises from the fact that the trajectory end-effector follows to reach its target can differ based on the specific robot controller Cr, and the characteristics of the robot's hardware, even when employing shared inverse kinematics solver. For example, consider continuous or non-blocking controllers, which interrupt the robot with a new command at regular intervals At. Due to movement limitations imposed by the robot's kinematic configuration, the resulting trajectory will be highly nonlinear given the commanded pose pi and the current state. Consequently, even with standardized control stack, the displacement Apf proves insufficient for learning shared action space with consistent learning signal because its interpretation differs per robot.
One attempt to circumvent this issue would be to use a blocking controller, which waits for a robot to reach its target pose before issuing new command. Firstly, we would relabel the actions in our replay buffer as the change in achieved poses p and train a blocking controller to reach these poses. Since teleoperation with blocking controllers is difficult, we would instead use continuous control to collect the trajectory {Po. Ou, ag). (p1,01, 01), ..... (pr, or, ar)). We would then construct new dataset (po, Op: Apo), (Pn, On, Apn), (Phn, Okn, Aptn)) }. where Apk.. P(k++)n Pkn- However, we find that even with blocking controllers, Pt+1- Due to limited degrees of freedom of the hardware and inaccuracies with the controller, not all Apr+1 actions are easily reachable from the current state. As a result, one robot may have significant difficulty following trajectories from other robots for tasks that require more complex 6-DoF motion. Due to these challenges, we instead use separate heads म. (r((a t) to learn each robot's individual dynamics. By doing so, our policy benefits from shared visual features and overall movement directions, while enabling each head to learn the specific means to reach the desired goals.
+ Apf can have significant error from

---

3.3 Aligning Internal Representations
Aligning the internal representations of our policy can allow it to harness the advantages of training with a consistent signal, even in the absence of a unified action space. Our use of a shared kinematics solver provides a consistent proprioceptive signal P: with respect the robot's base. As a result, two (theoretical) trajectories with same exact same motion of the end effector, T1 E D T2 Dr, will r1 differ only by some translation (i.e. for all timesteps and Pr..t: E Ti, T2, (Pri.t - Pra.t) for some constant k). Even if we do not directly act this signal when rolling out our policy, can still exploit it to learn better features from our observations. With this in mind, we propose to pretrain aobot-agnostic features, then fine-tune to learn robot-specific controls.
In order to pretrain using the shared pose signal, we use a contrastive method that maps together similar states across trajectories and robots. We define the notion of "state similarity" by computing the changes in pose between a state and a predefined set of "fixed states" in the trajectory. Fixed states defined as states in trajectory that have a similar notion of task completion. For exam- ple, in quasi-static environments, these "fixed states" can be defined as states where subtask is completed, since all successful demonstrations must contain this state. More specifically, consider a trajectory T := {Po. oo. ao), (Pi, 0), (,,,),)..))) a.)..... (Pr, Or, ar)} and a set of fixed poses Py == (p'r), We define the difference between a trajectory state and fixed state by
d(p;, Pt,, - u 2. d(p;: Pt,) )uuat pl, (p, (pquaty-1
zyz is the proprioceptive Cartesian position of the state and p; quat is the orientation of the state. Since we may have more than one fixed state for trajectory, we additionally define a closest fixed state difference d(p:) = d(pr,Pty). where iy is the first timestep greater than or equal to i which corresponds to fixed states. After randomly sampling an anchor batch A, for each state, observation pair (p) E A, we uniformly sample corresponding positive and negative states (++ ((Pp)), (p-,0-) ~ U(Np.) by thresholding the distances between closest fixed state differences:
where P;
Pp. == [(p+, 0+); [ld(pa) ayz d(p+) )mpp) )o < 20
( Dy(((pp))uu ((p+))ous < nuat
(2)
Np. = op) [Id(pa) yayz v
d(p-) vay < e0 (d((p( guat ) quat
(3)
We define (p, q) cos (26p, g 1), the geodesic distance between the two quaternions p and q. We finally use triplet contrastive loss to pretrain policy: L(o., 04, o-) max(0, m + [lfe(oo) - )1 - (o.) - fo where fo is the shared encoder parameterized by O. These embeddings explicitly encourage mapping similar states together across trajectories and robots. This is achieved by sampling states from other rollouts in the positive buffer that correspond to similar poses with respect to fixed pose. Fig. 2 depicts example positive and negative pair across two trajectories from different robots.
By finetuning our policy using these embeddings, we can learn robot-specific elements built upon robot-agnostic characteristics. To accomplish this, we train individualized dynamics modules ,,, (a)( 1) for each robot using a multi-headed training approach. Each head corresponds to a distinct action space that may vary across robots due to disparities in action interpretation. This is depicted in Fig. 1 by arrows denoting separate actions per robot. By adopting this technique, we circumvent potential challenges associated with trajectory matching involving blocking controllers, facilitating transferability across more intricate tasks that necessitate greater degrees of freedom.
4 Experiment Setup
We aim to test the following questions: (1) Does leveraging datasets from other robotic platforms enable zero-shot transfer and increase sample efficiency when learning a new task? (2) Can aligning internal representations help bridge the domain gap across robotic platforms? (3) How does our choice of multiheaded training over robot-specific dynamics compare using a shared action space through a blocking controller? (4) How does our decision of utilizing wrist cameras compare the past approaches of collecting data with exterior cameras? Videos of experiments be found on our project website: https: // /sttesssssssee//ee//eee/eeee..ee www.teeggggl.....o.eeeeoooooo... www.wwww///////eeee. wtt.//eemmmimroootteeoooooo multorobot
5

---

Robotic platforms. We evaluate our method with the WidowX 250S, Franka Emika Panda, and Sawyer robot arms. All three of the rohots are controlled with 6-DoF delta joint position control. For each robot, we collect 64 by 64 image observations from two sources: an exterior camera pointing at the robot, and wrist camera mounted on the arm itself.
Evaluation tasks. We assume access shared buffer con- taining experience for at least one task on all three robots. Then, we collect data for a target task on two robots that we want to transfer to a third. For zero-shot evaluation, we directly train on this data and evaluate on the third robot. We also test of learning new task with few-shot experi- ments, where we teleoperate demonstrations for the new task on the third robot.
To effectively leverage data from prior tasks to solve new tasks, the prior tasks must share structural similarity with the new tasks. With this in mind, we evaluate the robot on variants for two types of tasks: standard pick/place, and shelf manip- ulation. In the pick/place task, the robot needs pick up marker and drop it into white container from variety of po- sitions. In the shelf manipulation task, the robot rearranges a book from a container onto a shelf. The first set of tasks was chosen to evaluate multi-rohot transfer in environments with simple dynamics and greater similarity in control from one robot to another. The second set evaluates transfer for en- vironments that require greater degrees of motion. Each of the tasks is evaluated on 10 starting locations for the objects.
Figure 3: Pick/Place Tasks. The left column contains the shared pick/place task, while the other columns contain the new distractor and new object vari- ants. For zero-shot evaluation, clude data for the shared task across all robots. For few-shot, we also in- clude demonstrations a variant for one robot.
in-
For each type of task, we test for both scene generalization and task generalization. We define scene generalization as generalization to a task for which there exists an mapping observations to a previously seen task. More formally, given an optimal policy л (alo) for previous task Ti, new task T2 only differs with Ti by scene there is function f T2. Task generalization is more general than scene generalization, applying to new tasks which similar to an old one, but may require different actions and controls to solve. The following are the variations that we evaluate on:
such that x (a).(o)) solves
Scenario 1 (SI): New Distractor Pick/Place: Pick/place with a distractor object in the background Scenario 2 (S2): New Object Pick/Place: Pick/place with a banana instead of a marker
Scenario 3 (S3): New Container Pick/Place: Pick/place putting a pen into a cup. This requires rotation motion that is not seen in the shared dataset.
Scenario 4 (S4): New Orientation Shelf: Shelf Manipulation with reversed orientation of the orig- inal container that contains the books. The motion to grasp and place the book is the same. Scenario 5 (S5): New Compartment Shelf: Shelf Manipulation putting a book into a new compart- ment on the bottom
Comparison methods. For few-shot transfer, we compare our main method (denoted are Con- trastive + Multiheaded or Polybot two different baselines. The first is Naive Multi-Robol Training, which trains task-conditioned multiheaded policy on exterior camera angles without contrastive pretraining. We also evaluate the Single Robot Training baseline, which only contains demonstra- tions for the target robot. We also consider two different ablations for our method: task-conditioned multiheaded training without constrastive pretraining denoted as Ours w/o Contr, and constrastive pretraining with blocking controller denoted as Contr. + Blocking.
Dataset collection. Details are provided in Appendix A.2.
5 Experimental Results

---

Robol
Method
New Distr. (81)
New Obj. (S2) New Cont. (S3) Pick and Place
New Orient. (S4) New Comp. (85)
Shelf
Pulybot
0.9
0.8
0.9
1.0
0.9
Pranka
Naive Multi-Robol
0,4
0.3
0.0
0.0
Single Robot
0.2
0.2
0.0
0.0
0.0
Polybot
0.9
0.9
0.9
0.7
Sawyer
Naive Multi-Robor
0.3
0.2
0.2
0.0
0.0
Single Robot
0.2
0.1
0.0
0.0
0.0
Polybot
0.9
1.0
0.7
0.8
0.7
WidowX
Naive Multi-Robot
0.4
0.1
0.2
0.0
0.0
Single Robot
0.3
0.2
0.0
0.0
0.0
Table 1: Few-shot multi-robot transfer results. Given 5 demonstrations on a new task, Polybot performs significantly better than baseline without data from other robots. In addition, our task- conditioned multiheaded policy enables the transfer of multi-robot data for shelf manipulation tasks, where a blocking controller fails.
Robot
Method
S1 S2
S3
S4 S5
Utilizing data from other robots signfi- cantly improves few-shot generalization per- formance on a new task. In Table I, the suc- cess rate for Polybot demonstrates significant improvement on all tasks and all robots over sin- gle robot training. On the Franka, the results show an average of 0.56 higher success rate on Pick/Place (0.9, 0.8, 1.0 versus 0.4, 0.3, 0.3) and 0.95 higher success rate on Shelf Manipulation. (1.0, 0.9 versus 0.0, 0.0). This indicates that Polybot can effectively utilize data from other robots to learn new task with high sample efficiency. Notably, single-robot training fails to get any success Shelf Manipulation. Qualitatively, observe that this policy quickly falls out-of- distribution and in unable grasp the book. For tasks with high structural similarity to those in the shared buffer such as New Distractor Pick/Place, training without other robot data has nonzero performance, likely due to the variation in scenes we collect in our shared dataset. However, demonstrations are not enough to cover the entire distribution object positions for a new task, leading to suboptimal performance.
Polybot
0.4 0.6
0.0
0.4 0.0
Franka
Contr. Blucking
0.3 0.3
0.0
0.0
0.0
Polybot
0.6 0.5 0.0
0.4 0.0
Sawyer
Contr. + Blocking 0.6 0.3 0.0
0.0 0.0
Polybot 0.4 0.4 0.0 0.5 0.0
WidowX
Contr. Blocking 0.4 0.3 0.0 0.0 0.0
Table Zero-shot results. Polybot can learn new task with high structural similarity to tasks in shared multi-robot buffer given only data from other robots.
Polybot facilitates better transfer than naive task-conditioned multiheaded training with exterior cameras. Table shows significantly higher success rates for Polybot over naive multi- robot training (0.9, 0.8, 0.9, 1.0, 1.0 versus 0.4, 0.3, 0.3, 0.0, 0.0 for the Franka). This suggests that task-conditioned multiheaded training on exterior camera struggles to utilize data from other robots, validating our hypothesis that standardized observation space is crucial for this transfer to occur. For Shelf Manipulation scenarios, naive multi-robot training fails to achieve any success, For Pick/- Place scenarios, this method does have some success, but it is likely due to the similarities between the new task and other tasks from the same robot.
Utilizing data from other robots allows for zero-shot generalization performance on a new task. For tasks that have higher structural similarity to that scene in previous data for the robot such as scene generalization tasks described in the experiment setup section, we see good success rate for zero-shot multi- robot transfer. Table 2 showcases this, with both Polybot having success rates of 0.4, 0.6, 0.4 and the blocking controller having success rates of 0.3, 0.6, 0.4 for the New Distractor Pick/Place task. Without data from other robots, the success rate would be near because the new task would not be present in the replay buffer. For Scenario 4, or New Orientation Shelf, Polybot has nonzero performance while the blocking method doesn't. This is because while both methods are able to pick up the book from the reversed bookshelf, the blocking controller struggles with placing the book onto the shelf.
Aligning the internal representations of the observations between robots leads to learning more generalizable features. Table 3 compares the performances of Polybot with an ablation of the
7

---

Robot
Method
S1
S2
S3
S4
S5
Method
S1
S2
S3
S4
S5
Polybot
0.9
0.8
0.9
1.0
0.9
Polybot
0.9
0.8
0.9
1.0
0.9
Franka
Polyhot w/o Contr.
0.8
0.5
0.7
0.6
0.7
Contr. Blocking
0.8
0.6
0.0
0.0
0.0
Polybot
0.9
0.7
0.7
Polybot
0.9
0.9
0.7
0.9
0.7
Sawyer
Polybot w/o Contr.
0.7
0.5
0.4
0.6
0.7
Contr. Blocking
0.9
0.9
0.0
0.0
0.0
Polyhot
0.9
1.0
0.7
0.8
0.7
Polybot
0.9
1.0
0.7
0.8
0.7
WidowX
Polybot w/o Contr.
0.8
0.7
0.7
0.6
0.7
Contr. Blocking
0.8
0.9
0.0
0.0
0.0
ours vs ours without contrastive. Our contrastive pretraining and multiheaded finetuning approach provides an average of 19%0 improvement on few-shot transfer for a new task over regular mul- tiheaded training.
Table 3: Ablation:
Table 4: Ablation: Ours vs contrastive + blocking Although blocking controller has similar few-shot performance to Poly- bot simple Pick/Place variants, it strug- gles with tasks that require 6-DoF motion.
contrastive pretraining phase. Aligning the internal representation of the policy causes an average of 19% increase in performance over a baseline without contrastive pretraining. This suggests that aligning the internal representation of the policy assists in learning features that generalize across robots. We hypothesize that training on an aligned proprioceptive signal across robots. Notably, multiheaded training alone seems have reasonable performance on all tasks, albict lower than our method.
Multiheaded policies can better learn from 6-DoF, cross-robot demonstrations over blocking controllers. Table 9 shows that on the Shelf Manipulation tasks, Polybot achieves high success rates 1.0, 0.9 on the Franka compared to zero success rate on the blocking controller. In addition, in the New Container Pick/Place, Polybot has a 0.9 success rate over the 0.0 with blocking. This large discrepancy is explained by the inability of one robot to precisely imitate another robot. For instance, variations in the length of the wrist link can lead to large discrepancies in the radius of the rotation necessary to perform the Shelf Manipulation task. To verify this, we provide the error over timestep of imitating each task in the Appendix. Notably, the blocking controller has high performance on SI and S2 due to lower discrepancy between robots for translational movement.
6 Discussion
Summary. We have developed method, Polybot, that efficiently learns new tasks using data col- lected on other robots. This is enabled through careful design decisions driven by fundamental observation: transferring data across domains requires aligning the domains as much as possible without making assumptions that limit their applicability. For example, Polybot uses wrist cam- eras, which can be mounted on a wide range of robots, while exhibiting significantly less variation than exterior cameras, even if the mounting position is not fixed. In addition, Polybot uses shared higher-level action representation and varying lower-level controller, which can align the policy's actions while accommodating the diversity of robotic setups. Finally, our choice of contrastive loss can align internal representations across robots. Despite the simplicity of each design decision, their combination sufficiently aligns our multi-embodiment dataset to enable consddmebod transfer. Polybot achieves over a 70% success rate on all tasks, outperforming naive multi-robot training.
Limitations and Future Work. One limitation of our approach is that our method requires a shared dataset between robots learn their correspondences. As a result, our method not able transfer policies to new robot with no demonstrations. Our method also does not allow for zero-shot transfer on tasks with different motion than seen in the shared dataset. In addition, the scope of our evaluation has been on parallel-jaw robotic manipulators. Generalization can become more difficult with a more diverse set of end-effectors, although our method does not preclude this type of transfer. Our use of egocentric cameras may lead to difficulties in partially observable settings, such as settings without clear view of the end-effector. Finally, our representation alignment relies on scalable method to compute fixed states across all trajectories, which may not apply for all manipulation tasks. In the future, we plan to scale our datasets, we believe that can allow for more new tasks, camera-viewppoinss and end-effectors be in-distribution.
8

---

Acknowledgments
We thank Tony Zhao, Moojin Kim, and Alexander Khazatsky for the numerous discussions about real-world robot learning and Kyle Hsu, Hengyuan Yu, and Suvir Mirchandani for their helpful feedback. This research was supported by the Office of Naval Research grants N00014-22-1-2621 and N00014-22-1-2293.
References
[1] Deng, W. Dong, R. Socher, L.-J. K. Li, and L. Fei-Fei. Imagenet: large-scale hierarchi- cal image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255 2009.
[2] A. Radford, K. Narasimhan, T. Salimans, and Sutskever. Improving language understanding
by generative pre-training. 2018. 8.
[3] S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and
C. Finn. Robonet: Large-scale multi-robot learning. In Conference on Robot Learning, 2019.
[4] C. Devin, A. Gupta, T. Darrell, P. Abbeel, and S. Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In International Conference on Robotics and Automation, 2016.
[5] T. Chen, A. Murali, and A. Gupta. Hardware conditioned policies for multi-robot transfer learning, 2019.
[6] E. S. Hu, K. Huang, O. Rybkin, and D. Jayaraman. Know thyself: Transferable visual control policies through robot-awareness. In International Conference on Learning Representations, 2022.
[7] G. Salhotra, xl Chun Arthur Liu, and G. Sukhatme. Bridging action space mismatch in learning from demonstrations. 2023.
[8] P. Sharma, L. Mohan, L. Pinto, and A. Gupta. Multiple interactions made easy (mime): Large scale demonstrations data for imitation, 2018.
[9] A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay. S. Savarese, and L. Fei-Fei. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. 2018.
[10] S. Young, D. Gandhi, S. Tulsiani, A. Gupta, P. Abbeel, and L. Pinto. Visual imitation made easy. 2020.
[11] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learn- ing, 2021.
[12] Ebert, Y. Yang. K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. In Robotics: Science and Systems, 2022.
[13] Yu, D. Quillen, Z. He, R. Julian, A. Narayan, H. Shively, A. Bellathur, K. Hausman, Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforce- ment learning. In Conference on Robot Learning. 2019.
[14] S. James, Z. Ma, D. R. Arrojo, and A. Davison. Rlbench: The robot learning benchmark & learning environment. 2019.
[15] Y. Zhu, J. Wong, A. Mandlekar, Martin-Martin, A. Joshi, S. Nasiriany, and Zhu. robo- suite: A modular simulation framework and benchmark for robot learning. In arxiv preprint arXiv:2009.12299 2020.

---

[16] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. 2021.
[17] Sermanet, C. Lynch, Y. Chebotar, Hsu, E. Jang, S. Schaal, Levine, and Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 1334-114,, 2018. doi:10.1 109/1O. 2018.8462891.
[18] K. Schmeckpeper, O. Rybkin, K. Damiilidis, S. Levine, and C. Finn. Reinforcement learning with videos: Combining offline observations with interaction, 2021.
[19] Radosavovic, Xiao, S. James, Abbeel, Malik, and Darrell. Real-world robot learning with masked visual pre-training. 2022.
[20] K. Grauman et al. Ego4d: Around the world in 3,000 hours of egocentric video, 2022.
[21] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation. In Conference on Robot Learning. 2022.
[22] Y. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. In International Conference Learning Representutions, 2023.
[23] A. Majumdar, K. Yadav, S. Arnaud, Y. J. Ma, C. Chen, S. Silwal, A. Jain, V.P.. Berges, Abbecl, J. Malik, D. Batra, Y. Lin, O. Maksymets, A. Rajeswaran, and F. Meier. Where are we in the search for an artificial visual cortex for embodied intelligence?, 2023.
[24] S. Karamcheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang. Language- driven representation learning for roboties. 2023.
[25] A. S. Chen, S. Nair, and C. Finn. Learning generalizable robotic reward functions from "in- the-wild" human videos. 2021.
[26] Bahl, A. Gupta, and D. Pathak. Human-to-robot imitation in the wild. 2022.
[27] I.. Shao, T. Migimatsu, Q. Zhang, K. Yang, and J. Bohg. Concept2robot: Learning manipula-
tion concepts from instructions and human demonstrations. 2020.
[28] R. Shah and V. Kumar. Rrl: Resnet as representation for reinforcement learning. 2021.
[29] L. Yen-Chen, A. Zeng, S. Song, Isola, and T.-Y. Lin. Learning to see before learning act: Visual pre-traine pre-training for manipulation. 2021.
[30] M. Sharma, C. Fantacci, Y. Zhou, S. Koppula, N. Heess, J. Scholz, and Y. Aytar. Lossless adaptation of pretrained vision models for robotic manipulation. 2023.
[31] C. Wang, X. Luo, K. Ross, and D. Li. Vr13: A data-driven framework for visual deep rein- forcement learning. 2023.
[32] M. Chang, A. Gupta, and S. Gupta. Semantic visual navigation by watching youtube videos. 2020.
[33] S. Reed et al. A generalist agent. Transactions on Machine Learning Research, 2022.
[34] H. You, T. Yang. Y. Zheng, J. Hao, and Taylor, Matthew. Cross-domain adaptive transfer reinforcement
learning based state-action correspondence. In Cussens and K. Zhang, editors, Proceed- ings of the Thirty- Eighth Conference Uncertainty in Artificial Intelligence, volume 180 of Proceedings of Machine Learning Research, pages 2299 2009, PMLR, 01-05 Aug 2022.
10

---

[35] P. Christiano, Z. Shah, I. Mordatch, J. Schneider, T. Blackwell, J. Tobin, P. Abbeel, and W. Zaremba. Transfer from simulation to real world through learning deep inverse dynam- ics model. In CoRR, 2016.
[36] W. Yu, J. Tan, C. K. Liu, and G. Turk. Preparing for the unknown: Learning universal policy with online system identification. In CoRR, 2017.
[37] X. B. Peng, M. Andrychowicz. W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic con- trol with dynamics randomization. In International Conference on Robotics and Automation. IEEE, 2018.
[38] Q. Zhang, T. Xiao, A. Efros, L.. Pinto, and Wang. Learning cross-domain correspondence for control with dynamics cycle-consistency, 2021.
[39] F. Sadeghi, A. Toshev, E. Jang, and S. Levine. Sim2real view invariant visual servoing by recurrent control. In International Conference on Robotics and Automation, 2017.
[40] D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine. Gnm: A general navigation model to drive any robot. In International Conference Robotics and Automation, 2023.
[41] K. Bousmalis, G. Vezzani, Rao, C. Devin, A. X. Lee, M. Bauza, T. Davehev, Y. Zhou, A. Gupta, Raju, A. Laurens, C. Fantacci, Dalibard, M. Zambelli, M. Martins, R. Pevce- viciute, M. Blokzijl, M. Denil, N. Batchelor, T. Lampe, E. Parisotto, K. Zolna, S. Reed, S. G. Colmenarejo, J. Scholz, A. Abdolmaleki, O. Groth, J.-B. Regli, O. Sushkov, T. Rothörl, J. E. Chen, Y. Aytar, D. Barker, J. Ortiz, M. Riedmiller, J. T. Springenberg, R. Hadsell, F. Nori, and N. Heess. Robocat: A self-improving foundation agent for robotic manipulation, 2023.
[42] E. Parisotto, J. L. Ba, and R. Salakhutdinov. Actor-mimic: Deep multitask and transfer rein- forcement learning, 2016.
[43] Y. Duan, M. Andrychowizz, C. Stadie, Ho, Schneider, Sutskever, P. Abbeel, and W. Zaremba. One-shot imitation learning, 2017.
[44] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine. One-shot visual imitation learning via meta-learning, 2017.
[45] K. Hausman, J. T. Springenherg, Z. Wang, N. Heess, and M. Riedmiller. Learning embed- ding space for transferable robot skills. In International Conference Learning Representa- tions, 2019,
[46] Z. Xu, K. Wu, Che, Tang, and Ye. Knowledge transfer in multi-task deep reinforcement learning for continuous control, 2020.
[47] S. Sodhani, A. Zhang, and J. Pineau. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning, 2021.
[48] A. Brohan et al. Rt-1: Robotics transformer for real-world control at scale, 2022.
[49] M. Ahn al. Do as can, not as say: Grounding language in robotic affordances. 2022.
[50] J. Tobin, R. Fong, A. Ray. J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. 2017.
[51] Z.-W. Hong, C. Yu-Ming, S.-Y. Y. Su, T.-Y. Shann, Y.-H. Chang, H.-K. Yang, B. H.-L. C.-C. Tu, Y.C. Chang, T.-C. Hsiao, H.-W. Hsiao, S.PP. Lai, and C.-Y. Lee. Virtual-to-real: Learning to control in visual semantic segmentation. In International Joint Conferences on Artificial Intelligence, 2018
[52] A. A. Rusu, M. Vecerik, T. Rothörl, N. Heess, R. Pascanu, and R. Hadsell. Sim-to-real robot learning from pixels with progressive nets, 2018.

---

[53] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. Ratifff, and D. Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Robotics and Automation (ICRA), pages 8973-8979, 2019. doi: 21100//CAA2219.8877555
[54] M. Kaspar, J. D. M. Osorio, and J. Bock. Sim2real transfer for reinforcement learning without dynamics randomization, 2020.
[55] D. Ho, K. Rao, Z. Xu, Jang, M. Khansari, and Y. Bai. Retinagan: An object-aware approach to sim-to-real transfer, 2021.
[56] K. Bousmalis, A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L. Downs, J. Ibarz, Pastor, K. Konolige, Levine, and V. Vanhoucke. Using simulation and domain adaptation improve efficiency of deep robotic grasping. 2017.
[57] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. 2017 IEEE International Conference on Computer Vision (ICCV), pages 2242-2251, 2017. 1t1110.110//CC22211.222.2
[58] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and
V. Lempitsky. Domain-adversarial training of neural networks, 2016.
[59] K. Fang, Y. Bai. S. Hinterstoisser, S. Savarese, and M. Kalakrishnan. Multi-task domain adaptation for deep learning of instance grasping from simulation. In 2018 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 3516-3523, 2018. doi: 2018.46111....
[60] M. Xu, M. Islam, C. M. Lim, and H. Ren. Learning domain adaptation with model calibration for surgical report generation in robotic surgery. In International Conference on Robotics and Automation, 2021.
[61] A. Gupta, C. Devin, Y. Liu, Abbeel, and S. Levine. Learning invariant feature spaces to transfer skills with reinforcement learning, 2017.
[62] N. H. Kim, Z. Xie, and M. van de Panne. Learning to correspond dynamical systems, 2020.
[63] S. J. Wang and A. M. Johnson. Domain adaptation using system invariant dynamics models. In S. J. Wang and A. M. Johnson, editors, Proceedings of the 3rd Conference on Learning for Dynamics and Control, volume 144 of Proceedings of Machine Learning Research, pages 1130-1141. PMLR, 07 08 June 2021. URL ptoopedg/rooceeiingg mlr, press/v14A/ wang2ic. html.
[64] K. Hsu, M. J. Kim, R. Rafailov, J. Wu, and C. Finn. Vision-based manipulators need to also see from their hands, 2022.
12

---

A Appendix
Further details and videos of experiments can be found on our project website: https://sice google. coom/vier/polybot-moeee
A.1
Robotic Setup
Wrist Cameras
Exterior Cameras
Figure 4: Our robotic setups. For each robot, we collect data with both wrist camera and exterior camera. The cameras are Logitech C920s and Zeds. Although these cameras do have slight differ- ences brightness and contrast, this does not seem to affect results.
A.2 Dataset Collection
We collect two types of datasets: a shared dataset containing data from similar tasks for all three robots and target dataset containing a new task from one robot platform which we want to transfer to other platforms. For the purposes of evaluation, our shared dataset consists of the original tasks we defined above. For each variant, we collect data on diverse scenes and backgrounds to ensure that the resulting policies have some degree of robustness to changes in the environment. In order to provide diversity visual observations, in the background. By collecting our datasets for each task over 3 variations of this scene, ensure that our policy is robust to changes in lighting conditions. Overall, our dataset contains 6 tasks over 3 robots with 3 different backgrounds per task and 50 demonstrations per (scene, robot, background) combination collected over the course 60 hours.
cups, plates, and wallpapers with intricate patterns
A.3 Higher-Level Environment Details
Our higher-level environment has of shared image server and action processor between robots. We use delta position control as our action space. This parameterized by 7-dimensional vector consisting of translational dimensions, 3 rotational dimensions, and dimension indicate the per- centage to close the parallel end-effector. The code for processing an action before sending the lower-level controller is shown below:
def step self, action) ): start_time time time
# Process Action assert len (action) = (self DoF + 1)
assert (action max () <= 1) and (action min () >= -1)
13

---

pos_action, angle_action action gripper = self _foo__tt_mt____ cermat_action (action)
lin_vel rot_vel = self _it_____m______ velocity (pos_action angle_ action) desired_pos = self _elr._curr_pos + vin_
desired_angle = [ud_angles(roo_velll
self utt_____dd____bbbteee desirre__pos. desired_angle. gripper)
comp_time time time () start_
sleep_ left max (0, (1 self hz) somp_time) time. sleep ( sleep_left)
Given delta position, angle, and gripper command, our environment first normalized and clips the commands to ensure that large actions are not sent to the robot. Then, we add the delta position to our current pose and the delta angle to our current angle. We pass the position and angle into our lower-level robot controller.
A.4 Robot-Specific Controller Details
Each robot-specific controller provides the following API the higher-level environment:
def update_poss (pos, angle):
def update_joints (joints)
def update_griperrcbooe_prrccctt
def get_joint_position ():
def cet_jjointvvlloiiee ():
def get get_grrpper_state ():
def get get_ee_posse ():
pose, update joints and update For each lower-level controller, update_poss pose and convert them into joint targets. For simplicity, URDF model specifications for each of our robots compute target joint positions from Cartesian poses. When computing inverse kinematics through Pybullet, we manually set the joint limits of each robot, since the solver does not automatically consider these limits by itself. We also use the IK solver to give us joint positions, joint velocities, gripper states, and end-effector poses. This allows us to use a standarized coordinate from with respect to the robot's base to get robot's Cartesian coordinates.
The functions update pose,
opdate_gripper gripper set targets for moving the robot. shared inverse kinematics solver to take the target poses in Pybullet-based solver and
For each robot, we implement two controllers: blocking version and nonblocking ver- sion. The blocking controller waits for entire movement command to finish before executing the next command. Meanwhile, the nonblocking or continuous controller continuously interrupts the robot with a new target pose every fixed period of time.
A.5
Network Architecture
A.6
Contrastive Learning Details
We train our encoder with triplet loss of margin m 0.5.
L(Oa, 0+, o-) = max(0, m + (o.) - fo(o+))l IIfo(oo) - fo(o-))13
We provide nearest neighbor lookup for our robot below. We first embed the left image via our encoder. Then, embed all observations in a dataset for different robot. For example, in the top-left image, we use the shelf manipulation dataset with only Franka data. Then, compute the
14

---

3x3
DorT Rolu
sarry
LOOTT
pool
Rolu
pocl
Relu
fatten
fc Z
(16. 16, 16)
(16, 18, 18)
RGB Image
(512)
(1024)
(64. 64, 3)
[32, 16)
(32. 16)
(64, 84, 16)
Figure 5: Our encoder architecture. We parameterize encoder CNN. The convolutional layers are flattened and then fed into two MLP layers to get representation 2. In order to learn correspondence between robots, train this encoder with contrastive loss. We use random crop and color jitter as image augmentations for our encoder.
10
Robot 1 Action
(256)
(256)
Z
fc
fc
fc
fc
+
Robot Action
(256)
(256)
(512)
1-hot task ID
Robot Action
(256)
(256)
Figure 6: Our decoder architecture. The output of our encoder z is concatenated with a one-hot task index and fed into the decoder. This task index specifies task either in the shared buffer, or a new task which we want to achieve. After passing the input through two MLP layers, we feed in into three-robot specific heads for each of the robots are evaluating on.
Attribute
Value
Input Width
64
Input Height
Input Channels
3
Kernel Sizes
[3, 3,3]
Number of Channels
[16, 16, 16]
Strides
[1, 1, 1]
Paddings
[1, 1,
Pool Type
Max 2D
Pool Sizes
[2, 2, 1]
Pool Strides
[2, 2. 1]
Pool Paddings
[0, O.
Image Augmentation
Random Crops/Color Jitter
Image Augmentation Padding
Table 5: CNN hyperparameters for our policy encoder. Our CNN uses 64 by 64 images, which passes through through 3 convolutional layers. Each layer has 3by3 kernel with 16 channels. We augment our architecture with random crop and color jitter.
embedding with the closest 12 distance from the embedding of the left image. Note that our method also aligns trajectories with same robot.
15

---

Hyperparameter
Value
Hyperparameter
Value
Batch Size
64
Batch Size
64
Number of Gradient Updates Per Epoch
1000
Number of Gradient Updates Per Epoch
1000
Learning Rate
3E-4
Learning Rate
1E-4
Optimizer
Adam
Optimizer
Adam
Table 6: Hyperparameters. The left table contains hyperparameters for behavior cloning, and the right table contains hyperparameters for contrastive learning.
Contrastive Nearest Neighbors. This figure shows nearest neighbors examples across the three robots for embeddings from our pretrained encoder. These examples are computed for both shelf and pick/place trajectories.
Figure 7:
A.7 Shelf Tasks
Figure 8: Shelf Tasks. The original shelf tasks consists of placing the book on the top compartment. The first target task requires doing the same from reversed book container, while the second tasks requires placing the book in the lower compartment. The tasks in the first column are part of the shared dataset while the second and third are target tasks test transfer.
16

---

A.8 Error between Commanded Delta Pose Target and Achieved Delta Pose
The following figures depict plot of the 12 norm between the translational components of the delta commanded pose targets and achieved delta poses for demonstration trajectories across 3 robots. At each timestep, the environment receives a delta commanded pose target, which gets added to the robot's current pose then sent to the lower-level controller. Although the controller defines trajectory to reach this target pose, due to errors the inverse kinematics solver and limitations on movement imposed by the hardware, it may not reach the pose. We plot the error for each timestep across trajectory from Pick/Place task and one from Shelf Manipulation task. Expectedly, the WidowX has the highest average error, followed by the Sawyer then the Franka. This error varies wildly between robots and timesteps. causing the commanded delta pose to be highly unpredictable from the achieved delta pose.
Commended end Achieved Delte Pose Error
Commandee rded Achiavad
ItMMaace
0.0175
Pick/lace Monipulazan
1aff Marip.iation
0.235
0.330
0.0125
0.0100
1200
0.015
0.0030
0.11
0.0025
25
75
200 Timestop
322
230
275
200
10
T1m000000
Figure 9: Action interpretation error for the Franka.
Figure 10: Action interpretation error for the Sawyer
Commanded and Achieved Delta Pose Error
Mym A
0.57
0.08
0.50
0.52
Rcuffaoe
sheff momputtioon
Timestop
Figure 11: Action interpretation error for the WidowX.
17

---

A.9 Ablation: Wrist Camera Variation
In order to more comprehensively evaluate the effect of wrist-camera variation, we have 3D-printed 2 new wrist-camera mounts for the WidowX250S. We have then collected data for the shared Pick- /Place task as well New Distractor Pick/Place task variant. Here are the results
Viewpoint 1 (V1): Original Mount: The original camera mount used the WidowX 250S for experiments. The mount is 20 degrees from vertical.
Viewpoint (V2): Original Mount + Masked Gripper: The original camera mount used on the WidowX 250S experiments with the bottom part of the image masked out.
Viewpoint (V3): 35 Degree Mount A wrist-camera viewpoint that is 35 degres from the vertical.
Viewpoint (V4): 50 Degree Mount A waist-camera viewpoint that is 50 degres from the vertical.
Viewpoint (V5): 50 Degree Mount + Change Height: The 50-degree wrist-camera mount is placed lower the WidowX.
Figure 12: Our wrist camera mounts. The follow depicts the original, 35-degree and 50-degree mounts.
Original Mount
Original Mount Mask
35 Degree Mount
Degree Mount
Degree Mount Height Change
Figure 13: Our wrist camera viewpoints. This figure depicts the viewpoints we used for our ablations. The lego block is in the same place directly under the gripper for all mounts. Different wrist camera angles can cause the same object to appear in different parts of an egocentric image.
Method
V1
V2
V3
V4
V5
CRADLE
0.8
0.8
0.9
0.8
0.7
Table 7: Ablation: Wrist Camera Viewpoint The table depicts the few-shot performances of the new-distract Pick/Place task with new distractor viewpoints. Each variation was evaluated 10 times on wide variety of angles.
Our results show consistently high success rate on target task for the variation in viewpoint evaluated Notably, though the location of the object the image changes, CRADLE is still able to learn good correspondences and transfer experience to these camera angles. We expect
18

---

that by training on a larger shared dataset with more variations in camera angle, we will be able to finetune directly on new camera viewpoints without requiring shared data.
Interestingly, the policy performs well on the Pick/Place even with the bottom part of the image cropped out. In order make these experiments work, had to include the robot's proprioceptive information alongside the 1-hot task ID and latent variable to the decoder. This allows the state to maintain full observability the Pick/Place tasks. With more than 1 viewpoint that does not contain the gripper, the policy may have problems determining the camera angle from the image alone. This is because there one can achieve the same image by either changing the camera angle or moving the robot's s end-effecctr.
A.10
Ablation: End-Effector Size Variation
In order to consider the effect of variations end-effector size, we have ran experiments on Pick- /Place tasks with a larger and smaller gripper. Although we reuse data with the larger gripper, we collect data both the shared Pick/Place task as well small number of demonstrations for each task variant using the smaller gripper. Similar to our main experiments, with 5 demonstrations for task variant. In order to succeed, a robot has to slightly adapt its actions in order to transfer robotic data collected from the larger gripper to the smaller. If incorrect features are transferred across these two settings, the policy may attempt to grasp the object too early.
finetune policy
Larger Gripper
Smaller Gripper
Figure 14: Egocentric Viewpoints for the Larger/Smaller Grippers. The fingers on the left are a larger 3D-printed variations of the smaller designed by Trossen Robotics the right.
Method
S1
S2
CRADLE
0.9
1.0
0.7
Table 8: Ablation: End-Effector Size Variation: CRADLE achieved an average of 87% success on few-shot generalization new gripper.
Our experiments show that CRADLE has high success rate on Pick/Place tasks with both small and large grippers. These results in-line with the results see from transfer across the Franka's Sawyer's and WidowX's 's end effectors. Since the grippers different sizes and widths, the pol- icy will need to learn how to adapt their individual constraints in order to get effective transfer performance.
A.11 Ablation: Joint Egocentric and Exterior Training
Although our settings do not require partial observability, to provide evidence that CRADLE is able to transfer information with joint egocentric and exterior training, we have runs New Distractor Pick- /Place experiments with both joint and exterior camera training. In order to process this information, we stack these viewpoints together channel-wir before passing through the convolutional neural network encoder. The following table describes the results:
19

---

Method
S1
S2
S3
CRADLE 0.7
0.7
0.6
Table 9: Ablation: Joint Egocentric and Exterior Training The table depicts the performances of the new-distract Pick/Place task with new distractor viewpoints. Each variation was evaluated 10 times on a wide variety of angles.
Although CRADLE is able to achieves >> 60 success rate with both egocentric and exterior camera, we see that its performance suffers. We believe that this is due to difficulties aligning the third- person observations. Although wrist-camera perspectives look similar across embodiments, third- person perspectives can have vary based the robot's appearance. We believe that by pretraining with more
20


Reviews:
[{"review_id": "u_PHTj17BT", "reviewer": "Reviewer_LHXF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The main idea of the paper is to address the challenge of transferring manipulation skills among various robotic platforms. The authors propose a novel framework to scale vision-based robotic manipulators by reusing large datasets. Their strategy involves aligning observation and action spaces across different robots using wrist cameras and a unified, modular codebase. They also bridge the domain shift by aligning the policy's internal representations, leveraging contrastive learning. The paper contributes a significant method to improve the success rate and sample efficiency for a policy using new task data from different robots, as validated by testing on a large dataset involving multiple tasks and robots.", "summary_of_recommendation": "The reviewer believe that this paper is above the bar of acceptance. Overall the technical method can still be improved so a weak acceptance is my recommendation before rebuttal.\n\n## After rebuttal\nThe rebuttal well address the concerns of my original question. The reviewer believes that this paper can be accepted.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Could the author offer a more logical method for aligning trajectories across different robots? No need for experimental data, a detailed description would suffice.\n\nCould the author elaborate on the degree of variation allowed for wrist-mounted poses? How well does this method generalize across diverse viewports? Is it feasible to complete the task even if the camera doesn't have a clear view of the gripper?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper delves into a critical issue in robot learning: utilizing data across various robotic morphologies for efficient training related to manipulation tasks. The reviewer acknowledges the considerable complexities and challenges involved in this project and commends the authors' determination to address them. The paper's strengths are manifold. \n- Firstly, it introduces a robust method to synchronize the observation and action spaces among different robotic entities. While employing cartesian control has proven to be more invariant to kinematics than joint control, incorporating this concept into a learning framework is an outstanding engineering accomplishment. \n- Secondly, the paper suggests a contrastive learning strategy to enhance the shared visual representation learning among various robots. This approach is quite intuitive given the design of robot-specific dynamics. The perceptual task involves understanding the spatial relationship between the object and the gripper, which can be common across morphologies, while action prediction is dependent on the individual controller characteristics of each hardware piece.\n\nHowever, the paper could benefit from enhancements in a couple of areas: \n1. The alignment method in contrastive learning could be improved to better define positive and negative examples. The current method is heavily reliant on the end-effector pose, a strong assumption for manipulation tasks. More consideration should be given to object information. \n2. More comprehensive evaluations with a wider range of wrist-mounted camera poses. Questions like whether the camera needs to have a clear view of the gripper, and what the implications would be if not, need addressing.\n\nTypo:\nLine 99 the the image observation", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "t7mMJVWcfh", "reviewer": "Reviewer_Fa2w", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes CRADLE - a framework for cross-embodiment adaptation and alignment. CRADLE is an imitation learning framework that tries to learn a skill of a chosen robot from different robot demonstrations. To align the observation space, the camera is mounted at the end-effector of each robot. It collects a dataset of demonstrations for a set of tasks from a set of robots, then trains a neural network policy with $R$ heads corresponding to $R$ robots to accommodate their different dynamics with shared visual presentation. During the training phase, to align the state latent representation, CRADLE adopts the contrastive learning method with the state distance defined on task space. The method’s efficacy is shown in real-world pick&place tasks between multiple robots.", "summary_of_recommendation": "Overall, due to:\n- incomplete paper discussions\n- computation drawback that hinders this method's significance\n- lack of clarity in some sections\n\nI recommend rejection but I can change my assessment based on the authors' rebuttal.\n\n------ Post Rebuttal ------\n\nThe authors addressed all my concerns. I decided to raise the score to Weak Accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1. Discussion on the possible further applications of CRADLE to different cross-embodiment imitation learning settings.\n2. How to efficiently implement the positive and negative sampling for contrastive learning in this setting?\n3. Further elaborations for Section 3.2 is desirable.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "++: Strength, --: Weakness\n\n**Novelty & Significance**\n\n++ Cross-embodiment imitation learning is a hard problem that is still an open question. I like the idea of CRADLE such that different engineering tricks can realize alignments in observation, action, and state representation.\n\n-- However, I think that this idea has its own downside. Essentially, CRADLE bypasses the correspondence problem between different embodiments by representing the state on task space, egocentric viewpoint, and custom controller that requires known robot models. I believe this limits the framework to object-centric manipulations having chain-like robot arms with a single end-effector. Scaling up to human - robot imitation or single-arm robot - bi-arm robot imitation is impractical.\n\n-- Furthermore, embodiment-specific mapping is not addressed. For example, consider the imitation learning task of navigating through a hole, robot demonstration moving through a narrow hole will be hard to transfer with CRADLE’s egocentric end-effector view.  \n\n-- Another limitation is in the contrastive learning method in this setting. To sample the positive and negative samples for each anchor point, CRADLE has to compute the distance from the anchor points to all state samples in the dataset, which hinders the scaling of this framework to a large dataset.\n\nOverall, I think that there should be a discussion in the paper on the egocentric characteristics with the above points. It is important for the imitation learning community for this paper to have a clear scope. Other limitations are already mentioned by the authors.\n\n**Quality & Clarity**\n\nI highlight some comments on the paper presentation.\n\n++ The paper is easy to follow and has a clear structure.\n\n++ Fig. 1 and 2 assist the paper’s understanding well.\n\n++ I believe this paper has already good experiment setups. However, more experiments on new tasks such as pouring, cleaning, etc. beyond pick&place task is appreciated.\n\n-- Further explanation in Section 3.2 is needed. What is the blocking controller? How is the shared inverse kinematic solver designed?\n\n-- There is a typo on the equation at line 178.\n\n-- There is an incomplete discussion on line 268.\n\n-- There are unnumbered equations.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "h39LcAyzedF", "reviewer": "Reviewer_tpTQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a framework to enable learning a morphology-agnostic policy to facilitate learning from datasets accumulated across different robots. It does this by aligning demonstrations from different robots using 3 steps: i) align the visual input by using a wrist-mounted cameras, ii) align action space by using multi-head policies to capture robot-specific dynamics, and iii) align internal representations using contrastive learning.", "summary_of_recommendation": "The paper is well written and easy to follow. It addresses an important and timely problem: being able to leverage datasets from other robots. The proposed method is simple and effective, and the experimental baselines are well-designed. However, the experiments are conducted on a fairly limited set of tasks that are very similar to one another.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please address my comments from the \"Weaknesses and Suggestions\" section of my review comments and let me know if there are any substantive misunderstandings on my part. In particular, please comment on how this method would be able to handle end-effectors that look significantly different, like multi-fingered hands or suction cups (ignoring, of course, the challenges of low-level control).", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Strengths**\n- The paper considers several practical challenges and systematically addresses them one by one.\n- The paper is well written and easy to follow.\n- The experimental results are presented and discussed in a comprehensive manner.\n- The results conclusively demonstrate both the effectiveness of the approach and the need for each of its components.\n\n**Weaknesses**\n- The tasks and robots used in the experiments are not that different. Indeed, as the authors acknowledge, the method does not solve the correspondence problem completely. Instead solves it within the context of a set of similar tasks on . \n- The method seems to be restricted to datasets collected with the wrist mount camera. It is unclear if results would be affected even if a different wrist-mounted camera is used.\n- Minor: 1. There might be a mismatch between the colors in Fig.2 and the caption. 2. Both \"i\" and \"n\" are used to denote task index first paragraph on Section 3 ", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "9KsOKYzAV95", "reviewer": "Reviewer_D1Nx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a new method to transfer the trained policy and also reuse the datasets across different robots. The visual observation alignment and action alignment are proposed, in addition, to solve the domain shift, the internal representation of the policy is also learned via contrastive learning.\n", "summary_of_recommendation": "A weak rejection is given based on the current status. But the reviewer is willing to increase the score if all concerns are resolved during the rebuttal.\n\nMy concerns have been addressed after reading the rebuttals, thus increasing my score to weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Please refer to the pros and cons part.\n\nQuestions,\n\nThe reviewer is curious about whether the proposed method is applicable to more diverse settings. For example, the two control tasks are the same, i.e., pushing a box to the target location. However, the friction may be different. In one task the box is on smooth ground, while in the other one, the box is on a carpet. In such a case, how to adapt the policy to the new setting? From the current version, the action alignment does not consider this.\n\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nStrengths\n\n1) This paper considers an important direction, which will draw the attention of the research community.\n\n2) Experiments show impressive results of the proposed method.\n\n\nWeaknesses:\n\n1) Some technical details need to be clarified.\n\n    The reviewer is confused about why the observations are the same since the height and angle of the cameras are different. Did the authors set a fixed height and angle, so that adjust all observations based on them?\n\n    The reviewer is confused about the problem setting since the notations are confusing. \n\n      a) In Line 102, 'would like to learn a policy $\\pi_{k,N+2}(a|o)$', why not using $\\pi_{j,N+1}(a|o)$? Does this mean the goal is to learn a new policy on a new task N+2 using robot k, given a dataset on the other new task N+1 using robot j?\n\n      b) In Line 110, does $t$ denote the time step? Then why $t$ is included in a policy? However, in Line 102, $t$ is not included in a policy denotation.\n\n    The action alignment part is also confusing. It seems like the action is converted to the state (observation) change, but under what space? The aligned observation space? Also, the reviewer is not familiar with inverse kinematics solver, it would be better to describe how and why ' a shared inverse kinematics solver could minimize inconsistencies in interpretation of pose commands'.\n\n2) Missing some discussions of related works.\n\n3) More limitations should be provided in the paper.\n\nQuestions,\n\nThe reviewer is curious about whether the proposed method is applicable to more diverse settings. For example, the two control tasks are the same, i.e., pushing a box to the target location. However, the friction may be different. In one task the box is on smooth ground, while in the other one, the box is on a carpet. In such a case, how to adapt the policy to the new setting? From the current version, the action alignment does not consider this.\n\nMinors,\n\nSome typos need to be fixed, for example, in Line 139, 'a end-effector follows...' -> 'an end-effector follows...'\n\n\nSome related works are not investigated in this paper, for example [1-4].\n\n[1] Learning cross-domain correspondence for control with dynamics cycle-consistency.\n\n[2] Reinforcement Learning with Videos: Combining Offline Observations with Interaction.\n\n[3] Cross-domain Adaptive Transfer Reinforcement Learning Based on State-Action Correspondence.\n\n[4] Sim-to-real robot learning from pixels with progressive nets.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a new method for robot manipulation where a single policy can be used for different robots.

After the rebuttal, the reviewers achieve consensus on accepting the paper.
Majority concerns of the reviewers have been addressed.

According to the scores of the reviewers, I recommend to accept the paper as a poster presentation.
Decision: Accept (Poster)