Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviation
Authors: Ria Doshi
Homer Rich Walke
Oier Mees
Sudeep Dasari
Sergey Levine
Venue: CoRL 2024 (2024)
Keywords: Imitation Learning
Cross-Embodiment
TL;DR: We train a transformer-based policy on 900K robot trajectories across 20 embodiments with varying observation and action spaces and find that it matches performance of robot-specific approaches.

Abstract: Modern machine learning systems rely on large datasets to attain broad generalization, and this often poses a challenge in robotic learning, where each robotic platform and task might have only a small dataset. By training a single policy across many different kinds of robots, a robotic learning method can leverage much broader and more diverse datasets, which in turn can lead to better generalization and robustness. However, training a single policy on multi-robot data is challenging because robots can have widely varying sensors, actuators, and control frequencies. We propose CrossFormer, a scalable and flexible transformer-based policy that can consume data from any embodiment. We train CrossFormer on the largest and most diverse dataset to date, 900K trajectories across 20 different robot embodiments. We demonstrate that the same network weights can control vastly different robots, including single and dual arm manipulation systems, wheeled robots, quadcopters, and quadrupeds. Unlike prior work, our model does not require manual alignment of the observation or action spaces. Extensive experiments in the real world show that our method matches the performance of specialist policies tailored for each embodiment, while also significantly outperforming the prior state of the art in cross-embodiment learning.

Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviation
Ria Doshi Homer Walke*
Oier Mees
Sudeep Dasari²
Sergey Levine!
UC Berkeley Carnegie Mellon University
https://crossforme--mollell github.oo
Abstract: Modern machine learning systems rely on large datasets to attain broad generalization, and this often poses challenge in robot learning, where each robotic platform and task might have only small dataset. By training single policy across many different kinds of robots, robot learning method can leverage much broader and more diverse datasets, which in turn can lead to better gener- alization and robustness. However, training a single policy on multi-robot data is challenging because robots can have widely varying sensors, actuators, and con- trol frequencies. We propose CrossFormer, a scalable and flexible transformer- based policy that can consume data from any embodiment. We train CrossFormer the largest and most diverse dataset date, 900K trajectories across 20 dif- ferent robot embodiments. We demonstrate that the same network weights control vastly different robots, including single and dual arm manipulation sys- tems, wheeled robots, quadcopters, and quadrupeds. Unlike prior work, our model does not require manual aligmment of the observation or action spaces. Extensive experiments in the real world show that our method matches the performance specialist policies tailored for each embodiment, while also significantly outper- forming the prior state of the in cross-embodiment learning.
Keywords: Imitation Learning, Eross-Embodiment
Quadrupeds
Robot Trajectories
Single Arms
Embodiments
Cross-Embodied Transformer
Navigation
Monipulation
Locomation
E Flecible Observation Spaces
Navigation
Bimaaussi
1400...1000)
C Control Freq, :(SHz....20Hx]
Bimanual Arms
Quadcopters
Figure We introduce CrossFormer, a transformer-hased policy trained on 900K trajectories of diverse, multi-embodiee robot data that can control vastly different robots including single and dual arm manipulation systems, wheeled robots, quadcopters. and quadrupeds, while matching the performance of specialist policies targeted each embodiment and outperforming prior work in gros--emoodiment learning.
1 Introduction
Much of the recent success in machine learning has been driven by training general-purpose models on increasingly diverse and multi-task data. For example, visual and language tasks, once handled by task-specific methods, are now performed more effectively by general languale models that can transfer knowledge across tasks [1, 2, 3, 4]. Similarly, in roboties, recent data aggregation efforts [5] have made it possible to train general policies on robot data collected across
Equal contribution. Corresponding email: (riadoshi, homer homer_walke, wbe.kkeeeebbrrkee.........
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

to choose the task modality multiple embodiments, tasks, and environments. These generalist policies can out-perform narrow policies trained with data from only the target robot and task by transferring visual representations and skills [6. 5]. In addition to the benefits of positive transfer, training generalist cross-ent embodied policies minimizes the amount of engineering that must done to design and tune policy architec- tures for each robot.
purpose robot policy is uniquely challenging, since robot systems can vary widely in their camera views, proprioceptive inputs, joint configurations, action outputs, and control frequencies. Initial efforts at training large-scale cross-embodied policies have often been limited to single robot arms or ground navigation robots which can be controlled with a single cam- era view and relative waypoint action for the base or end-effector [5, 6, 7, 8]. Further increasing the diversity of embodiments these policies can control requires a model architecture that supports conditioning any number of camera views or proprioceptive observations, as well as predicting actions of any dimension. Following prior work, we take a sequential modeling approach to cross- embodied imitation learning [9, 10]. We propose a transformer- policy that supports variable observations and actions by casting inputs and outputs to sequence. We scale this approach to con- trol the most diverse set of embodiments with a single policy to date, including single and bimanual robot arms, ground navigation robots, quadcopters and quadrupeds.
However, training
With our transformer policy, prioceptive sensors by simply tokenizing and arranging the observations into a sequence. At the same time, we can predict actions of any dimension, crucially without the need to manually align the action spaces of different embodiments [8]. For each action type, we insert a set of action readout tokens into the input token sequence. Then, we pass the corresponding output embeddings into action- space specific heads to produce vector of the correct dimensions. Our policy can accept tasks in the form of language instructions or goal images, allowing that is most natural for given embodiment.
train robot data with any number of camera views pro-
Our primary contribution is a cross-embodied robot policy trained on the largest and most diverse robot dataset to date with 900K trajectories and 20 distinct embodiments. Our policy can control robots with varying observation and action types, from quadruped with proprioceptive sensors and 12 joints to a bimanual robot with cameras and 14 joints. We find extensive real world experiments that our policy matches the performance of the same architecture trained on just the target robot data, as well as the best prior method in each setting, demonstrating that our architecture can absorb heterogeneous robot data without negative transfer, while performing comparably to state-of-the-arr specialist methods tailored for each robot. Additionally, we find that our method outperforms the state of the art in cross-embodiment learning while alleviating the need for manually aligning observation and action spaces.
2 Related Work
Early work on cross emboddied robot policy learning has explored number of techniques, including conditioning explicit or learned representations of the embodiment, [11, 12], domain random- ization and adaptation [13, 14, 15, 16, 17, 18), modular policies [19, 20, 21, 22], or model-based RL [23, 24, 25). Generally, these prior projects have operated on a smaller scale, only evaluating in simulation or training policies on small amounts of robot data to control a small number of robots for a few tasks.
A number of prior works have sought scale up robot learning by using large amounts of data from a single robot embodiment, collected either autonomously [26, 27, 28, 29, 30, 31] or with human telcoperation [32, 33, 34, 35, 36. 37, 38, 39]. Other prior works train on data from multiple robots, but require each robot to have the same observation and action space [7, 40, 5, 41, 42, 43, 44]. For example, Shah et al. [42] train one policy across many navigation robots using an egocentric camera view and 2D waypoint actions, and the RT-X models [5] are trained across single robotic arms using a 3rd person camera view and 7-DoF end-effector position actions. Our method does not require the data to have common observation and action space, enabling simultancous control of robots with disjoint sets of sensors and actuators, c.g., robot arms and quadrupeds.

---

There have been few large-scale efforts to train a single policy on robot data with varying obser- vations and action spaces (10, 9, 6, 8). Octo [6] can be fine-tuned on robots with observations and actions that are distinct from those seen during pre-training. However, Octo only pre-trains on data from single robot arms and does not explore co-training on more heterogeneous data. Reed et al. [9] and Bousmalis et al. [10] propose a flexible transformer-based policy that handles varying obser- vation and action spaces during pre-training. They demonstrate their policy can control robot arms with different action spaces, including 4-DoF, 6-DoF, 7-DoF, and 14-DoF (using 3-finger hand), We explore the challenges in increasing both the diversity of embodiments and environments a sin- gle policy can operate in. Along with increasing the number of arm embodiments we can control, e.g., fighffrey bimanual arms, we demonstrate navigation and quadrupedal locomotion, and we evaluate across wide range of real-world settings, while they limit their evaluation to a standard- ized cage. We also explicitly evaluate for transfer across embodiments by comparing policies with and without cross-embodiment co-training, while their focus is on autonomous self-imprrvemment Perhaps most related to our work, Yang al. [8] study transfer across manipulation and navigation data. However, their focus is on leveraging the fact that egocentric motion in navigation looks simi- lar to egocentric motion from wrist cameras in manipulation, and they perform manual alignment of the actions across these two embodiments. Instead, our focus is on training a policy that can control embodiments beyond arms and ground navigation robots, including those with observations and ac- tion spaces that cannot be cast to common format. Our method, CrossFormer, is the first to co-train on four distinct action spaces (single-arm manipulation, ground navigation, bimanual manipulation, and quadrupedal locomotion) without any observation space constraints action-space alignment while maintaining state-of-the-ar performance on each robot.
Training generalist, cross-embodied policies requires multi-robot datasets, and there have been sev- eral efforts to collect such large-scale cross-embodied datasets [45, 41, 42, 5, 46). particular, the Open Cross-Embodimess dataset (OXE) [5] aggregates 1.5 million episodes of robot data, and we train subset of 900K trajectories. We note that the Octo [6] and RT-X [5] models are also trained on the OXE dataset, however they only subset with single robot arms. We additionally
the GNM [41] navigation and DROID [35] (large-scale Franka manipulation) datasets, along with Gol quadruped and ALOHA [47] bimanual data collected by concurrent work [48].
3 Designing a Cross-Embodied Policy
The primary challenge in robotic learning with many embodiments lies in handling widely vary- ing observation and actions spaces, well as differences in control frequency and other aspects of the robotic system. Robotic systems can have varying numbers of camera views or propriocep- tive sensors, and they may be controlled by a variety of different action representations, including joint angles, Cartesian positions, and motor torques. In order to standardize the data to a common format, some prior work on training cross-embodied policies has ignored certain observation types (such as either wrist or third person views in manipulation)
[5. 7] or aligned action spaces across robots [8]. Instead, following other prior work [9, 10, 6). we cast cross-embodied imitation learn- ing
pegueecc-to-sequenee problem and choose transformer-base policy architecture that can handle varying length sequential inputs and outputs.
Due to their sequential nature, a transformer policy enables encoding all available observation types from each embodiment by serializing them into a flat sequence. Similarly, we can decode variable length actions which allows us to use the optimal action type for each embodiment. Using this flexible output, we can also predict variably sized chunks of actions. Action chunking [49, 47, 50] improves the temporal consistency of actions and reduces compounding error which is particularly important for high-frequenc, fine manipulation. Together, a transformer backbone and action chunk- ing enables our policy to control robots ranging from a bimanual ALOHA system with joint position control at 20Hz, to ground and aerial navigation robots with 2D waypoint control at 5Hz.
At high level, our transformer policy follows prior work that trains transformers on multi-modal data [9, 10, 6]. Observations and task specification are tokenized by modality-specifff tokenizers, assembled into token sequence, and fed into a causal, decoder-only transformer backbone that is
3

---

Observation Image Tokenization
Workspooe Image
Narigatin Imoge
Wiit Image
Guadruped Proprio Binanual Poopr
Readout Tokens
Camnet
Sawee the otjecte
diappa
Cross-Embodied Transformer
Broodder
Gendiomm
ResNet
Ousdruped
Single
Navigation
Bimanual Action
Headd
Figure 2: Policy architecture. Our architecture enables cross-embodied policy learning through transformer backbone. Our policy accepts variable observation inputs by tokenizing images and proprioceptive information, predicts variable action outputs using action readout tokens, and condi- tions on language instructions or goal images.
shared across all embodiments. The output embeddings are then fed into separate action heads for each class embodiment to produce actions of the corresponding dimension, as shown in Figure 2.
3.1 Training data
Our training data mixture covers 20 different robot em- bodiments, varying widely in observation space, action space, and control frequency. We start from the single- arm manipulation subset of the Open Cross-Embodiment dataset used by Octo [6]. Then, we add the DROID Franka manipulation dataset [35], 7K trajectories of ALOHA data collected across two institutions (referred to as ALOHA-multi-Bee
2 Quadruped
8%
t Bimanual 17%
Dataset Embodiment Sampling Weights
60 hours of navigation from the GNM dataset (41], 25 minutes of walking data from a Gol quadruped (referred to Gol Gol-walk), and 200 trajec- tories of additional Franka data collected in own lab (referred to Franka-tabletop). We classify the datasets that are most relevant to each of our evaluation settings (see Section 4) as the target datasets and up-weight them relative to the other datasets during training. Our tar- get datasets are BridgeData [36] for WidowX evaluation, ILOHA--mlti-tar for ALOHA evaluation, GNM [41] for navigation evaluation, Gol-walk for quadruped eval- uation, and Tranka-tabletoyy for Franka evaluation. We collected the Gol data by rolling out expert policy trained with RI in simulation [51]. See Figure 3 for our training data mix and Appendix A for further details on the composition of datasets collected ourselves along with the sampling weights.
58%
Navigation
Single Anm
17%
Figure Training Data Mix. We group the 20 embodiments in our train- ing data into categories and visualize their contribution to our data mix. The pie chart shows the average composition of each training batch based on the sam- pling weights.
3.2 Tokenizing variable observation types and task specifications
The first step training our cross-embodied policy is creating the input sequence. The trajectories our robot training data are sequences of time-steps where each time-step contains image obser- vations I, proprioceptive observations P. and an action. Data from each embodiment may have different number of camera views per time-step and may may include proprioceptive obser- vations. To create the input sequence, first define observation history length k and chunk each trajectory into k-length segments, [Te, Pt..., T Pr+a.). We then tokenize each observation accord- ing to its modality. Images are processed with a ResNet-26 encoder [52] to produce feature map that is flattened along the spatial dimensions and projected to the token embedding size. Proprio- ceptive observations are simply projected to the token embedding size. Along with the observation sequence, the policy also accepts a task specification. Importantly for cross-embodied control, our policy accepts task specifications either in the form of language instruction or goal image g. In some settings, like navigation, tasks more naturally specified as image goal whereas in other settings, like manipulation, tasks are more easily specified with language. Language instructions are

---

jointly processed with image observations using FiLM [53]. Goal images are stacked on the current image along the channel dimension before being fed into the image encoder.
Because our training data consists of data from single-arm manipulators, dual-arm manipulators, quadrupeds, and ground navigation robots, our policy supports conditioning on any subset of the following observation types: (1) Workspace Image: A 3rd person camera view in manipulation set- tings. (2) Navigation Image: An egocentric camera view in navigation settings. (3) Wrist Image: A view from a wrist-mounted camera in manipulation settings. (4) Quadruped Proprioception: Joint positions and velocity estimates for quadrupeds. (5) Bimanual Proprioception: Joint posi- tions for bimanual manipulation settings. To maximize transfer across embodiments, we share image encoder weights for camera views of the same type. So, for example, the workspace images in single arm and bimanual manipulation settings are processed by the same ResNet image encoder. In total four image encoders: one for the workspace view in manipulation settings, one for the ego- centric view from ground navigation robots, and two for wrist cameras in manipulation settings. Af- ter input tokenization, we have sequence of observation tokens . ..., 11: t+k J. where I and M denote the number of tokens for image and proprioceptive observations.
3.3 Predicting variable length actions
After creating the input sequence, the next step is to process the input sequence with transformer to predict an action of an appropriate dimension for each embodiment. We use a transformer with block-wise causal attention mask such that observation tokens can only attend to observation tokens at the same or prior time-steps f. Following prior work [6], we insert special readout tokens R after the observation tokens at each time-step in the input token sequence. These readout tokens can only attend to prior observation tokens, and thus serve as a convenient representation from which to predict actions. The final input sequence is [L.. P P.I.M. RR. .... I1:L P R where N denotes the number of readout tokens. We pass the input token sequence through the transformer to obtain an embedding sequence. We then apply the readout tokens to produce actions. There are several possibilities for the action head. Past work has explored regression with an LI or L2 loss, classification with a cross-entropy loss, or diffusion. We choose to predict continuous actions and employ an L1 loss because of its success in prior work on high- figg-frry bimanual manipulation [47]. Accordingly, our action heads simply project the readout token embeddings to the action dimension. For some embodiments, we predict chunk of sequential actions. Action chunking has been shown to improve policy performance in prior work [47, 50] and is essential for embodiments with high control frequencies where compounding error would build up too quickly. Since our action heads project the readout tokens to the action dimension size, we match the number of readout tokens to the action chunk size for each embodiment. Our policy has 4 action heads which produce chunked actions of the following types: (1) Single arm Cartesian positions: 7-dimensional action denoting the relative change in Cartesian position of the end-effector and the gripper actuation. We predict a chunk of 4 actions and execute on single robot arms 5-15Hz [6]. (2) Navigation waypoints: A 2-dimensional action denoting a waypoint relative to the robot's 's current position. We predict chunk of 4 actions and execute navigation robots at 4Hz [42] (3) Bimanual joint positions: A 14-dimensional action denoting the joint positions of both arms. We predict a chunk of 100 actions and execute on bimanual robot arms at 20Hz [47]. (4) Quadruped joint positions: A 12-dimensional action, predicted with no chunking, denoting the joint positions of the legs. We predict only I action and execute on a quadruped at 20Hz [51]. Action chunk sizes are taken from prior work in each robot setting.
action head to the embeddings corresponding to
3.4 Training details
In practice, we mask observations that are missing for an embodiment, so that each batch element contains all observation types and all readout token groups, and these token groups occupy fixed locations within the context window. Alternatively, for memory efficiency, observation and readout tokens could be densely packed to remove padding and more time-steps of context for embod- iments with fewer observation types. This is strategy used by prior work [9]. However, by not fixing observation and readout token types to set location in the context window, the model would need to infer the embodiment purely from the observations in order to predict actions of the correct
5

---

WidowX
ALOHA
Franka
LoCoBot
Go1
TELLO
Figure 4: Evaluation Settings: Our tasks include single-am arm manipulation settings, dexterous and bimanual task settings, navigation, and aviation. Please refer to Sec 4 for full breakdown.
type (rather than relying on the positional embeddings of the readout tokens). The observations for some embodiments can look similar (such as navigation and manipulation with only a wrist camera), so this design may require appending a prefix to the token sequence indicating the embodiment.
Our transformer backbone has 12 layers, 8 attention heads, an MLP dimension of 2048, and token embedding size of 512. In total, our model has 130M parameters. We initialize the ResNet-26 encoders with ImageNet pre-trained weights. We use a context widow size of 2135 tokens, which fits 5 time-steps of context with all observation and readout token groups present. We found that good performance on navigation required time-steps of observation history and using this context length did not hurt performance for other embodiments. We trained for 300K gradient steps with batch size of 512 which took 47 hours TPU V pod. We use the AdamW optimizer [54], an inverse square root decay learning rate schedule [55], weight decay of 0. and gradient clipping of 1.0. We apply standard image augmentations. During training, we use hindsight goal relabeling and sample future observations uniformly at random to use as goals [56]. If language instruction available for a trajectory, we randomly mask either the language or goal so that at test time we can condition our policy using either task specification [57, 58].
4 Evaluation
Our experiments are designed to test whether our single cross-embodied policy can control multi- ple robot embodiments without sacrificing performance when compared to robot-specific imitation learning approaches. Specifically, we aim to answer the following questions: (1) Can our policy trained on many robots match the performance of a policy trained on data from only the target robot? (2) Does our policy match the performance of the best prior imitation learning method in each robot setting? (3) How does our policy compare to prior cross-embodied policies that align observation and action spaces?
Evaluation setups. We evaluate over a wide range of tasks and embodiments (see Fig. 4): (1) WidowX Manipulation: We use the Bridge setup from Walke et al. [36]. We use an over-the- shoulder camera view and and sample actions from the single arm head of our policy. We perform 48 trials per policy over two kunguage-conditione tasks and two conditond tasks. (2) Franka Manipulation: We the DROID setup from Khazatsky et al. [35]. We
over-the-shoulder camera view and sample actions from the single arm head of our policy, We perform 39 trials per policy over two anguuge-condittoone tasks. (3) ALOHA Bimanual Manipulation: We use the ALOHA setup from Zhao et al. [47]. We use 3 camera views, one overhead and two wrist, and sample actions from the bimanual head of our policy. We perform 20 trials per policy over two language-conditionee tasks. (4) LoCoBot Navigation: We use the LoCoBot setup from Shah et al. [42] which has one camera view. We evaluate on suite of three skills: path-following, obstacle avoidance, and sharp corner-turning. We collect topological map of goals similar to Shah et al. [42] and evaluate success based distance the closest node within the topological map of goal images. We sample actions from the navigation head of our policy. We evaluate in 6 locations, using one trial per location and policy as done in prior work [41, 42]. (5) Gol Quadruped: We evaluate on a Unitree Gol which uses proprioceptive observations 0, E R 50 We sample actions from the quadruped head of our policy. As our evaluation metric we report the average reward achieved over 25 minutes normalized by the the reward achieved by the RL-trained expert policy that generated the data (see Section 3).. (6) Tello Quadcopter: Finally, we experiment with Tello quadcopter using the navigation head of our policy. Since the navigation head outputs 2-D relative waypoints, maintain static height throughout the trajectory [42, 41]. Notably, we do not train quadcopter

---

1
0.75
0.5
0.25
0
Average
WidowX
Franka
ALOHA
LoCoBot
Go1
TELLO
Best Priar Method
Single-Robot Dataset
CrossFormer
Figure 5: Real Evaluation. We compare CrossFormer to the architecture trained just the data from the target robot, as well as the best prior method on the data from the target robot.
data so this setting requires zero-shot generalization to new embodiment (though not to a new set of observation inputs and action outputs). We evaluate in locations, using one trial per location and policy as done in prior work [41, 42].
Baselines and comparison methods. To evaluate how co-training on data from other embodi- ments affects performance for given embodiment, we train our policy architecture just the data from the target embodiment using the target dataset (see Section 3.1) for each evaluation setting. Because these datasets are smaller than our full data mix, we create smaller versions of our archi- tecture, ranging from 5M to 95M parameters, to avoid overfitting. We refer to this method as the single-robot dataset baseline. We also compare our method to the best prior method in each set- ting to evaluate whether for each robot. For the WidowX we use Octo [6], for the Franka we use OpenVLA [59], for ALOHA we use ACT [47], for the LoCoBot and Tello quadcopter we use ViNT [42]. We do not compare to a prior satte-of-thh-ar method for the Gol setting due to the lack of corresponding imitation learning method for quadrupedal locomotion. Finally, in the navigation and manipulation settings, we compare to the policy from Yang et al. [8], by using the original 186M parameter policy. Yang al. [8] leverage the fact that the egocentric view navigation looks similar wrist camera views in manipulation, and they align the action spaces of these two settings accordingly. This comparison evaluates whether approach can perform similarly with manual alignment of action spaces.
match the performance of approach that specifically tuned
4.1 Can CrossFormer match the performance of training on only a single-robot dataset, as well the best prior method in each setting?
Figure 5 shows the performance of our method compared to our architecture trained on just the single-embodiment target datasets for each setting (see Section 3.1) and the best prior method for each setting. Overall, we find that our method performs comparably to training on just the target dataset all evaluations, demonstrating that our architecture can absorb data from widely varying embodiments with no negative transfer. On average over all embodiments, CrossFormer has 73% success rate, while the single-robot dataset baseline has 67% success rate. Additionally, Cross- Former performs similarly to the best prior imitation learning method in each evaluation setting, demonstrating that our policy architecture can match state-of-the-arr methods that have been specifi- cally developed and tuned for each robot. On average over all embodiments, CrossFormer has a 73% success rate, while the best prior method has a 51% success rate. Because there is no suitable prior imitation learning method for the Gol, it is not included in the reported best prior method average.
4.2 Is aligning the observation and action spaces really needed?
Lastly, we compare our policy to Yang et al. [8] in the LoCoBot navigation and WidowX manip- ulation settings. the navigation setting, our policy significantly outperforms Yang al. [8] in all tested scenarios including turning corners, avoiding obstacles, and making sharp turns. Qualita- tively, we also found our policy to be much smoother, navigating in straighter lines with fewer stops and starts. Both policies trained on the same amount of LoCoBot data (from the GNM dataset),
7

---

0.75
+ Sharp Turns
0.5
0.25
0
Basic Coreer
Obstacle Avoidance
Pick Place
Yang et. al.
CrossFormer
Figure 6: Comparison to Yang et al. [8]. We compare CrossFormer to Yang et al. [8], which aligns actions for navigation and manipulation and only uses a single camera view at time, CrossFormer outperforms Yang al. [8] by 3x overall, on both tabletop manipulation from third person camera view as well as common navigation tasks.
so this result indicates that our policy architecture can better fit its cross-embodiment training data. Surprisingly, in the WidowX manipulation setting, found that Yang et al. [8] obtained success rate, despite training on the same relevant WidowX data as CrossFormer (i.e the Bridge dataset) and additional WidowX data collected in their own lab. We hypothesize that this is because we evaluate from the 3rd-person camera view. While Yang et al. [8] train with both 3rd-person and wrist camera views, only one view is input to the policy at a time, potentially resulting in underfitting to 3rd-person control. CrossFormer only uses the 3rd-person view for the Widow setting due to lack of publicly available WidowX wrist-camera data. However, we note that our policy can accept varied number of camera views (using up to three simultaneously for bimanual manipulation) and does not require forcing two camera views into the same input slot. In summary, our policy out- performs Yang et al. [8] in both settings, demonstrating that manual alignment of observation and action spaces is not necessary for good performance, and that our flexible architecture allows our policy to better fit heterogencous cross- cross-embodied data.
Discussion and Conclusion
We introduced CrossFormer, a scalable and fiexible transformer policy trained the largest and most diverse dataset to date, 900K trajectories across 20 different robot embodiments. We demon- strated a principled way to learn a single policy that can control vastly different embodiments in- cluding single and dual arm manipulation systems, wheeled robots, quadcopters, and quadrupeds. Our results show that CrossFormer matches the performance of specialist policies tailored for indi- vidual embodiments, while also significantly outperforming the state of the art cross-embodimenn learning. However, our work does have limitations. Our results do not yet show significant posi- tive transfer across embodiments. We anticipate that train larger robot datasets with more embodiments, we will see greater positive transfer. Another limitation is that our data mix uses hand-picked sampling weights to avoid over-training on datasets with many repetitive episodes and under-training on the data most relevant to our evaluation settings. In principle, as we seale model size, the policy should have the capacity to fit all the data equally well without any data weight- ing. Finally, given that we need large models fit large multi-robot datasets, the model's inference speed can become limiting factor. In this work we successfully applied our policy frequency, fine-grained bimanual manipulation task, but as scale the model's size may not be able to control these higher frequency embodiments. Future hardware improvements will help to alleviate the issue, but further research is needed on techniques for using large models to control fighfrryyenccy robots. Future work could also include exploring techniques to enable greater pos- itive transfer across embodiments while maintaining the flexibility of our architecture, techniques for data curation, and incorporating even more diverse data sources like sub-optimal robot data or action-free human video. We hope that this work will open the door for more general-purpose and flexible robot policies that efficiently learn and transfer knowledge from the experience collected across diverse robot embodiments.
high-

---

Acknowledgments
We thank Laura Smith, Kyle Stachowicz, Karl Pertsch, Ajay Sridhar, Jonathan Yang, and Catherine Glossop for help with setting up the robots and baselines. This research was supported by the TPU Research Cloud, NSF I1S-2150826, ARL DCIST CRA W911NF-11-2--0188 and ONR N00014-20- 1-2383.
References
[1] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. Chat- terji, A. Chen, K. Creel, J. Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, Etchemendy, K. Ethayarajh,
Fei-Fei, C. Finn, T. Gale, L.. Gillespie, K. Goel, N. Goodman, Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D. E. Ho, Hong, K. Hsu, J. Huang, T. Icard, S. Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling, F. Khani, O. Khattab, P. W. Koh, M. Krass, R. Krishna, R. Kuditipudi, A. Kumar, F. Ladhak, M. Lee, T. Lee, J. Leskovec, I. Levent, X. L. Li, X. Li, T. Ma, A. Malik, C. D. Manning, S. Mirchan- dani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nie, J. C. Niebles, H. Nilforoshan, J. Nyarko, G. Ogut, L. Orr, I. Papadimitriou, J. S. Park, C. Piech, E. Portelance, C. Potts, A. Raghunathan, R. Reich, H. Ren, F. Rong. Y. Roohani, C. Ruiz, Ryan, C. Ré, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. Srinivasan, A. Tamkin, R. Taori, A. W. Thomas, F. Tramèr, R. E. Wang, W. Wang. B. Wu, Wu, Y. Wu, S. M. Xie, M. Yasunaga, J. You, M. Zaharia, M. Zhang, T. Zhang, X. Zhang, Y. Zhang. L. Zheng, K. Zhou, and P. Liang. On the opportunities and risks of foundation models, 2022.
[2] W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y. Wang, J. Ji, Z. Yang, L. Zhao, X. Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.0307,, 2023.
[3] H. Liu, Ci, Q. Wu, and Y. J. Lee. Visual instruction tuning, 2023.
[4] OpenAI. Gpt-4 technical report, 2024.
[5] Open X-Embodiment Collaboration, A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, A. Raffin, A. Wahid, B. Burgess- Limerick, B. Kim, B. Schölkopf, B. Ichter, C. Lu, C. Xu, C. Finn, C. Xu, C. Chi, C. Huang, C. Chan, C. Pan, C. Fu, C. Devin, D. Driess, D. Pathak, D. Shah, D. Büchler, D. Kalash- nikov, D. Sadigh, E. Johns, F. Ceola, Xia, F. Stulp, G. Zhou, G. S. Sukhatme, G. Salhotra, G. Yan, G. Schiavi, H. Su, H.-S. Fang, H. Shi, H. B. Amor, H. I. Christensen, H. Furuta, H. Walke, H. Fang, I. Mordatch, I. Radosavovic, I. Leal, J. Liang, J. Kim, J. Schneider, J. Hsu, J. Bohg. J. Bingham, J. Wa, J. Wu, J. Luo, J. Gu, J. Tan, J. Oh, J. Malik, J. Tompson, J. Yang, J. J. Lim, J. Silvério, J. Han, K. Rao, K. Pertsch, K. Hausman, K. Go, K. Gopalakrishnan, K. Goldberg, K. Byrne K. Oslund, K. Kawaharazuka, K. Zhang, K. Majd, K. Rana, K. Srini- vasan, L. Y. Chen, L. Pinto, L. Tan, L. Ott, L. Lee, M. Tomizuka, M. Du, M. Ahn, M. Zhang, M. Ding, M. K. Srirama, M. Sharma, M. Kim, N. Kanazawa, N. Hansen, N. Heess, N. J. Joshi, N. Suenderhauf, N. D. Palo, N. M. M. Shafiullah, O. Mees, O. Kroemer, P. R. Sanketi, P. Wohlhart, P. Xu, P. Sermanet, P. Sundaresan, Q. Vuong, R. Rafailov, R. Tian, R. Doshi, R. Martín-Martnn, R. Mendonca, R. Shah, R. Hoque, R. Julian, S. Bustamante, S. Kirmani, S. Levine, S. Moore, S. Bahl, S. Dass, S. Song, S. Xu, S. Haldar, S. Adebola, S. Guist, S. Nasiri- any. S. Schaal, S. Welker, S. Tian, S. Dasari, S. Belkhale, T. Osa, T. Harada, T. Matsushima, T. Xiao, T. Yu, T. Ding. T. Davchev, T. Z. Zhao, T. Armstrong, T. Darrell, V. Jain, V. Van- houcke, W. Zhan, W. Zhou, W. Burgard, X. Chen, X. Wang. X. Zhu, X. Li, Y. Lu, Y. Chebotar, Y. Zhou, Y. Zhu, Y. Xu, Y. Wang, Bisk, Y. Cho, Y. Lee, Y. Cui, Y. hua Wu, Y. Tang, Y. Zhu, Y. Li, Y. Iwasawa, Y. Matsuo, Z. Xu, and Z. J. Cui. Open X-Embodiment: Robotic learning datasets and RT-X models. https: /awxiv.vvvvaa/aa wrw..ogg//aa............ 22tp//1///331..... 08864,08888...88... 2023.
[6] Octo Model Team, D. Ghosh. H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y. Tan, P. Sanketi, Q Vuong, T. Xiao, D. Sadigh, C. Finn, and

---

S. Levine. Octo: An open-source generalist robot policy, In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024.
[7] J. H. Yang, D. Sadigh, and C. Finn. Polybot: Training one policy across robots while embracing variability. In 7th Anmual Conference on Robot Learning, 2023. URL URLAAttp::/opparee.e: net/forum? Het.ooummmn---HRi111eeee
[8] J. Yang, C. Glossop, A. Bhorkar, D. Shah, Q. Vuong, C. Finn, D. Sadigh, and S. Levine. Push- ing the limits of cross-embodiment learning for manipulation and navigation. arXiv preprint arXiv:2402. 19432, 2024.
[9] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikow, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175, 2022.
[10] K. Bousmalis, G. Vezzani, D. Rao, C. Devin, A. X. Lee, M. Bauza, T. Davehev, Y. Zhou, A. Gupta, A. Raju, al. Robocat: A self-improving foundation agent for robotic manipulation. arXiv preprint arXiv:2306. 11706, 2023.
[11] T. Chen, A. Murali, and A. Gupta. Hardware conditioned policies for multi-robot transfer learning. Advances in Neural Information Processing Systems, 31, 2018.
[12] C. Schafff D. Yunis, A. Chakrabarti, and M. R. Walter. Jointly learning to construct and control agents using deep reinforcement learning. In 2019 international conference on robotics and automation (ICRA), pages 9798-9805 IEEE, 2019.
[13] X. B. Peng, M. Andrychowicz, W. Zaremba, and Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803-3810. IEEE, 2018.
[14] A. Kumar, Fu, D. Pathak, and Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2 040004 2021.
[15] G. Feng, H. Zhang, Z. Li, X. B. Peng, B. Basireddy, L. Yue, Z. Song, L. Yang, Y. Liu, K. Sreenath, et al. Genloco: Generalized locomotion controllers for quadrupedal robots. In Conference Robot Learning, pages 1893-1903. PMLR, 2023.
[16] A. Rajeswaran, S. Ghotra, B. Ravindran, and S. Levine. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint arXiv: 2016.
[17] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
[18] W. Yu, J. Tan, C. K. Liu, and G. Turk. Preparing for the unknown: Learning universal policy with online system identification. arXiv preprint 1702:11002004444 2017.
[19] C. Devin, A. Gupta, T. Darrell, Abbeel, and Levine. Learning modular neural network policies for multi-task and multi-robot transfer. 2017 IEEE international conference on robotics and automation (ICRA), pages 2169-2176 IEEE, 2017.
[20] W. Huang, I. Mordatch, and D. Pathak. One policy control them all: Shared modular policies for agent-agnostic control. In International Conference Machine Learning, pages 4455 4464. PMLR, 2020.
[21] Y. Chen, Y. Chen, Z. Hu, T. Yang, C. Fan, Y. Yu, and J. Hao. Learning cetion-transferable policy with action embedding, arXiv preprint
2019.
10

---

[22] H. You, T. Yang, Y. Zheng, J. Hao, and M. E Taylor. Cross-domain adaptive transfer reinforce- ment learning based on state-action correspondence. In Uncertainty in Artificial Intelligence pages 2299-2309. PMLR, 2022.
[23] E. S. Hu, K. Huang, O. Rybkin, and D. Jayaraman. Know thyself: Transferable visual control policies through robot-awareness. arXiv preprint arXiv:2107.0904 2021.
[24] Fu, Levine, and Abbeel. One-shot learning of manipulation skills with online dynamies adaptation and neural network priors. In 2016 IEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS), pages 4019-4026. IEEE, 2016.
[25] G. Salhotra, I.-C. A. Liu, and G. S. Sukhatme. Learning robot manipulation from cross- morphology demonstration. 7th Annual Conference on Robot Learning. 2023.
[26] L. Pinto and A. Gupta, Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference on robotics and automation (ICRA), pages 3406 3416.341. IEEE, 2016.
[27] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International journal of robotics research, 37(4-)):2114453 436,, 2018.
(28) D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. QT-Opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv: 10293, 2018.
[29] C. Finn and S. Levine. Deep visual foresight for planning robot motion. In 2017 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 2786-2793. IEEE, 2017.
[30] A. Gupta, A. Murali, D. P. Gandhi, and L. Pinto. Robot learning in homes: Improving gen- cralization and reducing dataset bias. Advances in neural information processing systems, 31, 2018.
[31] T. Lampe, A. Abdolmalcki. S. Bechtle, S. H. Huang, J. T. Springenberg, M. Bloesch, O. Groth, R. Hafner, T. Hertweck, M. Neunert, et al. Mastering stacking of diverse shapes with large- scale iterative reinforcement learning on real robots. arXiv preprint arXiv:2312.11374 2023.
[32] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint 2rXiv:22/2.068//1 2022.
[33] B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, al. Rt-2: Vision-languaae-actionn models transfer web knowledge to robotic control. In 7th Annual Conference on Robot Learning, 2023.
[34] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and I. Fan. VIMA: Robot manipulation with multimodal prompts. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 14975-15022. PMLR, 23-29 Jul 2023. URL https: / /ttp///rrcceee....//////////..se mlr ..reesss// v202/jiang23b. html.
[35] A. Khazatsky, K. Pertsch, S. Nair. A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403. /2945, 2024.
[36] H. Walke, K. Black, A. Lee, M. J. Kim, M. Du, C. Zheng, T. Zhao, Hansen-Estruch. Q. Vuong, A. He, V. Myers, K. Fang, C. Finn, and S. Levine. Bridgedata v2: A dataset for robot learning at scale, 2023.

---

[37] F. Ebert, Y. Yang. K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine. Bridge data: Boosting generalization robotic skills with cross-domain datasets. arXiv preprint N1v:2109.1139990 2021.
[38) A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imita- tion. In Conference on Robot Learning, pages 879-893, PMLR, 2018.
[39] H. Bharadhwaj. J. Vakil, M. Sharma, A. Gupta, S. Tulsiani, and V. Kumar. Roboagent: Gener- alization and efficiency in robot manipulation via semantic augmentations and action chunking. arXiv preprint 1.XXi:2309001914 2023.
[40] K. Kang, G. Kahn, and S. Levine. Hierarchically integrated models: Learning navigate from heterogeneous robots. arXiv preprint arXiv:2106.1322 13280, 2021.
[41] D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine. Gnm: general navigation model drive any robot. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 7226-7233. IEEE, 2023.
[42] D. Shah, A. Sridhar, Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine. ViNT: A foundation model for visual navigation. In 7th Annual Conference on Robot Learning, 2023.
URL https: aarxivvvvv./////v///ee///aaaav.a org/ abs 2206. 14846.
[43] A. Loquercio, A. I. Maqueda, C. R. Del-Blanco, and D. Scaramuzza. Dronet: Learning to fly
by driving. IEEE Robotics and Automation Letters, 3(2):1088 1095, 2018.
[44] M. Shafice, G. Bellegarda, and A. Ijspeert. Manyquadrupeds: Learning a single locomotion policy for diverse quadruped robots. arXiv preprint arXiv:2310.1048. 2023.
[45] S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and
C. Finn. Robonet: Large-scale scale multi-robot learning. In Conference on Robot Learning, pages 885-897. PMLR, 2020.
[46] H.-S. Fang, H. Fang, Z. Tang, J. Liu, C. Wang, J. Wang. H. Zhu, and C. Lu. Rh20t: A com- prehensive robotic dataset for learning diverse skills in one-shot. Towards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition @ CoRL2023, 3:5, 2023.
[47] T. 7. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 13705, 2023.
[48] S. Dasari, Mees, S. Zhao, M. Srirama, and Levine. The ingredients for robotic diffu- sion transformers. arxiv TBD, 2024.
[49] D. Q. Mayne and H. Michalska. Receding horizon control of nonlinear systems. Proceedings of the 27th IEEE Conference on Decision and Control, pages 464-465. IEEE, 1988.
[50] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[51] I.. Smith, I. Kostrikov, and S. Levine. A walk in the park: Learning walk in 20 minutes with frrel-free reinforcement learning, arXiv preprint arXiv:2208.077 077600 2022.
[52] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[53] E. Perez, F. Strub, De Vries, V. Dumoulin, and A. Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
12

---

[54] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In Internatiionl Confer- ence on Learning Representations, 2018.
[55] X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer. Scaling vision transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12104- 12113, 2022.
[56] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. In NeurlPS, 2017.
[57] C. Lynch and Sermanet. Language conditioned imitation learning over unstructured data. In RSS, 2021.
[58] O. Mees, L. Hermann, and Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Aulomation Letfers, 7(4): 1205-11222 2022.
[59] M. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, P. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and model. arXiv preprint, 2024.
C. Finn. Openvla: An open-soeree source Asioo-language
[60] E. Jang, A. Irpan, M. Khansari, Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learn- ing, pages 991-1002. PMLR, 2022.
[61] S. Belkhale, Y. Cui, and D. Sadigh. Hydra: Hybrid robot actions for imitation learning, arxiv, 2023.
[62] C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, Armstrong, and Florence. Interactive language: Talking to robots real time. IEEE Robotics and Automation Letters, 2023.
[63] E. Rosete- Beas, O. Mees, G. Kalweit, J. Boedecker, and W. Burgard. Latent plans for task ag- nostic offline reinforcement learning. In Proceedings of the 6th Conference on Robot Learning (CoRL), 2022.
[64] O. Mees, Borja-Diaz, and W. Burgard. Grounding language with visual affordances over unstructured data. In Proceedings of the IEEE International Conference on Robotics and Au- tomation (ICRA), London, UK, 2023.
[65] M. Heo, Y. Lee, D. Lee, and J. J. Lim. Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation. In Robotics: Science and Systems, 2023.
[66] R. Shah, R. Martín-Martin, and Y. Zhu. MUTEX: Learning unified policies from multimodal
task specifications. In 7th Annual Conference on Robot Learning, 2023. URL https:// openreview. wtt.oorummmdddxiiiaaa......
[67] S. Nasiriany, T. Gao, A. Mandlekar, and Y. Zhu. Learning and retrieval from prior data for skill-based imitation learning. In Conference Robot Learning (CoRL), 2022.
[68] A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay, S. Savarese, and L. Fei-Fei. RoboTurk: A crowdsourcing platform for robotic skill learning through imitation. CoRR, abs/1811.02790, 2018. URL 02790.
org/abs//////aaooo abs/
[69] G. Zhou, V. Dean, M. K. Srirama, A. Rajeswaran, Pari, K. Hatch, A. Jain, T. Yu, P. Abbeel, L. Pinto, C. Finn, and A. Gupta. Train offline, test online: A real robot learning benchmark, 2023.
13

---

[70] H. Liu, S. Nasiriany, L. Zhang, Z. Bao, and Y. Zhu. Robot learning on the job: Human in- the-loop autonomy and learning during deployment. In Robotics: Science and Systems (RSS), 2023.
[71] L. Y. Chen, S. Adebola, and K. Goldberg. Berkeley URS demonstration dataset. https: //sites. google. http://ieeebeeeleeyurrrttee
[72] S. Saxena, M. Sharma, and O. Kroemer. Multi-resolution sensing for real-time control with
vision-language models. In 7th Annual Conference on Robol Learning. 2023. URL https:
//pppreeeieww.o ///ppntevee..ne/fforrr mett/orrmmidd-uBBVV. 1d-WuBy9-IGDUA.
[73] Y. Zhu, A. Joshi, Stone, and Y. Zhu, Viola: Imitation learning for vision-based manipulation with object proposal priors. 2023.
[74] X. Zhu, R. Tian, C. Xu, M. Ding, W. Zhan, and M. Tomizuka. Fanuc manipulation: A dataset for learning- based manipulation with fanuc mate 200id robot. 2023.
[75] Z. J. Cui, Y. Wang, N. M. M. Shafiullah, and L. Pinto. From play to policy: Conditional behavior generation from uncurated robot data. In The Eleventh International Conference Learning Representations, 2022.
[76] S. Dass, J. Yapeter, Zhang, J. Zhang, K. Pertsch, Nikolaidis, and J. J. Lim. CLVR jaco play dataset, 2023. URL attpps//githii
haap//lvraa/llvrjjaaaaa plaaco______ddatset.... dataset.
[77] Luo. C. Xu, X. Geng, G. Feng, K. Fang. L. Tan, S. Schaal, and S. Levine. Multi-stage cable routing through hierarchical imitation learning. arXiv preprint arXiv:2307.08927, 2023.
[78] Y. Zhu, P. Stone, and Y. Zhu. Bottom-up skill discovery from unsegmented demonstrations for long-horizon robot manipulation, IEEE Robotics and Automation Letters, 7(2):4126-4133, 2022.
[79] R. Mendonca, S. Bahl, and D. Pathak. Structured world models from human videos. CoRL, 2023.
[80] G. Quere, A. Hagengruber, M. Iskandar, S. Bustamante, Leidner, F. Stulp, and J. Vogel. Shared Control Templates for Assistive Roboties. In 2020 IEEE International Conference Robotics and Automation (ICRA), page 7, Paris, France, 2020.
14

---

More information and videos can be found on our website: seessformer-moollliiibbbb..
A
Training Data
We list the sampling weights for each dataset in data mixture in Table 1. We up-weight larget datasets: Bridge, ALOHA-multitassk GNM, Gol-walk, and Franka-tabletop.
CrossFormer Dataset Mixture
Fractal [32]
17%
Kuka [28]
2.2%
BC-Z [60]
2.2%
Stanford Hydra Dataset [61]
0.015%
Language Table [62]
1.5%
Taco Play [63, 64]
1.2%
Furniture Bench Dataset [65]
0.83%
UTAuutin Mutex [66]
0.76%
Austin Sailor Dataset [67]
0.74%
Roboturk [68]
0.79%
Toto [69]
0.68%
Austin Sirius Dataset [70]
0.59%
Berkeley Autolab URS [71]
0.41%
IAMLab CMU Pickup Insert [72]
0.31%
Viola [73]
0.32%
Berkeley Fanuc Manipulation [74]
0.26%
NYU Franka Play Dataset [75]
0.28%
Jaco Play [76]
1.6%
Berkeley Cable Routing [77]
0.089%
Austin Buds Dataset [78]
0.072%
CMU Stretch [79]
0.053%
DLR EDAN Shared Control [80]
0.019%
DROID [35]
0.022%
Bridge [37, 36]
17%
GNM [41]
17%
ALOHA-multi-tass
17%
Gol-walk
8.5%
Franka-tabletop
8.5%
Table 1: The CrossFormer training data mixture uses datasets from the Open X-Embodiment
dataset [5] and additional data collected for this project.
B Training Hyperparameters
In Table we list the hyperparameters for the optimizer and policy architecture. In total, along with the ResNet-26 image encoders and action heads, our model has 130M parameters. We initialize the ResNet-26 encoders with ImageNet pre-trained weights. Training took 80 hours TPU V5e-256 pod. We apply color jitter and random resizing/cropping image augmentations. During training, we use hindsight goal relabeling and sample future observations uniformly at random to use as goals [56]. a language instruction is available for a trajectory, we randomly mask either the language or goal so that at test time we can condition our policy using either task specification [57]. For our method and baselines we trained, we selected checkpoints based the validation mean squared error since found this very roughly correlates with robot performance.
C Evaluation Setups
Below we provide further details on our evaluation settings (see Fig. for images). For goal- condtioned tasks, arrange the scene and take the goal image before rolling out the policy. For the ianguage-conditione tasks, use language instructions that semantically in the distribution of the training data (though may not appear exactly in the training data).
15

---

Hyperparameter
Value
Optimizer
AdamW [54]
Learning Rate
3e-4
Warmup Steps
2000
LR Scheduler
reciprocal square-root
Weight Decay
0.1
Gradient Clip Threshold
1
Batch Size
512
Layers
12
Attention heads
8
Token embedding size
512
MLP dimension
2048
Context length
2135
Total training steps
300K
Table 2:
Training hyperparameters for CrossFormer.
For the manipulation settings, the models machine with NVIDIA 4090 (or similar GPU) that was connected to the robot. For the ground navigation and drone experiments, we ran the models on a separate server and sent actions over wireless network.
WidowX Manipulation We use the Bridge setup from Walke et al. [36]. We use over-the- shoulder camera view and and sample actions from the single arm head of our policy. We evaluated two language-conditioned conditioned tasks, putting spoon goal-conditioned tasks, putting mushroom in a pot and putting a cloth on a saucer. We used 12 trials per policy and task. Positions of the objects were varied between trials.
cloth and putting carrot on plate, and two
Franka Manipulation We use the DROID setup from Khazatsky et al. [35]. We use an over-the- shoulder camera view and sample actions from the single arm head our policy. We evaluated for 27 trials per policy on the language-conditiong conditioned task of using a sponge to sweep pinecones into a dustpan. We also evaluated for 12 trials per policy on the task of flipping a pot upright. The positions of the objects were varied between trials.
ALOHA Bimanual Manipulation We use the ALOHA setup from Zhao et al. [47]. We use three camera views, one overhead and two wrist, and sample actions from the bimanual head of our policy. We evaluated two language-conditiones tasks, taking the cap off of pen and picking up a knife and using it to cut piece of sushi. We used 10 trials per policy and task. The position of the pen, knife, and sushi was varied between trials.
LoCoBot Navigation We use the LoCoBot setup from Shah al. [42] which has camera view. We evaluate suite of three skills: path-following, obstacle avoidance, and sharp comne-- turning, We sample actions from the navigation head of our policy. We combine our policy with the graph-based planner and distance function from Shah et al. [42]. We first obtain a topological map M the environment by teleoperating the robot. Then, at every time-step we find the closest node in M, search the graph for the shortest path from this node to the goal, and command the policy with the most immediate subgoal in the path. We evaluate the success of a trajectory based on the number of subgoals between the closest node at the end of the trajectory and the goal. We evaluate in 6 locations, using trial per location and policy done in prior work [41,42] 42].
Gol Quadruped We evaluate on a Unitree Gol which uses proprioceptive observations o, 5599 We sample actions from the quadruped head of our policy, and the task to walk forward. Importantly, unlike prior work [8], we directly control the quadruped's joints rather than doing higher level control with navigation waypoints. As evaluation metric report reward achieved
16

---

Task
Single-Robot Datasct
Bost Prior Method
CrossFormer
Put the spoon the cloth
0.25 (12)
0.25 (12)
0.25 (12)
Put the mushroom in the Put the cloth on the saucer
0.00 (12)
0.17 (12)
0.25 (12)
Widow
0.75 (12)
0.67 (12)
0.75 (12)
Put the carrot on the plate
0.67 (12)
0.25 (12)
0,33 (12)
Sweep the pinecones into the dustpan
0.41 (27)
0.52 (27)
0.41 (27)
Franka
Flip the pot upright
0.83 (12)
0.67 (12)
0.83 (12)
Uncap the
0.60 (10)
0.70 (10)
0.80 (10)
ALOHA
Use the knife to cut the sushi
0.40
0.30 (10)
0.60 (10)
Obstaclc avoidance
0.95 (2)
0.30 (2)
0,95
LoCoBot
Comering
0.95 (2)
0.85 (2)
0.95 (2)
Sharp comering
0.85 (2)
0.30 (2)
0.88 (2)
Tello Quadcopter
Comering
0.68 (3)
0.68 (3)
0.82
Walking
N/A
Average
0.68 (116)
0.51 (116)
0.73 (116)
Table 3: Real Evaluation (detailed). We compare CrossFormer the same architecture trained on just the data from the target robot, as well as the best prior method on the data from the target robot. The number of trials parentheses. For ground and aerial navigation (LoCoBot and Tello Quadcopter) the success metric is the proportion of subgoals reached on the path towards the goal. For the Gol, we report reward achieved over 25 minutes normalized by the reward achieved by the RL-trained expert policy that generated the data. Because there is no suitable best prior imitation learning method for the Go1, we do include in the best prior method average.
over 25 minutes normalized by the reward achieved by the RL-trained expert policy that generated the data (see Section 3.1).
Tello Quadcopter
Finally, we perform evaluation on a Tello quadcopter using the navigation head of our policy. Since the navigation head outputs 2-D relative waypoints, we maintain a static height throughout the trajectory [42, 41]. Notably, we requires zero-shot generalization to new embodiment. We evaluated the skill of turning corner. We evaluated in locations, using trial per policy and location done in prior work
train on quadcopter data so this setting
[41,43] 42].
D Evaluation Results
In Table we provide a more detailed breakdown of our evaluation results by task.
17


Reviews:
[{"review_id": "jgZnzlIb9b", "reviewer": "Reviewer_MK5T", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper focuses on leveraging information collected by/for other robots to increase the performance of other robots and introduces a method called Crossformer. The method is evaluated using several different settings using several different robots with various morphologies against other state-of-the-art methods. The evaluations also include real-world testing.", "summary_of_recommendation": "I find the method and the motivation interesting. The evaluation is a bit weak, the method description is slightly lacking, and some parts of the model are not used in any of the evaluations (bimanual output).", "technical_quality": 5, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- Can you explain the language conditioning?\n - Can you provide details on how the model performance changes during different checkpoints in training?\n - About using the single arm action head for bi-manual evaluation, the most significant increase between using single robot data and using all data is seen in the ALOHA evaluation, Figure 3 shows that only 17 percent of the data is from bimanual robots (appendix shows that all of it is from ALOHA), since during the evaluation single arm head is used, and the remaining single-arm data is the largest portion of the data, could that be the reason in the significant increase in the evaluation success? Would we be seeing a similarly significant increase in performance if the bi-manual action head was used? Please explain.\n - One question I have is about the motivation. From what I understand, the motivation is to create a single model for all robot embodiments so that they can help each other’s performance. However, some are completely unrelated to each other and as the paper states, don’t help each other at all like quadruped and navigation. So, why use them together in the same model? As the limitations section states, as the model grows larger, the latency will increase and as we add more data from more embodiments to the data mix the model will have to grow to learn to them, so using them together can also be detrimental to model performance. Please explain.\n - The limitations section states “CrossFormer does not require any additional engineering to add data from new embodiments with different observation or action types, making scaling the training data straightforward.” How can you add the data of a humanoid robot?\n - In section 4.2, you state “We hypothesize that this is because we evaluate from the 3rd person camera view. While Yang et al. [8] train with both 3rd person and wrist camera views, only one view is input to the policy at a time, potentially resulting in underfitting to 3rd-person control.” How do the results change when the models are trained using only one view? Does the proposed model still outperform the baseline? You also state in section 4.2 “In comparison, our policy is not bound to a fixed observation space and can jointly train with 3rd-person views, a wrist-camera view, both, or neither”, yet in section limitations, it is stated that, “Another limitation is that our data mix uses hand-picked sampling weights to avoid over-training on datasets with many repetitive episodes and under training on the data most relevant to our evaluation settings.” so without the handpicked weights your model would be over trained too? Please explain\n - You state that you can alleviate this problem of hand-picked weights using model scaling. Yet, the models you used for comparison can potentially outperform your model with model scaling. What are the sizes of the models you used for baseline comparison methods?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tumM0Fb2jX", "reviewer": "Reviewer_xtGM", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper (1) builds on prior work on cross-embodiment in robot learning, such as the Open Cross Embodiment dataset, the use of readout token for action representation in Transformer-based sequence modeling, (2) proposes a tokenization scheme that covers both image and proprioceptive observations, (3) proposes an action token decoding scheme that covers navigation, manipulation, and quadruped locomotion, (4) includes data from additional robot embodiment types, and (5) demonstrates successfully that the overall scheme supports imitation learning of a single policy for an unprecedently diverse set of robot embodiments with competitive performance.", "summary_of_recommendation": "A very well written paper with solid technical contribution in a direction (cross-embodiment robot learning) that is timely. The real-robot demonstrations are compelling. The results are believable. It is likely going to stimulate follow-up work in this direction that is at least partly based on the work presented here.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "1. As of now, the information provided is inadequate for producing the work. For example, the specific values of the parameters in the tokenization scheme (L, M, N) and action token projection details etc are missing. Please consider including these in the main text, appendix, or open-sourced code (if code will be released).\n\n2. Please bring the Limitations sectoin into the main text. The discussion there is highly valuable. Also, in discussing limitations of the current approach, there is no need to reiterate the merit of CrossFormer (e.g. the sentence starting on Line 560). Yet another limitation that is worth discussing is the situation with zero padding: How pervasive is zero padding in observation tokenization? How could that negatively affect model efficiency or performance? Should we consider possible architectures that entirely do away with zero padding?\n\n3. Typo: Line 132 \"Figure 3\" should be \"Figure 2\". \n\n4. The bald claim on Lines 285-286: \"On average over all embodiments, CrossFormer has a 70% success rate while the best prior method has a 51.8% success rate\" should be unpacked. For example, given Figure 5, it seems that \"the best prior method\" scores 0 on Go1, which presumably contributed a lot to the 70% vs. 51.8% success rate gap. The situation here should be explained.\n\n5. More discussion on the conceptual underpinnings of the design choices could help, naturally without increasing the length of the paper.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K6iTWummy1", "reviewer": "Reviewer_Cs6U", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces CrossFormer, a transformer-based policy designed to manage diverse robotic tasks across various embodiments, such as single and dual arm manipulation, wheeled robots, quadcopters, and quadrupeds. It leverages a large and varied dataset (900K trajectories across 30 different robot embodiments) to achieve generalized control, without the need for manual alignment of observation or action spaces. The method aims to match the performance of specialized policies while outperforming previous state-of-the-art approaches in cross-embodiment learning.", "summary_of_recommendation": "The paper proposes a novel model architecture for a larger dataset. The paper mentions that their results are over one task in each embodiment. This is not a strong enough result and needs more analysis.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "For each robot, can you describe how the model was deployed on the real robot, for example, what hardware was used to fit a 110M parameter model onto each of these embodiments? Was it controlled via a laptop/cloud or was it deployed on the actual robot?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ppziTskLkp", "reviewer": "Reviewer_ZGgs", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces CrossFormer, a scalable and flexible transformer-based policy designed to control 30 different types of robots, each equipped with varying sensors, actuators, and control frequencies. The robots encompass single and dual-arm manipulation systems, wheeled robots, quadcopters, and quadrupeds. CrossFormer is a transformer-based policy that supports variable observations and actions by converting inputs and outputs into sequences. This approach allows for the training of robot data with any number of camera views or proprioceptive sensors by tokenizing and arranging the observations into a sequence. Although this cross-embodied imitation learning method can handle a wide range of embodiments, it does not achieve the highest performance compared to specialized policies. For instance, reinforcement learning (RL)-trained policies for locomotion outperform the trained policy of this work in terms of optimality, robustness, agility, etc.", "summary_of_recommendation": "The paper addresses one of the most significant challenges in robotics learning: developing a cross-embodiment and cross-task learning framework. It presents promising results both in hardware and simulation. Therefore, I recommend accepting the paper.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Please check the above comments for the questions.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a sound and interesting method for the challenging problem of integrating diverse robot (arms, legged robots, drones) sensorimotor spaces. The method outperforms the baselines, is evaluated with compelling real-robot experiments, and provides scalability. The paper can be improved by addressing the reviewers’ comments about the missing details in the methods and the experimental evaluations.

The problem is challenging and timely, the method is sound, the approach is verified in various systems, and the results are impressive. I recommend accepting the paper as oral presentation.
Decision: Accept