Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviation

Authors: ria  doshi
homer  walke
oier  mees
sudeep  dasari
sergey  levine
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 13:25:34

Abstract: Modern machine learning systems rely on large datasets to attain broad generalization, and this often poses a challenge in robotic learning, where each robotic platform and task might have only a small dataset. By training a single policy across many different kinds of robots, a robotic learning method can leverage much broader and more diverse datasets, which in turn can lead to better generalization and robustness. However, training a single policy on multi-robot data is challenging because robots can have widely varying sensors, actuators, and control frequencies. We propose CrossFormer, a scalable and flexible transformer-based policy that can consume data from any embodiment. We train CrossFormer on the largest and most diverse dataset to date, 900K trajectories across 20 different robot embodiments. We demonstrate that the same network weights can control vastly different robots, including single and dual arm manipulation systems, wheeled robots, quadcopters, and quadrupeds. Unlike prior work, our model does not require manual alignment of the observation or action spaces. Extensive experiments in the real world show that our method matches the performance of specialist policies tailored for each embodiment, while also significantly outperforming the prior state of the art in cross-embodiment learning.

Main paper: 
 introduction: 
 Much of the recent success in machine learning has been driven by training general-purpose models on increasingly diverse and multi-task data. For example, visual and language tasks, once handled by task-specific methods, are now performed more effectively by general vision-language models that can transfer knowledge across tasks [1, 2, 3, 4] . Similarly, in robotics, recent data aggregation efforts [5] have made it possible to train general-purpose policies on robot data collected across multiple embodiments, tasks, and environments. These generalist policies can out-perform narrow policies trained with data from only the target robot and task by transferring visual representations and skills [6, 5] . In addition to the benefits of positive transfer, training generalist cross-embodied policies minimizes the amount of engineering that must be done to design and tune policy architectures for each robot.However, training a general-purpose robot policy is uniquely challenging, since robot systems can vary widely in their camera views, proprioceptive inputs, joint configurations, action outputs, and control frequencies. Initial efforts at training large-scale cross-embodied policies have often been limited to single robot arms or ground navigation robots which can be controlled with a single camera view and relative waypoint action for the base or end-effector [5, 6, 7, 8] . Further increasing the diversity of embodiments these policies can control requires a model architecture that supports conditioning on any number of camera views or proprioceptive observations, as well as predicting actions of any dimension. Following prior work, we take a sequential modeling approach to crossembodied imitation learning [9, 10] . We propose a transformer-based policy that supports variable observations and actions by casting inputs and outputs to a sequence. We scale this approach to control the most diverse set of embodiments with a single policy to date, including single and bimanual robot arms, ground navigation robots, quadcopters and quadrupeds.With our transformer policy, we can train on robot data with any number of camera views or proprioceptive sensors by simply tokenizing and arranging the observations into a sequence. At the same time, we can predict actions of any dimension, crucially without the need to manually align the action spaces of different embodiments [8] . For each action type, we insert a set of action readout tokens into the input token sequence. Then, we pass the corresponding output embeddings into action-space specific heads to produce a vector of the correct dimensions. Our policy can accept tasks in the form of language instructions or goal images, allowing a user to choose the task modality that is most natural for a given embodiment.Our primary contribution is a cross-embodied robot policy trained on the largest and most diverse robot dataset to date with 900K trajectories and 20 distinct embodiments. Our policy can control robots with varying observation and action types, from a quadruped with proprioceptive sensors and 12 joints to a bimanual robot with 3 cameras and 14 joints. We find in extensive real world experiments that our policy matches the performance of the same architecture trained on just the target robot data, as well as the best prior method in each setting, demonstrating that our architecture can absorb heterogeneous robot data without negative transfer, while performing comparably to state-of-the-art specialist methods tailored for each robot. Additionally, we find that our method outperforms the state of the art in cross-embodiment learning while alleviating the need for manually aligning observation and action spaces. 
related work: 
 Early work on cross-embodied robot policy learning has explored a number of techniques, including conditioning on explicit or learned representations of the embodiment, [11, 12] , domain randomization and adaptation [13, 14, 15, 16, 17, 18] , modular policies [19, 20, 21, 22] , or model-based RL [23, 24, 25] . Generally, these prior projects have operated on a smaller scale, only evaluating in simulation or training policies on small amounts of robot data to control a small number of robots for a few tasks.A number of prior works have sought to scale up robot learning by using large amounts of data from a single robot embodiment, collected either autonomously [26, 27, 28, 29, 30, 31] or with human teleoperation [32, 33, 34, 35, 36, 37, 38, 39] . Other prior works train on data from multiple robots, but require each robot to have the same observation and action space [7, 40, 5, 41, 42, 43, 44] . For example, Shah et al. [42] train one policy across many navigation robots using an egocentric camera view and 2D waypoint actions, and the RT-X models [5] are trained across single robotic arms using a 3rd person camera view and 7-DoF end-effector position actions. Our method does not require the data to have a common observation and action space, enabling simultaneous control of robots with disjoint sets of sensors and actuators, e.g., robot arms and quadrupeds.There have been a few large-scale efforts to train a single policy on robot data with varying observations and action spaces [10, 9, 6, 8] . Octo [6] can be fine-tuned on robots with observations and actions that are distinct from those seen during pre-training. However, Octo only pre-trains on data from single robot arms and does not explore co-training on more heterogeneous data. Reed et al. [9] and Bousmalis et al. [10] propose a flexible transformer-based policy that handles varying observation and action spaces during pre-training. They demonstrate their policy can control robot arms with different action spaces, including 4-DoF, 6-DoF, 7-DoF, and 14-DoF (using a 3-finger hand). We explore the challenges in increasing both the diversity of embodiments and environments a single policy can operate in. Along with increasing the number of arm embodiments we can control, e.g., high-frequency bimanual arms, we demonstrate navigation and quadrupedal locomotion, and we evaluate across a wide range of real-world settings, while they limit their evaluation to a standardized cage. We also explicitly evaluate for transfer across embodiments by comparing policies with and without cross-embodiment co-training, while their focus is on autonomous self-improvement. Perhaps most related to our work, Yang et al. [8] study transfer across manipulation and navigation data. However, their focus is on leveraging the fact that egocentric motion in navigation looks similar to egocentric motion from wrist cameras in manipulation, and they perform manual alignment of the actions across these two embodiments. Instead, our focus is on training a policy that can control embodiments beyond arms and ground navigation robots, including those with observations and action spaces that cannot be cast to a common format. Our method, CrossFormer, is the first to co-train on four distinct action spaces (single-arm manipulation, ground navigation, bimanual manipulation, and quadrupedal locomotion) without any observation space constraints or action-space alignment while maintaining state-of-the-art performance on each robot.Training generalist, cross-embodied policies requires multi-robot datasets, and there have been several efforts to collect such large-scale cross-embodied datasets [45, 41, 42, 5, 46] . In particular, the Open Cross-Embodiment dataset (OXE) [5] aggregates 1.5 million episodes of robot data, and we train on a subset of 900K trajectories. We note that the Octo [6] and RT-X [5] models are also trained on the OXE dataset, however they only use a subset with single robot arms. We additionally use the GNM [41] navigation and DROID [35] (large-scale Franka manipulation) datasets, along with Go1 quadruped and ALOHA [47] bimanual data collected by concurrent work [48] . 
designing a cross-embodied policy: 
 The primary challenge in robotic learning with many embodiments lies in handling widely varying observation and actions spaces, as well as differences in control frequency and other aspects of the robotic system. Robotic systems can have varying numbers of camera views or proprioceptive sensors, and they may be controlled by a variety of different action representations, including joint angles, Cartesian positions, and motor torques. In order to standardize the data to a common format, some prior work on training cross-embodied policies has ignored certain observation types (such as either wrist or third person views in manipulation) [5, 7] or aligned action spaces across robots [8] . Instead, following other prior work [9, 10, 6] , we cast cross-embodied imitation learning as a sequence-to-sequence problem and choose a transformer-based policy architecture that can handle varying length sequential inputs and outputs.Due to their sequential nature, a transformer policy enables encoding all available observation types from each embodiment by serializing them into a flat sequence. Similarly, we can decode variable length actions which allows us to use the optimal action type for each embodiment. Using this flexible output, we can also predict variably sized chunks of actions. Action chunking [49, 47, 50] improves the temporal consistency of actions and reduces compounding error which is particularly important for high-frequency fine manipulation. Together, a transformer backbone and action chunking enables our policy to control robots ranging from a bimanual ALOHA system with joint position control at 20Hz, to ground and aerial navigation robots with 2D waypoint control at 5Hz. At a high level, our transformer policy follows prior work that trains transformers on multi-modal data [9, 10, 6] . Observations and a task specification are tokenized by modality-specific tokenizers, assembled into a token sequence, and fed into a causal, decoder-only transformer backbone that is shared across all embodiments. The output embeddings are then fed into separate action heads for each class of embodiment to produce actions of the corresponding dimension, as shown in Figure 2 . Our training data mixture covers 20 different robot embodiments, varying widely in observation space, action space, and control frequency. We start from the singlearm manipulation subset of the Open Cross-Embodiment dataset used by Octo [6] . Then, we add the DROID Franka manipulation dataset [35] , 7K trajectories of ALOHA data collected across two institutions (referred to as ALOHA-multi-task), 60 hours of navigation from the GNM dataset [41] , 25 minutes of walking data from a Go1 quadruped (referred to as Go1-walk), and 200 trajectories of additional Franka data collected in our own lab (referred to as Franka-tabletop). We classify the datasets that are most relevant to each of our evaluation settings (see Section 4) as the target datasets and up-weight them relative to the other datasets during training. Our target datasets are BridgeData [36] for WidowX evaluation, ALOHA-multi-task for ALOHA evaluation, GNM [41] for navigation evaluation, Go1-walk for quadruped evaluation, and Franka-tabletop for Franka evaluation. We collected the Go1 data by rolling out an expert policy trained with RL in simulation [51] . See Figure 3 for our training data mix and Appendix A for further details on the composition of datasets collected ourselves along with the sampling weights.The first step in training our cross-embodied policy is creating the input sequence. The trajectories in our robot training data are sequences of time-steps where each time-step contains image observations I, proprioceptive observations P , and an action. Data from each embodiment may have a different number of camera views per time-step and may or may not include proprioceptive observations. To create the input sequence, we first define an observation history length k and chunk each trajectory into k-length segments, [I t , P t ..., I t+k , P t+k ]. We then tokenize each observation according to its modality. Images are processed with a ResNet-26 encoder [52] to produce a feature map that is flattened along the spatial dimensions and projected to the token embedding size. Proprioceptive observations are simply projected to the token embedding size. Along with the observation sequence, the policy also accepts a task specification. Importantly for cross-embodied control, our policy accepts task specifications either in the form of a language instruction l or a goal image g. In some settings, like navigation, tasks are more naturally specified as an image goal whereas in other settings, like manipulation, tasks are more easily specified with language. Language instructions are jointly processed with image observations using FiLM [53] . Goal images are stacked on the current image along the channel dimension before being fed into the image encoder.Because our training data consists of data from single-arm manipulators, dual-arm manipulators, quadrupeds, and ground navigation robots, our policy supports conditioning on any subset of the following observation types: (1) Workspace Image: A 3rd person camera view in manipulation settings.(2) Navigation Image: An egocentric camera view in navigation settings.(3) Wrist Image:A view from a wrist-mounted camera in manipulation settings. (4) Quadruped Proprioception: Joint positions and velocity estimates for quadrupeds. (5) Bimanual Proprioception: Joint positions for bimanual manipulation settings. To maximize transfer across embodiments, we share image encoder weights for camera views of the same type. So, for example, the workspace images in single arm and bimanual manipulation settings are processed by the same ResNet image encoder. In total we use four image encoders: one for the workspace view in manipulation settings, one for the egocentric view from ground navigation robots, and two for wrist cameras in manipulation settings. After input tokenization, we have a sequence of observation tokens [I 1:L t , P 1:M t ..., I 1:L t+k , P 1:M t+k ], where L and M denote the number of tokens for image and proprioceptive observations.After creating the input sequence, the next step is to process the input sequence with a transformer to predict an action of an appropriate dimension for each embodiment. We use a transformer with a block-wise causal attention mask such that observation tokens can only attend to observation tokens at the same or prior time-steps t. Following prior work [6] , we insert special readout tokens R after the observation tokens at each time-step in the input token sequence. These readout tokens can only attend to prior observation tokens, and thus serve as a convenient representation from which to predict actions. The final input sequence is[I 1:L t , P 1:M t , R 1:N t ..., I 1:L t+k , P 1:M t+k , R 1:N t+k ]where N denotes the number of readout tokens. We pass the input token sequence through the transformer to obtain an embedding sequence. We then apply an action head to the embeddings corresponding to the readout tokens to produce actions. There are several possibilities for the action head. Past work has explored regression with an L1 or L2 loss, classification with a cross-entropy loss, or diffusion. We choose to predict continuous actions and employ an L1 loss because of its success in prior work on high-frequency bimanual manipulation [47] . Accordingly, our action heads simply project the readout token embeddings to the action dimension. For some embodiments, we predict a chunk of sequential actions. Action chunking has been shown to improve policy performance in prior work [47, 50] and is essential for embodiments with high control frequencies where compounding error would build up too quickly. Since our action heads project the readout tokens to the action dimension size, we match the number of readout tokens to the action chunk size for each embodiment. Our policy has 4 action heads which produce chunked actions of the following types: (1) Single arm Cartesian positions: A 7-dimensional action denoting the relative change in Cartesian position of the end-effector and the gripper actuation. We predict a chunk of 4 actions and execute on single robot arms at 5-15Hz [6] . (2) Navigation waypoints: A 2-dimensional action denoting a waypoint relative to the robot's current position. We predict a chunk of 4 actions and execute on navigation robots at 4Hz [42] . (3) Bimanual joint positions: A 14-dimensional action denoting the joint positions of both arms. We predict a chunk of 100 actions and execute on bimanual robot arms at 20Hz [47] . (4) Quadruped joint positions: A 12-dimensional action, predicted with no chunking, denoting the joint positions of the legs. We predict only 1 action and execute on a quadruped at 20Hz [51] . Action chunk sizes are taken from prior work in each robot setting.In practice, we mask observations that are missing for an embodiment, so that each batch element contains all observation types and all readout token groups, and these token groups occupy fixed locations within the context window. Alternatively, for memory efficiency, observation and readout tokens could be densely packed to remove padding and fit more time-steps of context for embodiments with fewer observation types. This is a strategy used by prior work [9] . However, by not fixing observation and readout token types to a set location in the context window, the model would need to infer the embodiment purely from the observations in order to predict actions of the correct type (rather than relying on the positional embeddings of the readout tokens). The observations for some embodiments can look similar (such as navigation and manipulation with only a wrist camera), so this design may require appending a prefix to the token sequence indicating the embodiment.Our transformer backbone has 12 layers, 8 attention heads, an MLP dimension of 2048, and a token embedding size of 512. In total, our model has 130M parameters. We initialize the ResNet-26 encoders with ImageNet pre-trained weights. We use a context widow size of 2135 tokens, which fits 5 time-steps of context with all observation and readout token groups present. We found that good performance on navigation required 5 time-steps of observation history and using this context length did not hurt performance for other embodiments. We trained for 300K gradient steps with a batch size of 512 which took 47 hours on a TPU V5e-256 pod. We use the AdamW optimizer [54] , an inverse square root decay learning rate schedule [55] , weight decay of 0.1, and gradient clipping of 1.0. We apply standard image augmentations. During training, we use hindsight goal relabeling and sample future observations uniformly at random to use as goals [56] . If a language instruction is available for a trajectory, we randomly mask either the language or goal so that at test time we can condition our policy using either task specification [57, 58] . 
evaluation: 
 Our experiments are designed to test whether our single cross-embodied policy can control multiple robot embodiments without sacrificing performance when compared to robot-specific imitation learning approaches. Specifically, we aim to answer the following questions: (1) Can our policy trained on many robots match the performance of a policy trained on data from only the target robot? (2) Does our policy match the performance of the best prior imitation learning method in each robot setting? (3) How does our policy compare to prior cross-embodied policies that align observation and action spaces? Evaluation setups. We evaluate over a wide range of tasks and embodiments (see Fig. 4 ): (1) WidowX Manipulation: We use the Bridge setup from Walke et al. [36] . We use an over-theshoulder camera view and and sample actions from the single arm head of our policy. We perform 48 trials per policy over two language-conditioned tasks and two goal-conditioned tasks. (2) Franka Manipulation: We use the DROID setup from Khazatsky et al. [35] . We use an over-the-shoulder camera view and sample actions from the single arm head of our policy. We perform 39 trials per policy over two language-conditioned tasks. (3) ALOHA Bimanual Manipulation: We use the ALOHA setup from Zhao et al. [47] . We use 3 camera views, one overhead and two wrist, and sample actions from the bimanual head of our policy. We perform 20 trials per policy over two language-conditioned tasks. (4) LoCoBot Navigation: We use the LoCoBot setup from Shah et al. [42] which has one camera view. We evaluate on suite of three skills: path-following, obstacle avoidance, and sharp corner-turning. We collect a topological map of goals similar to Shah et al. [42] and evaluate success based on distance to the closest node within the topological map of goal images. We sample actions from the navigation head of our policy. We evaluate in 6 locations, using one trial per location and policy as done in prior work [41, 42] . (5) Go1 Quadruped: We evaluate on a Unitree Go1 which uses proprioceptive observations o t ∈ R 59 . We sample actions from the quadruped head of our policy. As our evaluation metric we report the average reward achieved over 25 minutes normalized by the the reward achieved by the RL-trained expert policy that generated the data (see Section 3.1). (6) Tello Quadcopter: Finally, we experiment with a Tello quadcopter using the navigation head of our policy. Since the navigation head outputs 2-D relative waypoints, we maintain a static height throughout the trajectory [42, 41] . Notably, we do not train on quadcopter We compare CrossFormer to the same architecture trained on just the data from the target robot, as well as the best prior method on the data from the target robot.data so this setting requires zero-shot generalization to a new embodiment (though not to a new set of observation inputs and action outputs). We evaluate in 3 locations, using one trial per location and policy as done in prior work [41, 42] .Baselines and comparison methods. To evaluate how co-training on data from other embodiments affects performance for a given embodiment, we train our policy architecture on just the data from the target embodiment using the target dataset (see Section 3.1) for each evaluation setting. Because these datasets are smaller than our full data mix, we create smaller versions of our architecture, ranging from 5M to 95M parameters, to avoid overfitting. We refer to this method as the single-robot dataset baseline. We also compare our method to the best prior method in each setting to evaluate whether we can match the performance of an approach that was specifically tuned for each robot. For the WidowX we use Octo [6] , for the Franka we use OpenVLA [59] , for ALOHA we use ACT [47] , for the LoCoBot and Tello quadcopter we use ViNT [42] . We do not compare to a prior state-of-the-art method for the Go1 setting due to the lack of a corresponding imitation learning method for quadrupedal locomotion. Finally, in the navigation and manipulation settings, we compare to the policy from Yang et al. [8] , by using the original 186M parameter policy. Yang et al. [8] leverage the fact that the egocentric view in navigation looks similar to wrist camera views in manipulation, and they align the action spaces of these two settings accordingly. This comparison evaluates whether our approach can perform similarly with no manual alignment of action spaces. 4.1 Can CrossFormer match the performance of training on only a single-robot dataset, as well as the best prior method in each setting? Figure 5 shows the performance of our method compared to our architecture trained on just the single-embodiment target datasets for each setting (see Section 3.1) and the best prior method for each setting. Overall, we find that our method performs comparably to training on just the target dataset in all evaluations, demonstrating that our architecture can absorb data from widely varying embodiments with no negative transfer. On average over all embodiments, CrossFormer has a 73% success rate, while the single-robot dataset baseline has a 67% success rate. Additionally, Cross-Former performs similarly to the best prior imitation learning method in each evaluation setting, demonstrating that our policy architecture can match state-of-the-art methods that have been specifically developed and tuned for each robot. On average over all embodiments, CrossFormer has a 73% success rate, while the best prior method has a 51% success rate. Because there is no suitable prior imitation learning method for the Go1, it is not included in the reported best prior method average. 4.2 Is aligning the observation and action spaces really needed? Lastly, we compare our policy to Yang et al. [8] in the LoCoBot navigation and WidowX manipulation settings. In the navigation setting, our policy significantly outperforms Yang et al. [8] in all tested scenarios including turning corners, avoiding obstacles, and making sharp turns. Qualitatively, we also found our policy to be much smoother, navigating in straighter lines with fewer stops and starts. Both policies are trained on the same amount of LoCoBot data (from the GNM dataset), [8] . We compare CrossFormer to Yang et al. [8] , which aligns actions for navigation and manipulation and only uses a single camera view at a time. CrossFormer outperforms Yang et al. [8] by 3x overall, on both tabletop manipulation from a third person camera view as well as common navigation tasks. so this result indicates that our policy architecture can better fit its cross-embodiment training data. Surprisingly, in the WidowX manipulation setting, we found that Yang et al. [8] obtained a zero success rate, despite training on the same relevant WidowX data as CrossFormer (i.e the Bridge dataset) and additional WidowX data collected in their own lab. We hypothesize that this is because we evaluate from the 3rd-person camera view. While Yang et al. [8] train with both 3rd-person and wrist camera views, only one view is input to the policy at a time, potentially resulting in underfitting to 3rd-person control. CrossFormer only uses the 3rd-person view for the WidowX setting due to a lack of publicly available WidowX wrist-camera data. However, we note that our policy can accept a varied number of camera views (using up to three simultaneously for bimanual manipulation) and does not require forcing two camera views into the same input slot. In summary, our policy outperforms Yang et al. [8] in both settings, demonstrating that manual alignment of observation and action spaces is not necessary for good performance, and that our flexible architecture allows our policy to better fit heterogeneous cross-embodied data. 
discussion and conclusion: 
 We introduced CrossFormer, a scalable and flexible transformer policy trained on the largest and most diverse dataset to date, 900K trajectories across 20 different robot embodiments. We demonstrated a principled way to learn a single policy that can control vastly different embodiments including single and dual arm manipulation systems, wheeled robots, quadcopters, and quadrupeds. Our results show that CrossFormer matches the performance of specialist policies tailored for individual embodiments, while also significantly outperforming the state of the art in cross-embodiment learning. However, our work does have limitations. Our results do not yet show significant positive transfer across embodiments. We anticipate that as we train on larger robot datasets with more embodiments, we will see greater positive transfer. Another limitation is that our data mix uses hand-picked sampling weights to avoid over-training on datasets with many repetitive episodes and under-training on the data most relevant to our evaluation settings. In principle, as we scale model size, the policy should have the capacity to fit all the data equally well without any data weighting. Finally, given that we need large models to fit large multi-robot datasets, the model's inference speed can become a limiting factor. In this work we successfully applied our policy to a highfrequency, fine-grained bimanual manipulation task, but as we scale the model's size we may not be able to control these higher frequency embodiments. Future hardware improvements will help to alleviate the issue, but further research is needed on techniques for using large models to control high-frequency robots. Future work could also include exploring techniques to enable greater positive transfer across embodiments while maintaining the flexibility of our architecture, techniques for data curation, and incorporating even more diverse data sources like sub-optimal robot data or action-free human video. We hope that this work will open the door for more general-purpose and flexible robot policies that can efficiently learn and transfer knowledge from the experience collected across diverse robot embodiments.More information and videos can be found on our website: crossformer-model.github.io. 
 
 Appendix: We list the sampling weights for each dataset in our data mixture in Table 1 . We up-weight our target datasets: Bridge, ALOHA-multi-task, GNM, Go1-walk, and Franka-tabletop.Fractal [32] 17% Kuka [28] 2.2% BC-Z [60] 2.2% Stanford Hydra Dataset [61] 0.015% Language Table [62] 1.5% Taco Play [63, 64] 1.2% Furniture Bench Dataset [65] 0.83% UTAustin Mutex [66] 0.76% Austin Sailor Dataset [67] 0.74% Roboturk [68] 0.79% Toto [69] 0.68% Austin Sirius Dataset [70] 0.59% Berkeley Autolab UR5 [71] 0.41% IAMLab CMU Pickup Insert [72] 0.31% Viola [73] 0.32% Berkeley Fanuc Manipulation [74] 0.26% NYU Franka Play Dataset [75] 0.28% Jaco Play [76] 1.6% Berkeley Cable Routing [77] 0.089% Austin Buds Dataset [78] 0.072% CMU Stretch [79] 0.053% DLR EDAN Shared Control [80] 0.019% DROID [35] 0.022% Bridge [37, 36] 17% GNM [41] 17% ALOHA-multi-task 17% Go1-walk 8.5% Franka-tabletop 8.5%Table 1 : The CrossFormer training data mixture uses datasets from the Open X-Embodiment dataset [5] and additional data collected for this project.In Table 2 we list the hyperparameters for the optimizer and policy architecture. In total, along with the ResNet-26 image encoders and action heads, our model has 130M parameters. We initialize the ResNet-26 encoders with ImageNet pre-trained weights. Training took 80 hours on a TPU V5e-256 pod. We apply color jitter and random resizing/cropping image augmentations. During training, we use hindsight goal relabeling and sample future observations uniformly at random to use as goals [56] . If a language instruction is available for a trajectory, we randomly mask either the language or goal so that at test time we can condition our policy using either task specification [57] . For our method and baselines we trained, we selected checkpoints based on the validation mean squared error since we found this very roughly correlates with robot performance.Below we provide further details on our evaluation settings (see Fig. 4 for images). For goalcondtioned tasks, we arrange the scene and take the goal image before rolling out the policy. For the language-conditioned tasks, we use language instructions that are semantically in the distribution of the training data (though may not appear exactly in the training data). For the manipulation settings, we ran the models on a machine with an NVIDIA 4090 (or similar GPU) that was connected to the robot. For the ground navigation and drone experiments, we ran the models on a separate server and sent actions over a wireless network.WidowX Manipulation We use the Bridge setup from Walke et al. [36] . We use an over-theshoulder camera view and and sample actions from the single arm head of our policy. We evaluated two language-conditioned tasks, putting a spoon on a cloth and putting a carrot on a plate, and two goal-conditioned tasks, putting a mushroom in a pot and putting a cloth on a saucer. We used 12 trials per policy and task. Positions of the objects were varied between trials.Franka Manipulation We use the DROID setup from Khazatsky et al. [35] . We use an over-theshoulder camera view and sample actions from the single arm head of our policy. We evaluated for 27 trials per policy on the language-conditioned task of using a sponge to sweep pinecones into a dustpan. We also evaluated for 12 trials per policy on the task of flipping a pot upright. The positions of the objects were varied between trials.We use the ALOHA setup from Zhao et al. [47] . We use three camera views, one overhead and two wrist, and sample actions from the bimanual head of our policy. We evaluated two language-conditioned tasks, taking the cap off of a pen and picking up a knife and using it to cut a piece of sushi. We used 10 trials per policy and task. The position of the pen, knife, and sushi was varied between trials.LoCoBot Navigation We use the LoCoBot setup from Shah et al. [42] which has one camera view. We evaluate on suite of three skills: path-following, obstacle avoidance, and sharp cornerturning. We sample actions from the navigation head of our policy. We combine our policy with the graph-based planner and distance function from Shah et al. [42] . We first obtain a topological map M of the environment by teleoperating the robot. Then, at every time-step we find the closest node in M, search the graph for the shortest path from this node to the goal, and command the policy with the most immediate subgoal in the path. We evaluate the success of a trajectory based on the number of subgoals between the closest node at the end of the trajectory and the goal. We evaluate in 6 locations, using one trial per location and policy as done in prior work [41, 42] .Go1 Quadruped We evaluate on a Unitree Go1 which uses proprioceptive observations o t ∈ R 59 . We sample actions from the quadruped head of our policy, and the task is to walk forward. Importantly, unlike prior work [8] , we directly control the quadruped's joints rather than doing higher level control with navigation waypoints. As our evaluation metric we report reward achieved 


Decision: The paper presents a sound and interesting method for the challenging problem of integrating diverse robot (arms, legged robots, drones) sensorimotor spaces. The method outperforms the baselines, is evaluated with compelling real-robot experiments, and provides scalability. The paper can be improved by addressing the reviewers’ comments about the missing details in the methods and the experimental evaluations.

The problem is challenging and timely, the method is sound, the approach is verified in various systems, and the results are impressive. I recommend accepting the paper as oral presentation.

Reviews: review: title: title: Review on Crossformer
paper_summary: paper_summary: The paper focuses on leveraging information collected by/for other robots to increase the performance of other robots and introduces a method called Crossformer. The method is evaluated using several different settings using several different robots with various morphologies against other state-of-the-art methods. The evaluations also include real-world testing.
main_review: main_review: Strength:

With the introduction of large robotic datasets such as OXE, it became vital for the robot learning field to develop Large Models that can leverage these large datasets. The paper introduces a model that tackles this, and this is why I believe this paper is important. Furthermore, according to the results of the evaluations the method’s performance increases (or does not drop) when additional data from other embodiments are used during training. The model also significantly outperforms the baseline (Yang et al. [8]) on navigation and manipulation tasks.

Weakness:
 - The limitations section should not be in the supplementary.
 - There are some design decisions in the paper that are not justified properly like the chunk sizes for different actions.
 - Figure 5 includes the comparison to baseline methods and includes both single robot dataset and mixed dataset results and that result is named crossformer. In Figure 6, there is only crossformer is used in the legends. While it is explained in section 4.2, it creates confusion. The fact that the text does not state in 4.2 which dataset is used to train the models for WidowX comparison adds to the confusion.
 - The number of trials in the experiments seems a little too low.
 - How the language conditioning part of the system works or how it is utilized during evaluations is not properly explained.
 - The manuscript explains the training details comprehensively. Yet, it only states that the training took 900K steps. I would like to see how the model performance changes as the training progresses. How do you deal with overfit/underfit?
 - The evaluations of bi-manual manipulation use the single-arm action head which is confusing considering there is an action head in the model for bi-manual control. 
 - I don’t think using 2D navigation for Tello and stating it as evidence of zero-shot generalization to a new embodiment is appropriate. With the same logic, since it uses cartesian position change of end effector, the single-arm head can be used for generalization to almost any single-arm manipulator which can all be considered as new embodiments. 
 - From what the paper states, the performance of the model for a robot type increases if there is useful information in the data of the other robot types like wrist camera information of single-arm robots being useful for navigation and vice versa. However, based on the given results, this is a bit speculative. I suggest you include an additional analysis where the data of the others are added to the data mix gradually so that we can see adding the data of which robot to the mix increases the performance of which robot.
questions: questions: - Can you explain the language conditioning?
 - Can you provide details on how the model performance changes during different checkpoints in training?
 - About using the single arm action head for bi-manual evaluation, the most significant increase between using single robot data and using all data is seen in the ALOHA evaluation, Figure 3 shows that only 17 percent of the data is from bimanual robots (appendix shows that all of it is from ALOHA), since during the evaluation single arm head is used, and the remaining single-arm data is the largest portion of the data, could that be the reason in the significant increase in the evaluation success? Would we be seeing a similarly significant increase in performance if the bi-manual action head was used? Please explain.
 - One question I have is about the motivation. From what I understand, the motivation is to create a single model for all robot embodiments so that they can help each other’s performance. However, some are completely unrelated to each other and as the paper states, don’t help each other at all like quadruped and navigation. So, why use them together in the same model? As the limitations section states, as the model grows larger, the latency will increase and as we add more data from more embodiments to the data mix the model will have to grow to learn to them, so using them together can also be detrimental to model performance. Please explain.
 - The limitations section states “CrossFormer does not require any additional engineering to add data from new embodiments with different observation or action types, making scaling the training data straightforward.” How can you add the data of a humanoid robot?
 - In section 4.2, you state “We hypothesize that this is because we evaluate from the 3rd person camera view. While Yang et al. [8] train with both 3rd person and wrist camera views, only one view is input to the policy at a time, potentially resulting in underfitting to 3rd-person control.” How do the results change when the models are trained using only one view? Does the proposed model still outperform the baseline? You also state in section 4.2 “In comparison, our policy is not bound to a fixed observation space and can jointly train with 3rd-person views, a wrist-camera view, both, or neither”, yet in section limitations, it is stated that, “Another limitation is that our data mix uses hand-picked sampling weights to avoid over-training on datasets with many repetitive episodes and under training on the data most relevant to our evaluation settings.” so without the handpicked weights your model would be over trained too? Please explain
 - You state that you can alleviate this problem of hand-picked weights using model scaling. Yet, the models you used for comparison can potentially outperform your model with model scaling. What are the sizes of the models you used for baseline comparison methods?
limitations: limitations: 3
review_summary: review_summary: I find the method and the motivation interesting. The evaluation is a bit weak, the method description is slightly lacking, and some parts of the model are not used in any of the evaluations (bimanual output).
title: title: A good, well-written paper presenting a solid step forward towards an effective cross-embodiment learning architecture for robotics based on the Transformer architecture.
paper_summary: paper_summary: The paper (1) builds on prior work on cross-embodiment in robot learning, such as the Open Cross Embodiment dataset, the use of readout token for action representation in Transformer-based sequence modeling, (2) proposes a tokenization scheme that covers both image and proprioceptive observations, (3) proposes an action token decoding scheme that covers navigation, manipulation, and quadruped locomotion, (4) includes data from additional robot embodiment types, and (5) demonstrates successfully that the overall scheme supports imitation learning of a single policy for an unprecedently diverse set of robot embodiments with competitive performance.
main_review: main_review: The paper presents an architecture called CrossFormer that is capable of integrating diverse robot body types and observation and action spaces. The paper is very well written. Overall, it address the following question: how to reduce the need for hand-designed mappings of observations to and actions from a cross-embodiment policy model such that what remains is justifiably minimal. Given the choice of a Transformer architecture for implementing the policy model, that question becomes how to map into observation tokens and from action tokens. The proposed methods (building on existing work) were shown to cover navigation, dexterous manipulation and quadruped locomotion with robot body types ranging from arms to legged robots to drones. While the specific technical design still leaves much room for further improvement (e.g. positive transfer across body types is yet to be demonstrated; see also the Limitations section and Questions for Rebuttal below), the technical quality is high and the contribution is timely. The empirical evaluation on real robots is compelling. It is likely going to stimulate follow-up work in this direction that is at least partly based on the work presented here.
questions: questions: 1. As of now, the information provided is inadequate for producing the work. For example, the specific values of the parameters in the tokenization scheme (L, M, N) and action token projection details etc are missing. Please consider including these in the main text, appendix, or open-sourced code (if code will be released).

2. Please bring the Limitations sectoin into the main text. The discussion there is highly valuable. Also, in discussing limitations of the current approach, there is no need to reiterate the merit of CrossFormer (e.g. the sentence starting on Line 560). Yet another limitation that is worth discussing is the situation with zero padding: How pervasive is zero padding in observation tokenization? How could that negatively affect model efficiency or performance? Should we consider possible architectures that entirely do away with zero padding?

3. Typo: Line 132 "Figure 3" should be "Figure 2". 

4. The bald claim on Lines 285-286: "On average over all embodiments, CrossFormer has a 70% success rate while the best prior method has a 51.8% success rate" should be unpacked. For example, given Figure 5, it seems that "the best prior method" scores 0 on Go1, which presumably contributed a lot to the 70% vs. 51.8% success rate gap. The situation here should be explained.

5. More discussion on the conceptual underpinnings of the design choices could help, naturally without increasing the length of the paper.
limitations: limitations: 1
review_summary: review_summary: A very well written paper with solid technical contribution in a direction (cross-embodiment robot learning) that is timely. The real-robot demonstrations are compelling. The results are believable. It is likely going to stimulate follow-up work in this direction that is at least partly based on the work presented here.
title: title: A novel flexible-observation policy representation that enables training on a larger cross-embodiment dataset.
paper_summary: paper_summary: The paper introduces CrossFormer, a transformer-based policy designed to manage diverse robotic tasks across various embodiments, such as single and dual arm manipulation, wheeled robots, quadcopters, and quadrupeds. It leverages a large and varied dataset (900K trajectories across 30 different robot embodiments) to achieve generalized control, without the need for manual alignment of observation or action spaces. The method aims to match the performance of specialized policies while outperforming previous state-of-the-art approaches in cross-embodiment learning.
main_review: main_review: The paper is inspired by Yang et al.[1] work on cross-embodiment learning for Manipulation and Navigation. The use of a substantial dataset that spans multiple embodiments and tasks is a significant strength. By avoiding the need for manual alignment of observation and action spaces, the method simplifies the training process and reduces the potential for human error.

There is a novelty in the presented work, but the results are lacking and need much more analysis and details.

Criticism:
The evaluation tasks lack sufficient description in manipulation. The choice of using multiple trials on just “a language conditioned” and “a goal-conditioned” (line 245) task is questionable. The website consists of a single task per embodiment. If this is a typo and you do have multiple tasks, each with a different language/goal condition, then please correct it. Further, these tasks need more description explaining what they are and each task needs to have multiple trials.

Analysis in different settings for navigation. Yang et al provided a detailed analysis of their method in different settings/locations. This detail is missing in the author's results, making it weaker.
 
Although the paper does analyze the performance of their architecture trained on single policies, an ablation of different combinations of the data mixture would be extremely helpful (as was done in Yang et al). For example, is the quadruped data inconsequential in the presence of navigation data?

The authors hypothesize their mode works better on navigation than the baseline because their method involves training with a flexible observation space. This hypothesis can be tested with another training run where the policy is just shown one of the camera views during training. Please add this ablation result as well.

An ablation of the model capacity would be helpful as well.


References:
[1]: Yang, Jonathan, et al. "Pushing the limits of cross-embodiment learning for manipulation and navigation." arXiv preprint arXiv:2402.19432 (2024).
questions: questions: For each robot, can you describe how the model was deployed on the real robot, for example, what hardware was used to fit a 110M parameter model onto each of these embodiments? Was it controlled via a laptop/cloud or was it deployed on the actual robot?
limitations: limitations: 3
review_summary: review_summary: The paper proposes a novel model architecture for a larger dataset. The paper mentions that their results are over one task in each embodiment. This is not a strong enough result and needs more analysis.
title: title: Learning a single policy for manipulation, locomotion and aviation
paper_summary: paper_summary: This paper introduces CrossFormer, a scalable and flexible transformer-based policy designed to control 30 different types of robots, each equipped with varying sensors, actuators, and control frequencies. The robots encompass single and dual-arm manipulation systems, wheeled robots, quadcopters, and quadrupeds. CrossFormer is a transformer-based policy that supports variable observations and actions by converting inputs and outputs into sequences. This approach allows for the training of robot data with any number of camera views or proprioceptive sensors by tokenizing and arranging the observations into a sequence. Although this cross-embodied imitation learning method can handle a wide range of embodiments, it does not achieve the highest performance compared to specialized policies. For instance, reinforcement learning (RL)-trained policies for locomotion outperform the trained policy of this work in terms of optimality, robustness, agility, etc.
main_review: main_review: This paper introduces CrossFormer, a scalable and flexible transformer-based policy designed to control 30 different types of robots, each equipped with varying sensors, actuators, and control frequencies. The robots encompass single and dual-arm manipulation systems, wheeled robots, quadcopters, and quadrupeds. CrossFormer is a transformer-based policy that supports variable observations and actions by converting inputs and outputs into sequences. This approach allows for the training of robot data with any number of camera views or proprioceptive sensors by tokenizing and arranging the observations into a sequence. Although this cross-embodied imitation learning method can handle a wide range of embodiments, it does not achieve the highest performance compared to specialized policies. For instance, reinforcement learning (RL)-trained policies for locomotion outperform the trained policy of this work in terms of optimality, robustness, agility, etc.

Major comments:
- The proposed method excels in cross-embodiment learning, outperforming the state-of-the-art in this area, which is commendable. However, it still falls short compared to specialized policies in certain cases. This raises an important question: should we aim for a cross-embodiment and cross-task policy using a single policy, or should we develop a unique policy for each task and embodiment? For example, the locomotion performance of this work should be compared with the following study that focuses on cross-embodiments with only the locomotion task, which seems to exhibit much better locomotion behavior than the A1 robot proposed in this paper:

Shafiee, Milad, Guillaume Bellegarda, and Auke Ijspeert. "Manyquadrupeds: Learning a single locomotion policy for diverse quadruped robots." arXiv preprint arXiv:2310.10486 (2023).

- Additionally, the sim-to-real transfer of quadruped robots requires attention. The current simulation for locomotion needs improvement as it exhibits unnatural behavior, with legs making unnecessary contact with the ground during normal locomotion tasks, which have already been solved.

- Another consideration is the energy efficiency of tasks. If the collected data is not efficient and optimal, how can the method address and improve it?

- The potential of cross-embodiment policies is significant for researchers aiming to control and train their own embodiments. Do you plan to open-source your training framework? This would substantially increase the impact of your current work.

- A frequently asked question is, if we already have data from a working robot, meaning the task is solved, why should we use this data again to train a generalist policy?

- "Does our policy match the performance of the best prior imitation learning method in each robot setting?" This is an interesting question addressed in the paper. Another crucial question is whether the proposed policy matches the performance of the best prior reinforcement learning method in each robot setting.

- One limitation that should be addressed is that the current framework can only learn tasks that have already been solved. How will your future work address unseen tasks that have not yet been solved? This discussion should be included in the limitations section.

Minor comments:
- Refer to Figure 3 for an overview of our architecture --> should be Figure 2.
- Figure 3: Training Data,  has not been cited in the text of the paper.
questions: questions: Please check the above comments for the questions.
limitations: limitations: 2
review_summary: review_summary: The paper addresses one of the most significant challenges in robotics learning: developing a cross-embodiment and cross-task learning framework. It presents promising results both in hardware and simulation. Therefore, I recommend accepting the paper.