Scaling Cross-Embodied Learning: One Policy for Manipulation,
  Navigation, Locomotion and Aviation
Authors: name: Ria Doshi; name: Homer Walke; name: Oier Mees; name: Sudeep Dasari; name: Sergey Levine
Published: 2024-08-21 17:57:51

Abstract: Modern machine learning systems rely on large datasets to attain broad
generalization, and this often poses a challenge in robot learning, where each
robotic platform and task might have only a small dataset. By training a single
policy across many different kinds of robots, a robot learning method can
leverage much broader and more diverse datasets, which in turn can lead to
better generalization and robustness. However, training a single policy on
multi-robot data is challenging because robots can have widely varying sensors,
actuators, and control frequencies. We propose CrossFormer, a scalable and
flexible transformer-based policy that can consume data from any embodiment. We
train CrossFormer on the largest and most diverse dataset to date, 900K
trajectories across 20 different robot embodiments. We demonstrate that the
same network weights can control vastly different robots, including single and
dual arm manipulation systems, wheeled robots, quadcopters, and quadrupeds.
Unlike prior work, our model does not require manual alignment of the
observation or action spaces. Extensive experiments in the real world show that
our method matches the performance of specialist policies tailored for each
embodiment, while also significantly outperforming the prior state of the art
in cross-embodiment learning.