Robot Utility Models: General Policies for Zero-Shot Deployment in New
  Environments
Authors: name: Haritheja Etukuru; name: Norihito Naka; name: Zijin Hu; name: Seungjae Lee; name: Julian Mehu; name: Aaron Edsinger; name: Chris Paxton; name: Soumith Chintala; name: Lerrel Pinto; name: Nur Muhammad Mahi Shafiullah
Published: 2024-09-09 17:59:50

Abstract: Robot models, particularly those trained with large amounts of data, have
recently shown a plethora of real-world manipulation and navigation
capabilities. Several independent efforts have shown that given sufficient
training data in an environment, robot policies can generalize to demonstrated
variations in that environment. However, needing to finetune robot models to
every new environment stands in stark contrast to models in language or vision
that can be deployed zero-shot for open-world problems. In this work, we
present Robot Utility Models (RUMs), a framework for training and deploying
zero-shot robot policies that can directly generalize to new environments
without any finetuning. To create RUMs efficiently, we develop new tools to
quickly collect data for mobile manipulation tasks, integrate such data into a
policy with multi-modal imitation learning, and deploy policies on-device on
Hello Robot Stretch, a cheap commodity robot, with an external mLLM verifier
for retrying. We train five such utility models for opening cabinet doors,
opening drawers, picking up napkins, picking up paper bags, and reorienting
fallen objects. Our system, on average, achieves 90% success rate in unseen,
novel environments interacting with unseen objects. Moreover, the utility
models can also succeed in different robot and camera set-ups with no further
data, training, or fine-tuning. Primary among our lessons are the importance of
training data over training algorithm and policy class, guidance about data
scaling, necessity for diverse yet high-quality demonstrations, and a recipe
for robot introspection and retrying to improve performance on individual
environments. Our code, data, models, hardware designs, as well as our
experiment and deployment videos are open sourced and can be found on our
project website: https://robotutilitymodels.com