HomeRobot: Open-Vocabulary Mobile Manipulation
Authors: name: Sriram Yenamandra; name: Arun Ramachandran; name: Karmesh Yadav; name: Austin Wang; name: Mukul Khanna; name: Theophile Gervet; name: Tsung-Yen Yang; name: Vidhi Jain; name: Alexander William Clegg; name: John Turner; name: Zsolt Kira; name: Manolis Savva; name: Angel Chang; name: Devendra Singh Chaplot; name: Dhruv Batra; name: Roozbeh Mottaghi; name: Yonatan Bisk; name: Chris Paxton
Published: 2023-06-20 14:30:32

Abstract: HomeRobot (noun): An affordable compliant robot that navigates homes and
manipulates a wide range of objects in order to complete everyday tasks.
Open-Vocabulary Mobile Manipulation (OVMM) is the problem of picking any object
in any unseen environment, and placing it in a commanded location. This is a
foundational challenge for robots to be useful assistants in human
environments, because it involves tackling sub-problems from across robotics:
perception, language understanding, navigation, and manipulation are all
essential to OVMM. In addition, integration of the solutions to these
sub-problems poses its own substantial challenges. To drive research in this
area, we introduce the HomeRobot OVMM benchmark, where an agent navigates
household environments to grasp novel objects and place them on target
receptacles. HomeRobot has two components: a simulation component, which uses a
large and diverse curated object set in new, high-quality multi-room home
environments; and a real-world component, providing a software stack for the
low-cost Hello Robot Stretch to encourage replication of real-world experiments
across labs. We implement both reinforcement learning and heuristic
(model-based) baselines and show evidence of sim-to-real transfer. Our
baselines achieve a 20% success rate in the real world; our experiments
identify ways future research work improve performance. See videos on our
website: https://ovmm.github.io/.