Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots
Authors: name: Xavier Puig, name: Eric Undersander, name: Andrew Szot, name: Mikael Dallaire Cote, name: Tsung-Yen Yang, name: Ruslan Partsey, name: Ruta Desai, name: Alexander William Clegg, name: Michal Hlavac, name: So Yeon Min, name: Vladimír Vondruš, name: Theophile Gervet, name: Vincent-Pierre Berges, name: John M. Turner, name: Oleksandr Maksymets, name: Zsolt Kira, name: Mrinal Kalakrishnan, name: Jitendra Malik, name: Devendra Singh Chaplot, name: Unnat Jain, name: Dhruv Batra, name: Akshara Rai, name: Roozbeh Mottaghi

We present Habitat 3.0: a simulation platform for studying collaborative
human-robot tasks in home environments. Habitat 3.0 offers contributions across
three dimensions: (1) Accurate humanoid simulation: addressing challenges in
modeling complex deformable bodies and diversity in appearance and motion, all
while ensuring high simulation speed. (2) Human-in-the-loop infrastructure:
enabling real human interaction with simulated robots via mouse/keyboard or a
VR interface, facilitating evaluation of robot policies with human input. (3)
Collaborative tasks: studying two collaborative tasks, Social Navigation and
Social Rearrangement. Social Navigation investigates a robot's ability to
locate and follow humanoid avatars in unseen environments, whereas Social
Rearrangement addresses collaboration between a humanoid and robot while
rearranging a scene. These contributions allow us to study end-to-end learned
and heuristic baselines for human-robot collaboration in-depth, as well as
evaluate them with humans in the loop. Our experiments demonstrate that learned
robot policies lead to efficient task completion when collaborating with unseen
humanoid agents and human partners that might exhibit behaviors that the robot
has not seen before. Additionally, we observe emergent behaviors during
collaborative task execution, such as the robot yielding space when obstructing
a humanoid agent, thereby allowing the effective completion of the task by the
humanoid agent. Furthermore, our experiments using the human-in-the-loop tool
demonstrate that our automated evaluation with humanoids can provide an
indication of the relative ordering of different policies when evaluated with
real human collaborators. Habitat 3.0 unlocks interesting new features in
simulators for Embodied AI, and we hope it paves the way for a new frontier of
embodied human-AI interaction capabilities.