Habitat 2.0: Training Home Assistants to Rearrange their Habitat

1 Introduction . Consider a home assistant robot illustrated in Fig . 1 – a mobile manipulator ( Fetch [ 1 ] ) performing tasks like stocking groceries into the fridge , clearing the table and putting dishes into the dishwasher , ⇤Correspondence to : aszot3 @ gatech.edu 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . fetching objects on command and putting them back , etc . Developing such embodied intelligent systems is a goal of deep scientific and societal value . So how should we accomplish this goal ? Training and testing such robots in hardware directly is slow , expensive , and difficult to reproduce . We aim to advance the entire ‘ research stack ’ for developing such embodied agents in simulation – ( 1 ) data : curating house-scale interactive 3D assets ( e.g . kitchens with cabinets , drawers , fridges that can open/close ) that support studying generalization to unseen objects , receptacles , and home layouts , ( 2 ) simulation : developing the next generation of high-performance photo-realistic 3D simulators that support rich interactive environments , ( 3 ) tasks : setting up challenging representative benchmarks to enable reproducible comparisons and systematic tracking of progress over the years . To support this long-term research agenda , we present : • ReplicaCAD : an artist-authored fully-interactive recreation of ‘ FRL-apartment ’ spaces from the Replica dataset [ 2 ] consisting of 111 unique layouts of a single apartment background with 92 authored objects including dynamic parameters , semantic class and surface annotations , and efficient collision proxies , representing 900+ person-hours of professional 3D artist effort . ReplicaCAD ( illustrated in figures and videos ) was created with the consent of and compensation to artists , and will be shared under a Creative Commons license for non-commercial use with attribution ( CC-BY-NC ) . • Habitat 2.0 ( H2.0 ) : a high-performance physics-enabled 3D simulator , representing approximately 2 years of development effort and the next generation of the Habitat project [ 3 ] ( Habitat 1 . 0 ) . H2.0 supports piecewise-rigid objects ( e.g . door , cabinets , and drawers that can rotate about an axis or slide ) , articulated robots ( e.g . mobile manipulators like Fetch [ 1 ] , fixed-base arms like Franka [ 4 ] , quadrupeds like AlienGo [ 5 ] ) , and rigid-body mechanics ( kinematics and dynamics ) . The design philosophy of H2.0 is to prioritize performance ( or speed ) over the breadth of simulation capabilities . H2.0 by design and choice does not support non-rigid dynamics ( deformables , fluids , films , cloths , ropes ) , physical state transformations ( cutting , drilling , welding , melting ) , audio or tactile sensing – many of which are capabilities provided by other simulators [ 6–8 ] . The benefit of this focus is that we were able to design and optimize H2.0 to be exceedingly fast – simulating a Fetch robot interacting in ReplicaCAD scenes at 1200 steps per second ( SPS ) , where each ‘ step ’ involves rendering 1 RGBD observation ( 128⇥128 pixels ) and simulating rigid-body dynamics for 1/30 sec . Thus , 30 SPS would be considered ‘ real time ’ and 1200 SPS is 40⇥ real-time . H2.0 also scales well – achieving 8,200 SPS ( 273⇥ real-time ) multi-process on a single GPU and over 25,000 SPS ( 850⇥ real-time ) on a single node with 8 GPUs . For reference , existing simulators typically achieve 10-400 SPS ( see Tab . 1 ) . These 100⇥ simulation-speedups correspond to cutting experimentation time from 6 months to under 2 days , unlocking experiments that were hitherto infeasible , allowing us to answer questions that were hitherto unanswerable . As we will show , they also directly translate to training-time speed-up and accuracy improvements from training agents ( for object rearrangement tasks ) on more experience . • Home Assistant Benchmark ( HAB ) : a suite of common tasks for assistive robots ( TidyHouse , PrepareGroceries , SetTable ) that are specific instantiations of the generalized rearrangement problem [ 9 ] . Specifically , a mobile manipulator ( Fetch ) is asked to rearrange a list of objects from initial to desired positions – picking/placing objects from receptacles ( counter , sink , sofa , table ) , opening/closing containers ( drawers , fridges ) as necessary . We use the GeometricGoal specification prescribed by Batra et al . [ 9 ] – i.e. , initial and desired 3D ( center-of-mass ) position of each target object i to be rearranged s0i , s ⇤ i N i=1 . The choice of GeometricGoal is deliberate – we aim to create the PointNav [ 10 ] equivalent for mobile manipulators . As witnessed in the navigation literature , such a task becomes the testbed for exploring ideas [ 11–19 ] and a starting point for more semantic tasks [ 20–22 ] . The robot operates entirely from onboard sensing – head- and arm-mounted RGB-D cameras , proprioceptive joint-position sensors ( for the arm ) , and egomotion sensors ( for the mobile base ) – and may not access any privileged state information ( no prebuilt maps , no 3D models of rooms or objects , no physically-implausible sensors providing knowledge of mass , friction , articulation of containers , etc. ) . Notice that an object ’ s center-of-mass provides no information about its size or orientation . The target object may be located inside a container ( drawer , fridge ) , on top of supporting surfaces ( shelf , table , sofa ) of varying heights and sizes , and surrounded by clutter ; all of which must be sensed and maneuvered . Receptacles like drawers and fridges start closed , meaning that the agent must open and close articulated objects to succeed . An episode is considered successful if all target objects are placed within 15cm of their desired positions ( without considering orientation ) . The robot uses continuous end-effector control for the arm and velocity control for the base . We deliberately focus on gross motor control ( the base and arm ) and not fine motor control ( the gripper ) , following the ‘ abstracted grasping ’ recommendations from [ 9 ] . Specifically , once the end-effector reaches 15cm ( or closer ) to an object , a discrete grasp action becomes available that , if executed , snaps the object into its parallel-jaw gripper 2 . We conduct a systematic study of two distinct techniques – monolithic ‘ sensors-to-actions ’ policies trained with reinforcement learning ( RL ) at scale , and classical senseplan-act pipelines ( SPA ) [ 26 ] – with a particular emphasis on systematic generalization to new objects , receptacles , apartment layouts ( not just robot starting pose ) . Our findings include : 1 . Flat vs hierarchical : Monolithic RL policies successfully learn diverse individual skills ( pick/place , navigate , open/close drawer ) . However , crafting a combined reward function and learning scheme that elicits chaining of such skills for the long-horizon HAB tasks remained out of our reach . We saw significantly stronger results with a hierarchical approach that assumes knowledge of a perfect task planner ( via STRIPS [ 27 ] ) to break it down into a sequence of skills . 2 . Hierarchy cuts both ways : However , a hierarchy with independent skills suffers from ‘ hand-off problems ’ where a succeeding skill isn ’ t set up for success by the preceding one – e.g. , navigating to a bad location for subsequent manipulation , only partially opening a drawer to grab an object inside , or knocking an object out of reach that is later needed . 3 . Brittleness of SensePlanAct : For simple skills , SPA performs just as well as monolithic RL . However , it is significantly more brittle since it needs to map all obstacles in the workspace for planning . More complex settings involving clutter , challenging receptacles , and imperfect navigation can poorly frame the target object and obstacles in the robot ’ s camera , leading to incorrect plans . We hope our work will serve as a benchmark for many years to come . H2.0 is free , open-sourced under the MIT license , and under active development . 3 We believe it will reduce the community ’ s reliance on commercial lock-ins [ 28 , 29 ] and non-photorealistic simulation engines [ 30–32 ] . 2 Related Work . What is a simulator ? Abstractly speaking , a simulator has two components : ( 1 ) a physics engine that evolves the world state s over time st ! st+1 , and ( 2 ) a renderer that generates sensor observations o from states : st ! ot . The boundary between the two is often blurred as a matter of convenience . Many physics engines implement minimal renderers to visualize results , and some rendering engines include integrations with a physics engine . PyBullet [ 37 ] , MuJoCo [ 28 ] , DART [ 38 ] , ODE [ 39 ] , PhysX/FleX [ 40 , 41 ] , and Chrono [ 42 ] are primarily physics engines with some level of rendering , while Magnum [ 43 ] , ORRB [ 44 ] , and PyRender [ 45 ] are primarily renderers . Game engines like Unity [ 46 ] and Unreal [ 47 ] provide tightly coupled integration of physics and rendering . Some simulators [ 3 , 48 , 49 ] involve largely static environments – the agent can move but not change the state of the environment ( e.g . open cabinets ) . Thus , they are heavily invested in rendering with fairly lightweight physics ( e.g . collision checking with the agent approximated as a cylinder ) . How are interactive simulators built today ? Either by relying on game engines [ 6 , 50 , 51 ] or via a ‘ homebrew ’ integration of existing rendering and physics libraries [ 7 , 34 , 36 , 52 ] . Both options have problems . Game engines tend to be optimized for human needs ( high image-resolution , ⇠60 FPS , persistent display ) not for AI ’ s needs [ 53 ] ( 10k+ FPS , low-res , ‘ headless ’ deployment on a 2To be clear , H2.0 fully supports the rigid-body mechanics of grasping ; the abstract grasping is a task-level simplification that can be trivially undone . Grasping , in-hand manipulation , and goal-directed releasing of a grasp are all challenging open research problems [ 23–25 ] that we believe must further mature in the fixed-based close-range setting before being integrated into a long-horizon home-scale rearrangement problem . 3Start using Habitat 2.0 with the tutorial at https : //aihabitat.org/docs/habitat2 cluster ) . Reliance on them leads to limited control over the performance characteristics . On the other hand , they represent decades of knowledge and engineering effort whose value can not be discounted . This is perhaps why ‘ homebrew ’ efforts involve a high-level ( typically Python-based ) integration of existing libraries . Unfortunately but understandably , this results in simulation speeds of 10-100s of SPS , which is orders of magnitude sub-optimal . H2.0 involved a deep low-level ( C++ ) integration of rendering ( via Magnum [ 43 ] ) and physics ( via Bullet [ 37 ] ) , enabling precise control of scheduling and task-aware optimizations , resulting in substantial performance improvements . Object rearrangement . Task- and motion-planning [ 54 ] and mobile manipulation have a long history in AI and robotics , whose full survey is beyond the scope of this document . Batra et al . [ 9 ] provide a good summary of the historical background of rearrangement , a review of recent efforts , a general framework , and a set of recommendations that we adopt here . Broadly speaking , our work is distinguished from prior literature by a combination of the emphasis on visual perception , lack of access to state , systematic generalization , and the experimental setup of visually-complex and ecologically-realistic home-scale environments . We now situate w.r.t . a few recent efforts . [ 55 ] study replanning in the presence of partial observability but do not consider mobile manipulation . [ 52 ] tackle ‘ interactive navigation ’ , where the robot can bump into and push objects during navigation , but does not have an arm . Some works [ 56–58 ] abstract away gross motor control entirely by using symbolic interaction capabilities ( e.g . a ‘ pick up X ’ action ) or a ‘ magic pointer ’ [ 9 ] . We use abstracted grasping but not abstract manipulation . [ 19 ] develop hierarchical methods for mobile manipulation , combining RL policies for goal-generation and motion-planning for executing them . We use the opposite combination of planning and learning – using task-planning to generate goals and RL for skills . [ 33 ] is perhaps the most similar to our work . Their task involves moving a single object from one location to another , excluding interactions with container objects ( opening a drawer or fridge to place an object inside ) . We will see that rearrangement of multiple objects while handling containment is a much more challenging task . Interestingly , our experiments show evidence for the opposite conclusion reached therein – monolithic end-to-end trained RL methods are outperformed by a modular approach that is trained stage-wise to handle long-horizon rearrangement tasks .

Summary: This paper presents the development of a simulator of realistic home environments with robotics support, including physics simulation, photorealism and curated assets. The most relevant novel feature is the increase in simulation speed by orders of magnitude, which enables significantly faster training times. The simulator, which is openly available, also unifies a number of advances introduced by other simulators from the embodied AI community, such as converting real scenes into CAD models for enabling interaction, now into a high performance platform with a well designed and scalable software design. Overall, this paper presents a large engineering effort and investment into developing software for a simulator and environment assets that the community can benefit from.