Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents the development of a simulator of realistic home environments with robotics support, including physics simulation, photorealism and curated assets. The most relevant novel feature is the increase in simulation speed by orders of magnitude, which enables significantly faster training times. The simulator, which is openly available, also unifies a number of advances introduced by other simulators from the embodied AI community, such as converting real scenes into CAD models for enabling interaction, now into a high performance platform with a well designed and scalable software design. Overall, this paper presents a large engineering effort and investment into developing software for a simulator and environment assets that the community can benefit from. 
main_review: The paper has the following contributions:

-----(1) ReplicaCAD.----- A dataset of simulation environments created by CAD modelling after real spaces from the replica dataset. These assets are the outcome of extensive design and manual curation, including textures, physical parameters, objects classes, etc, enabling realistic object randomization. The specific idea of producing CAD-based environments is not novel, a point that perhaps deserves to be highlighted more in the paper. That doesn't diminish its importance, as the coupling between the simulator H2.0 and these assets enables comprehensive use of the proposed system. The assets seem to be higher quality than in related work. Clearly the outcome of a large investment as noted in the paper. 

-----(2)----- H2.0 high performance photorealistic simulator with physics engine and robot support. To me this is the main contribution of the paper in terms of novel development, as a new tool available to the community and that supports significantly higher simulation speeds. Answering novel research questions might benefit from this speed up in training times. This contribution is mainly covered in Section 4. I would have liked to see even more details on this section, as being the central breakthrough. 

* Fig. 3 shows the main point on how authors design concurrent physics stepping and rendering. Since this causes the Observation to be delayed one cycle, I found interesting the argument of biological plausibility (line 214), perhaps it would be interesting to even have this delay being an adjustable parameter. 

* I didn't find many details regarding the 1GPU vs 8GPU case. Is the 8GPU case just parallel environments or is it one environment distributed among the 8 GPU? Please offer more details.
* H2.0 takes the approach of keeping physics sim in the processor (using Bullet) and rendering on the GPU, and achieves high speed by using a number of performance optimizations. Another approach is distributed physics simulation in GPUs. For completeness, could the paper cover the pros and cons of these two approaches?

* One of the optimizations is not simulating the physics of contact between the floor and the robot's wheels. I understand the paper claims this has been shown to be a good approximation, but I still worry about the potential of transfering to the real robot. Mixing full physics sim with simplified sim can create a regime where it's hard to know what segments are realistic or not, and therefore hard to debug systems when thinking on transferring to real platforms. Of course we can also think about research purely on the simulator, for which kinematics states are enough to make progress on particular research questions. Same comment applies to the grasping simplification (which is also present in other simulators). While it's true that software-wise this is trivially undone, the main question might be not if H2.0 is able to simulate (actually Bullet), but instead how realistic it is and if that segment of manipulation would behave according to reality.



-----(3)----- Benchmark with household tasks. The paper proposes a small set of well defined robot tasks that can be fully simulated in H2.0. The tasks are mainly instances of the rearrangement problem, a particular way of looking at task and motion planning problems. I think these tasks are a good set of basic problems to support interesting research questions that can be investigated within the simulator. It is challenging to claim how solving the tasks in the simulator would transfer to real robot deployment. This is a broader research question that doesn't diminish the value of the proposed work, but it's worth to acknowledge. The paper also uses the simulator to study algorithms and models for mobile manipulation. From my point of view this section doesn't present a novel contribution in itself, as it treats topics long discussed in other robotics fields, but it's a solid way to verify that the proposed simulator H2.0 is in fact practically useful to investigate these questions. 

Other comments:

* I'm not sure using the terminology from the robotics paradigm of sense-plan-act (SPA) to be distinct from the reinforcement learning (RL) policy is the best way to highlight how these two overall approaches are different. In SPA, you refer to the pipeline sensor to motion planner to controller, and in some sense a policy also has a sensor input, to forward pass of the policy, to execution (sometimes policy output is still given to a similar low level platform controller). In robotics SPA refers to the overall flow of information and decision making (from sensor, to decision, to action execution). The most frequently used SPA uses a "motion planner", while RL uses the "forward pass of a neural network policy). This is just a thinking point about the terms.

* The paper is very clear and the details and board set of considerations are well thought. The writing makes frequent use of parentheses, which seemed too much but actually made the writing very concrete, grounding the general comments into specific aspects that people in the field will get right away. 

* While the presented experiments show the usability of the simulator and assets for this type of problems, I think it would also be important to report more quantitatively what quality to expect from the physics simulation in comparison with the real world objects. The paper reports to have authored the physical parameters of the assets but just from reading the paper I'm not sure what level of realism to expect if I wanted to use H2.0 for robotics research.

* One of the optimizations for speeding up the simulation is to use object sleeping for objects beyond a certain distance from the robot agent. How can the approach be adapted to multi-agent systems? For example, more than one robot in the scene.

* I'm not sure if this paper should be on the main paper submission track on in the benchmark track. 

* The video should have been submitted with supplementary materials, not only a link on the paper.

limitations_and_societal_impact: I don't have concerns here.
review_title: null

final_score: rating: 10: Top 5% of accepted NeurIPS papers, seminal paper
review_content: summary: This paper presents a simulator for embodied agents for manipulation tasks in house-scale indoor environments. The paper also presents a realistic dataset of household scenarios, and home assistant benchmark (HAB), a suite of tasks designed to evaluate the performance of classical and learning based methods on rearrangement problems.
main_review: Strengths
---------

**S1** Habitat 2.0 (H2.0) represents a significant advance on multiple fronts for the embodied AI community. By curating the ReplicaCAD dataset, providing a simulation package that speeds up training by over 100x, and by introducing a large-scale benchmark of tasks that are currently out-of-reach of deep RL techniques, this paper makes comprehensive contributions to the community and paves way for a potentially large volume of exciting future work.

**S2** The engineering and artistic efforts on ReplicaCAD and H2.0 are impressive. The embodied AI community has a dearth of interactive datasets; existing datasets that are rich are either small scale (AI2-THOR has room-scale scenes) or have very few scenes (iGibson has 14-15 scenes in total). Most simulators that are rich in terms of physics rely on underlying engines (e.g., Unity or Nvidia Flex) and end up being slow. H2.0 provides a highly optimized (CPU) physics engine and interleaves CPU-GPU usage for physics and graphics engines, enabling scaling conventional RL to extremely long training experiences / episodes.

**S3** This paper points out interesting limitations with the combinatorial generalization of learned atomic "skills" (Fig. 5). While atomic skills are learned with high success rates, they do not seem to chain well, and errors from early-on skills in the execution pipeline cascade, often resulting in task failure.

**S4** The work is extremely timely, tackles important yet-unsolved problems, and does so comprehensively. This paper was an unconventional, yet extremely well-written; making it an enjoyable read.


Weaknesses
----------

I have the following concerns that I hope to see discussed in the author response phase.

**W1** *Beyond success rate*: Currently, the sole metric that techniques are being compared upon is "success rate" (i.e., the percentage of test scenarios that end up being 'successful'). While success rate indicates a gross performance of a method, it does not capture the efficiency with which the task is carried out in general. Policies that achieve goals quickly may be preferable to ones that are sucessful, but do so at the expense of additional time. I would like to see more discussion around this in the author response. I believe this to be an important factor given the goal of the benchmark is to spur future work in rearrangement-style tasks involving manipulation.


**W2** *SensePlanAct (SPA) baseline*: Are there any insights as to why SPA-Privileged has a lower success rate in two of the settings in Table 3, when compared to MonolithicRL; despite access to perfect sensing and perfect control? The only (speculative) reason that comes to mind is that bidirectional RRD fails to compute a feasible motion plan before the episode terminates.


**W3** *Solvability*: Is there a mechanism to ensure that each task in the benchmark is "solvable"? If so, is there an "oracle" that would produce a solution to each task? (This might be a useful tool to gather demonstrations for techniques that leverage expert demonstrations)


Minor remarks
-------------

The following comments do not impact my overall evaluation of the paper. I believe these to be easily addressed in a minor revision.

- MonolithicRL: Were subsets of input modalities ablated upon, to analyze the imact of each sensing modality on "pick" success rate? (i.e., how would a depth-only policy compare to a RGB-D policy, etc.)

- Grasping - are there scenarios where unintended grasp releases cause unforseen / unsolvable scenarios? For example, a policy may arbitrarily choose to release an object before it reaches a target receptacle, and the object ends up in a location where it is inaccessible to the end effector.

- SensePlanAct (SPA) baseline: The "sensing" phase is claimed to accumate a 3D pointcloud. How is this constructed? Using an off-the-shelf SLAM system? Does this require relative pose for each input image to be known a priori?


Justification for score
-----------------------

Given the perceived impact such work would have on a significant fraction on the ML community, I would strongly argue in favour of accepting this contribution to Neurips. I believe this work to have immense scope for following-up on, particularly given the promises around permissibly-licensed releases of code and data. While I enlist a few concerns with this work, I believe the strengths comfortably overshadow these concerns, and that it might be difficult to pack more into what is already a content-rich 9-page scientific document.
limitations_and_societal_impact: I liked the limitations and societal impact section -- I believe it touches upon major societal ramifications of the work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a new simulation environment for robotic pick-and-place tasks. The main contribution is a significant engineering effort to accelerate simulation around an existing physics engine, Bullet. The work includes some models for simulation and three tasks to test solutions. The paper also evaluates “flat” reinforcement learning, and hierarchical structure, with learned subcomponents or motion planning ones.
main_review: Originality: The presented work presents a version of the simulator and novel models. However, intellectually, I do not appreciate considerably novel contributions. Different forms of the features presented here have been presented by others before (models of scenes and objects, interactive simulation, rearrangement tasks,…). In that sense, the paper does not present original ideas or simulation concepts. 

Quality: The text, images, and code are well executed. I appreciate the work of a large engineering group, as it is highlighted explicitly several times (see my comment on repeating the invested time in the “limitations” section).

Clarity: The paper is clearly written. However, there are some concepts that are presented in a way that sounds like novel contributions of this work instead of part of the codebase this work extends (like Bullet functionalities for sleeping objects, or the common rigid body physics simulation). It would be great to 1) make clear in the text what is really novel in this work, 2) compare to state of the art in those aspects. Any element that is listed as a contribution, should be compared in the related work.

Significance: The code presented here (the new simulation environment) could be useful for the community. The manuscript does contain some additional information. 

In summary, I see the engineering value of the work, but the intellectual novelty is not high. 


limitations_and_societal_impact: I see some limitations and unclear parts in the current manuscript that I list below. I hope this helps to improve the manuscript.

It is a bit unclear what is the number of apartments. “111 unique layouts of a single apartment background”. I’m interpreting this as “one apartment, 111 different ways of arranging the furniture”? In that case, it sounds limited with respect to AI2Thor, RoboThor (that enables multiple arrangements) and iGibson (that provides 15 environments and the possibility to change the objects). TDW includes 100 rooms. From a user’s perspective, why not unifying all of those? I get that AI2Thor is a bit harder because it is Unity, but iGibson is also Bullet? 92 objects do not sound very large as dataset. AI2Thor includes more than 100 objects. iGibson annotated 150 models. While valuable, I think to be a strong contribution, a dataset of objects should include many more objects, e.g., ShapeNet, PartNet… 

It is a bit strange to continuously highlight the time invested in the different parts (hours of work, years of work). I don’t think this should be a factor to decide on the value of something. I would prefer to remove references to that, especially assuming that a large company is behind this effort, and making the “hours of work” a feature to highlight would clearly benefit large corporations with large teams or the budget to hire contractors, which could be an ethical concern.

When explaining Habitat 2.0, the text explains the capabilities of any physics simulator, and that makes it hard to attribute credit. First, I think it should be necessary to cite the first time (line 35) the true source of the code, which is Couman’s Bullet because H2.0 is not advancing the field of rigid body simulation, the core equations are the same. Second, highlighting rigid body simulation capabilities in the way they are listed may be misleading: any rigid body physics simulator can do all of this, and if it supports simulating kinematic chains it can simulate any robot. Are all the listed robots modeled and included in H2 or are they just random examples? I couldn't find in the code. Please, clarify.  

The number of tasks is somewhat also not to pair with other benchmarks like metaworld, or RLBench. The way of defining the tasks is also very ad-hoc: there is a new term used “GeometricGoal”, but this is the common way of defining pick and place tasks in robotics: place the object at this location with some threshold. How scalable is the definition of tasks? Does it have to be manually defined for each new task? Can the tasks be defined more semantically instead of geometrically?

The related work section is lacking depth. Currently, there is a lot of simulation environments out there and it is increasingly difficult for users to understand their pros and cons, so I really think this section is critical. For example, I would recommend analyzing in detail the “closest neighbors”: AI2Thor (and variants), Sapien, iGibson, ThreeDWorld, RLBench. Right now, in two paragraphs, all physics simulators, renderers, and game engines are mentioned, but this feels more like a list than an in-depth comparison. The paper mentions several “points” that this work contributes but the related work focuses only on one: speed. In that sense, it feels like this is really the only difference to existing work. The subsections of the related work are also strange as they try to explain something instead of placing the current work in a scholarly context. The comparing table is a great idea but the information included right now is not very useful: it includes the names of the rendering libraries and the type of rendering (not explained or referred in the paper) but the only critical number is the speed, lacking a comparison in the other mentioned contributions (the scenes/objects, and the benchmark tasks). Also, the cited paper [33] is really similar, it would be great to go deeper in the comparison. Although it is only arxiv, I would recommend including a comparison to “The ThreeDWorld Transport Challenge” that is also extremely similar. 

For related work, would be possible to include name+number instead of only the number? E.g. “Garret et al. [55] study…” instead of “[55] study…”. This facilitates readability.

Joint type and limits are not dynamic properties, they are kinematics.

It is a bit unclear to me why highlighting and novel the “Localized Physics”. Bullet (as most mature physics simulators) already optimize in this front. Bullet uses the well-known concept of “islands” of objects to simulate and activate/sleep bodies. This is a default functionality. The only thing I can infer from this text is that this project may have improved a bit on that? Still, the way it is explained (“ if a robot is picking a mug from a living-room table, why must we check for collisions between the kitchen fridge shelf and objects on it?”) is to be misleading as it sounds like a novel contribution while it is only a (valuable) engineering fine-tuning of an existing concept and implementation. The second contribution listed there, “using a navigation mesh to move the robot base kinematically”: how does this affect if there are objects on the floor, for example, one of the picked objects is dropped? Can the objects be dropped? Maybe the “perfect grasping” impedes that? It would be interesting to deactivate the “perfect grasping” functionality in a small experiment (maybe the pick one) and evaluate the performance. This would tell us how realistic the results are.

Why is there an association between the “simulation step” and the “control step”? Is the control signal changed at 120Hz? Is there a controller running? If not, it is just physics simulation, right? Please, clarify. 

It would be great to have a discussion on parallelization. What is parallelized? The same environment? How is that possible? Copies of the same environment? I wonder how the use of parallelizing computers and GPUs relate to other simulators, for example, to Mujoco, concretely with respect to their analysis here: http://www.mujoco.org/performance.html and in related papers.

Does the robot have access to the absolute location of the base? This is given as “realistic proprioception” but I’m not sure this is the default on any robot. GPS won’t provide this level of accuracy, even if it would work in an apartment.

Why measuring the accumulated contact force in the experiments and use it to stop? What is that physically? Is the accumulation over the entire arm at a time step (in that case, 5000N seems way off the payload) or over time? If it is over time, there is no limit per time step?

I’m confused about the Sense-Plan-Act paradigm listed as “classical”. The alternative tested does task planning and uses RL-learned skills but is also doing the same three “primitives”. I went back to Murphy’s book and I do not understand the terminology in the paper. Maybe the RL option can be also explained in the same terms as the cited Murphy book? Is the main difference not just that the “act” part is executed by a learned policy? For example, in line 286, the solution explained is a “Sense-Act” (Murphy calls this “reactive paradigm”).

How are collisions implemented? Are there fully implemented? Some visualizations of the trajectories show the arm going through objects. Also, how what is the field of view of the cameras? Are they realistic?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces a high performance simulation platform called Habitat 2.0 (H2.0) for training virtual robots and benchmarks RL and classical robotics policies on home assistant benchmark (HAB) that consists of three long-range tasks:  `tidying house` (moving 5 objects from random reachable locations back to where they belong), `Preparing Groceries` which is removing 2 objects from the fridge to the counters and place one back in the fridge and `setting table` which is about getting a bowl from the drawer, fruit from the fridge and placing fruit inside the bowl on the table. 

The platform also involves a lot of engineering infrastructure development to increase the performance of the simulator leading to an impressive 850x improvement compared to real-time. It is worth noting that the grasping is abstracted i.e. whenever the gripper is within the 15cm range from the object, the object snaps into the gripper.  

Using this simulator platform hey benchmark RL and classical robotics pipeline which they call as sense-plan-act (SPA) and show that SPA based policies are brittle but RL based policies need hierarchical planners as pure image to action based RL policies fail to solve long-range tasks due to various challenges involving the length of the task and inability to come up with a sensible reward function. However, RL with hierarchical planners aren't perfect either and can suffer from inability to cope with the complexity of the environment. Therefore, the simulation platform provides a humbling peek into the inability of current state of the art algorithms. 
main_review: - The paper shows that pure image to action policies --- called Monolithic RL --- are simply not capable to solving tasks on their own. This is because of the complexity of the task that is well beyond the reach of current SOTA RL algorithms as well as the reward function. In general, for long-range tasks, coming up with the right objective remains elusive.

- Comparison between TP+SRL --- which is policies involving TaskPlanning + SkillsRL where TaskPlanner (STRIPS) is used to break the task down into atomic tasks which are all learned via RL --- and TP+SPA --- (STRIPS planner) and sense plan act policy involving classical robotics pipeline to solve the tasks --- doesn't reveal a straightforward pattern. On the `TidyHouse` task, TP+ SPA performs better than TP+ SRL while the trend is reversed for the `PrepareGroceries` task. 

It is humbling to see that even for seemingly simpler tasks with near-perfect planners guiding the chain of skills, it's not possible to fully solve the tasks requiring multiple interactions with the environments. This is even more surprising considering that the grasping has been abstracted in this work. The issue seems to be either compounding errors or agents reaching positions where the are unable to view the required workspace in their viewport. Providing an oracle navigation helps understand the gap as they tend to have better success rates. 

The engineering effort in building up the pipeline and platform to carry out tasks at extreme high frame-rates is very commendable. 
limitations_and_societal_impact: While exploring the robotics set-up, the paper seems to be mostly focussing on the understanding of the RL algorithms and cares less about the real problems involved in doing learning for robotics e.g. grasping is abstracted with considerable and generous tolerances of 15cm and that if the gripper is within this tolerance the objects snaps into it. Most often in the real world, the issues arise due to unstable or impossible grasping and that is abstracted completely in this work. In the end even if we have perfect planner designing generalisble low-level controllers is pretty hard. This work seems to completely sidestep that part. Obviously this makes the problem even more challenging but I'd have liked to see some work in that direction too. The success rate would go down quite a bit but it would be worth plotting. Moreover, as more contacts are made, the simulator will slow down too. Table 1 shows comparison against various other simulators but they don't all necessarily have similar settings so it is not fair. 

Also, when they simulate things at 30Hz (time step 1/30) there is no motion blur simulation which could also affect the policy. I wonder if there was any decision choices made for that I didn't see that in the paper. In the real world motion blur is certainly something that is going to affect the performance of the algorithm.
review_title: null

Meta-review: The paper presents the second version of a well-known and well used simulator, environment and benchmark for embodied computer vision, in particular navigation tasks. 

The paper received 4 expert reviews with a wide spread (minimum rating 4, maximum rating), and was initially on the fence. The main weaknesses raised where a lack of methodology, as the paper mainly describes an engineering contribution.

In the discussion phase, a consensus emerged on the usefulness of this kind of contribution for the community, as the impact of the new simulator can be estimated to be large.

The AC concurs and recommends acceptance.
Decision: Accept (Spotlight)