Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This work presents ProcTHOR, a procedural generation schema to generate synthetic house scenes subject to user-specified constraints. ProcTHOR has been applied to generate a set of 10,000 diverse house environment (fully equipped with sampled objects, materials, and other physics/rendering attributes). Training on these procedurally generated environments achieves state-of-the-art results over a range of embodied-AI tasks that solely rely on RGB images.
strengths_and_weaknesses: Strengths
=========

**S1** Embodied-AI is a very active area of research in the robot learning and computer vision communities. The tasks addressed in this paper are directly based on a number of benchmarks proposed over the last 5 years. I thus expect this work to be relevant to several members at Neurips.

**S2** This paper is extremely well-written. It is easy to follow the core contributions of this work, and the range of experiments (both in terms of environments and tasks) is somewhat broad. I also skimmed through some of the technical details about the dataset and I appreciated all of the thought and work that went into preparing the manuscript and supplemntary materials. Overall, this is thoroughly well-executed work.

---

Weaknesses
==========

While the work itself is well-executed, I have concerns over the technical aspects of this paper, which need further discussion.

**W1** *Core contribution a dataset?* The paper, in its current form, comes off as a dataset contribution: the core takeaway for me was that it is possible to procedurally generate as many scenes as desired; and that the combination of the procedural generation scheme and the scale it achieves results in state-of-the-art performance over a number of embodied-AI tasks. While this is a demonstration that scale (and to an extent domain randomization) results in good performance, the key contribution enabling this is the ProcTHOR generation technique, which is more of a dataset (generator). I feel that the "datasets and benchmarks" track at Neurips would have been a much more suitable fit for this paper. I would like to see this aspect discussed further in the author-response phase. Particularly because, the neurips guidelines (here)[https://neurips.cc/Conferences/2022/CallForDatasetsBenchmarks] and (here)[https://neurips.cc/Conferences/2022/ReviewerGuidelines] focus on "algorithmic advances, analysis, and applications"; and I have a hard time reconciling with either of these categories in terms of technical contributions of this submission.

**W2** *Focus on RGB-only agents*: The current submissions only evaluate RGB-only agents across environments. While this is arguably a harder task than the scenario where RGB-D data is available, several researchers interested in using the ProcTHOR environment are also likely interested in leveraging RGB-D sensors. It is therefore important to analyze performance on various other sensor configurations, and evaluate the gains ascribed by ProcTHOR when varying sensing modalities.

**W3** *Procedural generation*: Building on **W1**, if the procedural generation scheme itself were to be considered a core contribution, it would have helped to have evaluation with respect to other competing procedural generation schemes (such as those cited in Sec. 2), or with respect to the various design choices within the ProcTHOR environment itself.

---

(Another aspect that could be relevant for future exploration is to optimize aspects of the procedural generator itself, to maximize performance on a set of downstream tasks. A somewhat recent example of this is observed in "Meta-Sim: Learning to Generate Synthetic Datasets").
questions: I would like to see **W1**, **W2**, and **W3** discussed in the rebuttal phase
limitations: I find the discussion satisfactory
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents ProcThor, a framework to sample virtual and interactive 3D environments from an underlying distribution of room and object layouts. In the current work, 10000 3D environments of varying sizes, # rooms, and object distributions are sampled and enable simulation for object search and manipulation tasks. The experiments demonstrate that training policies on the synthetically generated environments and the finetuning it on other datasets like RoboThor, AI2Thor, and Habitat-Matterport3D lead to state-of-the-art performances. Furthermore, the ablation experiments reveal that the transfer task performance continues to improve as more and more virtual 3D environments are sampled for training. 
strengths_and_weaknesses: # ----------------------- Post-rebuttal update --------------------
The authors' responses during the rebuttal addressed majority of my concerns and I'm retaining my rating at 8. I request that the authors update the paper with the reviewer feedback where appropriate. 

I'd also like to see the HM3D-objectnav scaling experiments under fine-tuning settings in the final version of the paper (or even in supp). I feel like that would partially answer the question of whether synthetically generating training environments can enable meaningful performance-scaling in real-world like scenes (with very different visual characteristics).  


# ----------------------- Pre-rebuttal review --------------------
# Strengths
- The paper is well-written and easy to understand. The supplementary gives detailed information about generating scenes and helps clarify any ambiguity.
- The framework developed is very extensive and is a massive feat of engineering (as detailed in supplementary). The authors have shared the code for their experiments and promised to open source it. I expect this to be very impactful for the embodied AI literature and beyond. 
- L208 - The efficiency of generating scenes is also impressive. 10k scenes were generated in 1 hour with 4 GPUs. The scenes are also efficient to store in a small JSON file per scene.
- The experiments are well designed to cover 3 tasks from multiple online leaderboards. Both the zero-shot and finetuning experiments are impressive, especially considering that the agent only relies on RGB sensing (and not depth or panoramic sensing like some prior work). 

# Weaknesses
- The novelty of the scene generation process is a bit unclear. My impression is that regardless of the novelty of each step of the pipeline, this framework on its own is a massive feat of engineering. Nevertheless, it would be useful to do a careful related works comparison to different steps of the pipeline. For example, the Semantic Asset Groups are similar to hyper-relations from [1].
- In supplementary Sec. B2, the authors state the use of only 16 room specs for generating the 10000 scenes. This raises a question of diversity for generation. It's also not clear how room specs were obtained in the first place. 
- Table 1 - the rendering speeds are good, but there is no comparison to existing simulation platforms like iGibson, Gibson, Habitat, Habitat-2.0, etc. I feel it is important to benchmark these under a consistent hardware setup. It is okay even if ProcThor is slower than any of these individual frameworks since none of them support procedural generation of interactive scenes (and on such a large scale). 
- Table 2 - in the transfer results to HM3D-Semantic ObjectNav, the zero-shot performance is very low, but the fine-tuned performance is state-of-the-art. I'm not sure I understand what is being transferred here (visual representations? Navigation ability? Object-based scene priors? Ability to stop?)
- Table 3 - the ablation study is good, but it leaves two questions unanswered. The results are all zero-shot, so it is not clear if the trends hold true when models are finetuned. Also, the HM3D zero-shot performance isn't consistently increasing. This raises the question of whether this large-scale procedural generation will actually help for real-world robotics.

[1] Zhang, Shao-Kui, Wei-Yu Xie, and Song-Hai Zhang. "Geometry-based layout generation with hyper-relations among objects." Graphical Models 116 (2021): 101104.

questions: - Is the scene generation process novel? Could the authors do a detailed comparison of different steps to existing literature? This is essential for understanding ProcTHOR and improving it in the future.
- How are room specs obtained (and are they realistic)? Does having only 16 specs limit the diversity?
- How do rendering speeds compare to other frameworks like AI2Thor, iGibson, Gibson, Habitat, Habitat-2.0, etc?
- What is being transferred when the visual appearance is significantly different (like HM3D-Semantic ObjectNav)?
- Do the scaling ablations hold true when models are finetuned? Does the lack of consistent scaling for HM3D-Semantic ObjectNav reflect poorly on the ability to use ProcThor to benefit real-world robotics?

limitations: The authors have adequately addressed this.
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The work introduces ProcTHOR, a procedural indoor environment generator. Each environment generated by ProcTHOR is a plausible environment with a physics engine, which can be used for training agents to solve various indoor tasks. The authors also show that pretraining on a sample of 10k homes from this generator, either in a zero-shot setting or after fine-tuning, the trained agent outperforms top contributions on several leaderboards.
strengths_and_weaknesses: TL;DR my main concerns are (a) reproducibility/code release and (b) why this wasn't submitted to the datasets/benchmarks track.

### Strengths:
- **S.1)** The paper is really well written and structured. It was a blast to read. All the illustrations and plots are meaningful and illustrate the work.
- **S.2)** The contribution of ProcTHOR itself is great and really needed by the community. 
- **S.3)** The RL results look great. The fact that with a bit of fine-tuning (and sometimes without), this outperforms many baselines is impressive.

### Weaknesses:
- **W.1)** This really should've been submitted to the dataset & benchmarks track and I don't understand why the authors didn't do that. Because now I have to evaluate this for its methodological contribution and there isn't any. To my best knowledge, the ProcTHOR generator was pieced together with various artist assets, heuristics, and a lot of human annotation. That isn't a method that translates to any other domain and the authors also make no claim that their ProcTHOR can easily be adapted to car assembly factories for example. The experiments that are shown at the end of the paper, that I positively highlighted in (S.3), aren't methodologically new. The authors even go so far as to say they deliberately used a simple agent network to highlight the benefits of the dataset. Ultimately, the decision is not on me if this fits in into the main track of the conference, so I'll refer this to the AC.
- **W.2)** Part of reviewing a paper for me is checking if the contributions claimed by the authors live up to the methods and results explained in the paper. Sadly, ArchitecTHOR (great name btw) doesn't do that. It's not even mentioned in the main body of the paper. Why is this listed as contribution if the main paper doesn't even discuss it? Sure, there are some pictures in the appendix but the main paper has to stand on its own - the appendix is for clarification and additional detail. (a) Why did you create ArchitecTHOR? (b) What did the designers focus on in designing these spaces? (c) What wasn't there yet in AI2THOR that needed to be added here? (d) What are the statistics of the spaces in terms of floor size, rooms, number of objects? (e) How does that compare to envs generated by ProcTHOR? (f) When should I use one over the other for training or is A-THOR only for evaluation? If these questions are answered in the main body of the paper, I'm happy to recognize ArchitecTHOR as dataset contribution and increase my score.
- **W.3)** The authors make the vague promise that at some point, code will be released publicly. This is a dataset paper and I have no way to reproduce the hundreds if not thousands of person-hours that went into this. If this is not reproducible and not publicly available, it's of no use to the scientific community. If this was a paper introducing a new method, this would be different because, from the mathematical description of the method, I'd be able to at least reproduce some of it. Since this is not the case, I have to make a deduction in my rating. I have to insist that you release the dataset (generator) if you want us to accept a dataset paper.
- **W.4)** In (S.1) I praised the writing and structure and I stand by that, but this was made at the sacrifice of some essential details. Sadly most of the information on HOW ProcTHOR was made was relegated to the appendix. I understand that this is due to space constraints but it kind of diminishes the value of the main paper with respect to the appendix. 
- **W.5)** Validity of claims: In the related works section, you mention HM3D, OpenRooms, and Megaverse and your main criticism of these is that they're either too small or too game-like but you don't verify that claim as far as I can tell. A way to verify this would be to use these environments for pretraining, instead of ProcTHOR-10k, and check the 0-shot and fine-tuning performance, at least in object-nav tasks (because yes, some benchmarks require interaction, and that's not offered in these but that's beside the point). This way, you can verify that (a) the size/diversity of ProcTHOR-10k is necessary and (b) the visual fidelity of ProcTHOR is necessary for this demonstrated transfer performance.
questions: - **Q.1)** I'm not sure the paper sufficiently explains what you mean by "fully interactive". I understand that objects can be modeled as rigid bodies, but do your robot agents actually do friction-based grasping, or are objects tethered to the gripper when the "grasp" action is executed with the gripper reasonably close to the object? And for shelves/fridges/things that can be opened, do they only have a binary open/closed state, or can a robot open them by an inch?
- **Q.2)** If all objects are rigid bodies, how do you assign mass, friction, and elasticity? Are these also procedural or can they be changed?
- **Q.3)** What percentage of objects have these states (open/closed, etc)?
- **Q.4)** In the appendix, you mentioned how many frames of training are required but how does that translate to wall clock time, i.e. how long did you train a policy for on ProcTHOR-10k until convergence?
- **Q.5)** It's a bit crazy to me to say that you couldn't run additional experiments to verify performance and you took the first result because they're so expensive. In RL, it's a well-known phenomenon that different seeds can lead to vastly different performance or even different implementations of the same algorithm. Could you please elaborate, or if you had time since submission, report the mean/std of your results and how many seeds were used?
- **Q.6)** Did you mention the 16 different scene specifications somewhere? Why these?
- **Q.7)** Same for the 18 different Semantic Asset groups.
- **Q.8)** You mention customizability and "a few simple lines of specifications" but what does that actually look like?
- **Q.9)** Thanks for providing the performance table in Tab.1. That's always been my main gripe with AI2THOR. But how does the process distribution work? I.e. how do you run 15 processes on each of the 8 GPUs? 
limitations: ~~Limitations aren't mentioned in the main body of the paper. I thought that was supposed to be included last year or 2 years ago.~~

In addition to the limitations listed in the appendix, I'd add that the lighting model is still relatively simple and might not correspond to lighting conditions in real houses but this effect may be "domain-randomized" away. Also, robot navigation is vastly simplified; real-world environments might have different friction surfaces, carpets with bumps, stairs connecting different rooms on the same floor, etc.
Also, real environments might have more decoration and trinkets lying around, as well as soft objects that can appear in many different configurations (a hoody thrown over a chair or splayed out on the sofa)

EDIT: apparently limitations can go into the appendix and the authors did just that. So this is fine.
review_title: null

Meta-review: metareview: *Summary*

The paper presents ProcThor, a framework to generate interactive 3D environments from an underlying distribution of room and object layouts. In the current work, 10000 3D environments of varying sizes, # rooms, and object distributions are sampled and enable simulation for object search and manipulation tasks. The experiments demonstrate that training policies on the synthetically generated environments and the finetuning it on other datasets like RoboThor, AI2Thor, and Habitat-Matterport3D lead to state-of-the-art performances. Furthermore, the ablation experiments reveal that the transfer task performance continues to improve as more and more virtual 3D environments are sampled for training. Finally, 'zero-shot' experiments (without finetuning) are included with surprisingly good results.

*Reviews*

The reviewers' ratings are 4 (borderline reject), 8 (strong accept) and 9 (very strong accept). 

Reviewer L9s9 (borderline reject) did not respond to the rebuttal, but their main concerns were:
- (W1) the paper should be submitted to the datasets track
- (W2) the paper should include experiments with RGB-D sensors, and
- (W3) the paper should include experiments with other procedural generation schemes.

*Decision*

I have already discussed and dismissed concern W1 in my comments below. Regarding W2 and W3, these are asks for more experiments. The authors provided reasonable justifications and responses to these concerns. Further, I'm persuaded by the other reviewers that the paper already represents a 'massive feat of engineering' and note that the experiments already cover both zero-shot and finetuning settings on multiple leaderboards. Therefore I am putting less weight on L9s9's low rating.

AqAB and QxMz are united that this paper and the associated code release will have a significant impact on the Embodied AI community and I agree. The paper is also acknowledged as being extremely well-written, with very thorough experiments, and I feel it could be considered for an outstanding paper award.


confidence: Certain
Decision: Accept