Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents an open-source simulation environment iGibson 2.0 which supports a wide range of household tasks, extending the simulation from kinodynamics to other object states such as temperature etc. It provides a variety of functionalities, including logical states, generative scene initialization, and VR interface. 


strengths_and_weaknesses: Strengths:
- An open-source efforts on simulation with very extensive functionalities. It provides a wide range of logical states description, which alleviates users' efforts to manually define a lot of logic states on their own. This could also speed up the research efforts on related tasks, making people focus more on the algorithmic designs.
- An interactive interface for easily collecting demonstration data.

Weakness:
- Seems that the relational states are limited to a pair of objects only. Would be nice to see an extension to this.
- The simulation seems to focus more on the embodied AI research. It is hard to see how to transfer a policy from simulation to the real world for robotics tasks. 

summary_of_recommendation: This is  a simulation effort that will contribute to the community a lot. And the descriptions of the simulation is extensive.
issues: I mainly have two questions about the experiment section:

- For experiments on iGibson tasks, how is the reward defined for tasks 1) to 5) ? Is it just a sparse reward function? Does it come with any reward shaping?

- What is the reason of not running rl algorithms on task 6)? It would be nice to see some justifications for this to understand the difficulty of task 6) compared to other tasks.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents iGibson 2.0, an open-source simulation environment with three main features:
- a set of new physical properties such as temperature, wetness and cleanliness levels as well as unary and binary logical predicates such as "ToggledOn", "InsideOf", etc.
- generative functions to sample valid simulated states given the logical predicates
- a virtual reality interface to record human demonstrations for robot learning
strengths_and_weaknesses: The paper has two main strengths:
- The generative functions may enable researchers to easily generate diverse scenes with several objects to train and test robot learning algorithms.
- The virtual reality interface may be very useful to teach robots by demonstration or to initialize learning algorithms with human demonstrations.

Possible weaknesses of this work lie in the fact that the simulations are rather simplistic. When cooking, the food has only three stages: frozen, cooked or burnt; sliced objects are simply replaced by two halves; liquid is simulated as droplets. As a consequence, the utility of training learning algorithms in these simulated environments is questionable if the ultimate goal is to enable robots to solve tasks in the real world.

Some ideas for future work:
- an empty/full state for glasses, cups, bottles, etc.
- It might be interesting to consider simulating the possibility of making things dirty. For example, the preparation of food may make plates, cutlery and other objects dirty. One might want to work so that not too many things get dirty...


summary_of_recommendation: The generative capabilities of the proposed simulation environment may be helpful to researchers in need of creating diverse scenarios with several objects to train and test robot learning algorithms. The virtual reality interface presents a convenient way of giving demonstrations, especially for bimanual tasks.

The lack of realism of the simulations might prevent this environment from actually being used to train agents with the ultimate goal of deploying them in the real world.
issues: * Typos:
- Section 3: " They can also be contained in receptacles (e.g. cups) and poured later, leading to realistic behavior for the simulation of several household activities involving liquids, illustrated in Fig. 2a." --> The reference here should be to Fig. 2b.
- Figure 3b: "When it is toggled on and and its appearance changes accordingly." --> delete one "and"
- Figure 4: "Visualizations of densely populated ecological scenes"; " robot learning and ecological experience in virtual reality" --> is the desired term here really "ecological"?
- Section 6: "The results of this experiment are shown in Fig. 3b." --> The reference here should be to Fig. 5b.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents iGibson 2.0, an embodied robotics simulator that makes three novel contributions: (1) the introduction of object states such as temperature, cleanliness, and toggled states, (2) the inclusion of predicate logic functions that can be used to specify tasks and generate scenes, and (3) a virtual reality system to collect demonstrations. These contributions are aimed at increasing the complexity and diversity of embodied tasks while also facilitating human access to these tasks via improved user interfaces.
strengths_and_weaknesses: Strengths:
* The introduction is well-written and clear: it begins with a high-level discussion of the problem and then explains the solutions to address the problem
* The related work section is thorough: it lays out prior works and contrasts these works with the proposed approach
* The methods sections are thorough yet easy to understand: the figures especially complement the paper well
* The three technical contributions of the paper are all novel and useful to the embodied AI community, especially contribution #2: the ability to generate scenes with a large number of objects can enable the study of much more realistic tasks

Weaknesses:
* Some of the design choices — in particular modeling water as a set of discrete spheres — seem unrealistic, raising doubts on the possibility of sim-to-real transfer
* The RL experiments study relatively easy tasks: they consider relatively simple scenes, relatively short-horizon behaviors, dense rewards, and (presumably) full observability. Removing these assumptions is not explored in the experiments, making it difficult to understand the bottlenecks of learning-based agents in solving complex embodied tasks
* The imitation learning experiment is not informative: providing 5 demonstrations seems artificially low, are there constraints for having such few demonstrations?
summary_of_recommendation: Overall this paper is well-written, clear, and makes novel and useful technical contributions. The main consideration for improving the paper concerns the experiments — they study relatively simple tasks and the experimental analysis can go deeper. Refer to the list of issues for concrete suggestions.

**Update after rebuttal**: my recommendation stays the same. See post below for details.
issues: * Explore the strengths and weaknesses of learning-based agents further. Either discuss or present empirical studies to answer the following questions: (1) what types of behaviors / tasks are especially difficult to solve (in addition to slicing)? (2) is partial observability an issue? ie. are there situations where the agent needs to keep track of the history of previous observations? (3) how sensitive is the agent to the camera viewpoint and the presence of distractor objects? (4) are there any modifications that should be made to learning-agents for dealing with object states?
* Consider re-doing the imitation learning experiment with a substantially higher number of demonstrations
* discuss the limitations of this work and future avenues of research
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes the iGibson 2.0 simulation framework, which when compared to prior work, focuses on tasks involving object properties (material, wetness, temperature, etc.) and logical states (raw, cooked). The authors also provide a VR interface for users to interact with the environment - the demonstrations could be collected for the purpose of imitation learning. Six novel tasks were proposed (grasping a book from a table, cooking meat, slicing fruit, etc.) which are unique to the iGibson 2.0 simulator and imitation learning results were shown on the grasping task.
strengths_and_weaknesses: iGibson 2.0 is a really impressive simulator that is needed for the next step in embodied AI research to reason about more complex semantic tasks.

My biggest concern for this work is that it is not grounded in a solid application in robotics. How could the policies or demonstrations collected in this simulator be relevant for a real robot? If there are some sim2real transfer capabilities that would be a more valid contribution for CoRL and the robotics community. Otherwise, this work is more suited for a graphics or VR conference.

The grasping demo is not a contribution since there are other more established methods of doing sim2real grasping (DexNet, 6 DOF GraspNet, etc.). The other tasks look interesting.
summary_of_recommendation: The submission in its current form is a weak accept. 

issues: I have some further questions/comments/feedback:

I would suggest showing clear comparisons, in the form of a table, to illustrate the differences between iGibson 2.0 and prior work (since there are a lot of simulators).

Some miscellaneous questions on the implementation details (it would be great to include them in the paper)
- How is the sliced peach mesh represented?
- How do you place the fish in the pan at the right stable pose? Is this hardcoded?
- Can the fish be burnt, do you model that? (instead of being in the raw vs cooked state)
- How many assets are there in total? For instance, is there only 1 peach? There seems to be very limited assets in general
- Do you have licenses for the assets when they are publicly released?
review_title: null

Meta-review: All reviewers believe iGibson is an interesting simulator and they like the generative capabilities and tool for collecting demonstrations. The majority opinion seems positive. However, there are concerns about (a) simplistic representations (discrete states); (b) applicability to robotics — will sim2real work or how else do author envision this to be a CoRL paper; (c) detailed stud of what tasks would work and what would not. The authors should also redo IL experiments as kM4j asks for.

==
Post Rebuttal

The authors posted a strong response to the reviews and all reviews are now positive. AC appreciates all the work put in by the authors and recommends acceptance.
Decision: Accept (Poster)