Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work partially describes a novel benchmark of human VR demonstrations for everyday household activities.  The activities chosen are for actual common tasks (based on empirical frequency rankings).  Tasks are embodied within iGibson and coded for with a new logical description language. The work is situated alongside a quickly growing collection of embodied AI benchmarks and several of the new tasks are used for training baseline RL systems.
strengths_and_weaknesses: **Strengths**:
- The demonstration dataset is based on real action sequences collected at scale from humans.
- The data includes long horizon tasks with complex dynamics and are highly diverse within the home setting.

**Weaknesses**:
The current report spends more energy selling the concept of the dataset than providing useful or necessary details.  There are many places where a claim is made, decision taken, or statistic shown but then no follow-up throughout the paper explains it.  I'll note a few such cases here:
1. "Activity Length: 20,000" -- What activity is 20,000 steps long? and this is simply an error or something meaningful? What is the distribution or std dev?
2. "unlimited human-defined versions" and "infinite unique instantiations" -- is this simply saying that surfaces are technically R^2? This is therefore true of several other environments.
3. Section 5 outlines the primitives, this cross product is probably actually the accurate measure of the size and diversity of the dataset?
4. Why evaluate only the 12 most simple activities? and if compute was the bottleneck, why wasn't a simple imitation learning policy used first on a larger set of activities instead of RL?
5. If the primary difficulty is due to physics, it seems like we would want to specifically train a physics model (supervised) and incorporate it here? Once we've done this, if many of the simple tasks are now able to very quickly perform at ceiling, a more complete evaluation could be run efficiently over the full dataset?

Minor clarifications: 
1. In the current writing it is often difficult to determine which platform and action space is in use.  Minimally, am I correct in understanding that no results use the simulated Fetch platform? 
2. The continuous action space does not include fingers? So grasps are still effectively binary motion primitives here?  Can you therefore elaborate on what aspects of the physics prove most difficult?
summary_of_recommendation: I think the dataset collected has very serious potential for the community and I very much appreciate the effort and energies that have gone into its construction.  I do not however feel that the current manuscript does the work service.  I learned and enjoyed reading more the appendix of this paper than the actual paper.  This may be partially the fault of the 8-page limit, but I think the authors are attempting to do too many things in one paper: 1. Introduce new abilities of iGibson, 2. Introduce BDDL, 3. Sell a dataset as a task, 4. Define naive models, 5. Analyze said models.  4 and 5 felt particularly weak since they have such low coverage and physics is the sole difficulty.  The effect is that too many details (e.g. about values in Table 1) are never properly discussed.


Update: Thank you for a genuine effort to respond to all points and address the questions.  The responses definitely helped me better understand aspects of the data (and action space) that were unclear to me in the original reading.   My comment about selling may be partially due to the fact that I am already more than sold the area of research and the potential utility of the data, so the high-level discussion felt like it used up space that would be better used to expand on all the details in the replies here.  A future ArXiv version could include a more complete version of the paper.  A minor note, distributions for task types and overall for the data would be appreciated.  While the explanation of the very long trajectory makes sense, I think it is still misleading in the table.  Even just adding a column with the average would be helpful.
issues: I would appreciate addressing any of the weaknesses above and incorporating additional technical details in the primary paper, perhaps by removing unnecessary details about iGibson, selling of the dataset, and streamlining the modeling discussion.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes the BEHAVIOR benchmark for simulated household tasks for embodied AI agents. It builds on the iGibson2.0 simulator and includes 100 long-horizon indoor manipulation tasks. The paper defines a high-level PDDL-style language for defining tasks which enables the generation of arbitrary many instances of a task, and further introduces new datasets of teleoperated human demonstrations for the 100 tasks.

strengths_and_weaknesses: ## Strengths
- the benchmark features a substantially more diverse set of long-horizon tasks than any existing embodied AI benchmark --> this enables research on more challenging problems

- the PDDL-style task definition language allows to sample arbitrary many instances of a task that differ in object arrangement / scene layout etc --> this enables research on more robust / generalizable problem solving systems

- the paper provides a comprehensive overview of prior related benchmarks and clearly delineates the differences

- the paper provides human demonstrations for all tasks and defines a way to use them for evaluation of learning system that goes beyond raw goal-state reaching towards measuring whether a task was solved "in a reasonable way". I am not aware of prior benchmarks that evaluated like this (but I might be missing some)

- the paper is well written and easy to follow

- the paper provides a clear separation between the task-side of the benchmark and the simulator used for implementing it


## Weaknesses
My main concern is that the paper does not clearly explain the realism of the required interactions for solving tasks. There are many places in the paper that mention the realism of the benchmark, but they refer more to the high-level diversity of tasks, which I agree is more realistic than in existing benchmarks. But it lacks a clear description of what level of realism is present in the execution of each of the subtasks. Yet, this is crucial to understand the challenges posed to learning systems and how this benchmark differs from others.

In particular the paper leaves questions open about the realism on two levels:
(1) on the very low-level: how realistically are interactions eg between the agent and objects simulated. Reading through the attached iGibson2.0 paper I saw that it uses an "assisted grasping" mechanism that can remove a lot of the subtle complexity of realistic object interactions (contact forces, slipping etc). While I understand why such compromises need to be made I would like to (a) see them mentioned in the benchmark paper clearly to show what challenges the benchmark poses, including in Tab.1 where other benchmarks potentially feature more realistic object interaction physics etc and (b) understand whether such abstractions are also used for the robot experiments, eg does the robot also use assisted grasping for object interactions?

(2) on a higher level the paper lacks details what it means to "achieve" certain tasks. Eg what is required to "load the dishwasher" or to "fill the stockings"? How precise do I need to control the robot? Adding some examples of this in the main text and a more detailed account in the appendix could help understand the difficulty of the lower-level manipulations required for the tasks.

Similarly, it is unclear to me from reading the paper how much different the low-level manipulations required are from eg object rearrangement. I agree that the diversity of high-level tasks is much higher, but if all of these tasks just require moving the EEF close to object 1, then calling discrete grasp action A, then moving close to location 2 and calling discrete release action B, they are not a lot more diverse than rearrangement in terms of the low-level manipulations. More clarity on point (2) above could help to clarify this. (A discussion on such abstractions eg in grasping should probably also be added to the sim2real discussion in the appendix.)

Another limitation is the infeasibility of the tasks for current RL methods. Given the limited success of methods in all but the simplest settings, it is likely that some form of demonstration data is needed to make meaningful progress on these tasks. The authors mention that the human data could be used to help learning, but I am skeptical of the practicality of this due to the large disparity in embodiment. Instead the usability of the benchmark could increase a lot if robot demonstrations were provided, either by teleoperation or by providing scripted demonstrations. The latter might not be too hard to achieve given that the benchmark already comes with implemented action primitives.


## Questions
(A) Can users define new tasks in the framework? If yes, what does this take?

(B) What does it mean to provide two example definitions per task? (L126)

(C) How can we compare different solutions in the efficiency metric? We can probably trade one for the other, eg disarrange the environment more but minimize time / path length by knocking over a vase etc. Disarranging a chair is also a lot less severe than disarranging a vase by braking it etc. I am wondering whether the authors have thoughts on how to actually use these qualitative metrics for comparison / evaluation.
summary_of_recommendation: I am excited about this benchmark. If the authors can address some of the remaining unclear points I raised above and include their discussion in the paper, I am happy to recommend acceptance.
issues: I hope the authors can clarify some of my questions about the realism of the simulated tasks on the lower level and also clarify what kind of manipulations are required to solve the tasks. I strongly believe that the benchmark could benefit from additional robot demonstrations of some form to increase its usability and give hope that progress is possible, but I don't think it's strictly necessary for it to be a meaningful contribution.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper develops a realistic, diverse and complex benchmark with 100 household activities in simulation for embodied AI, aiming to reproduce the challenges that embodied agents must face in real world. To this end, the authors first propose a predicate logic-based language (BDDL) to enabling general definition of activities and unlimited generation of diverse instances of each activity. Besides, the authors claimed that BEHAVIOR is designed to be not bounded to any specific simulator and can be deployed in simulators that satisfy some functional requirements. Further, the authors propose a primary metric based on success and several secondary metrics for characterizing efficiency. The BEHAVIOR includes VR interfaces to collect human demonstration.
In the experiments, the authors show that: 1. the full BEHAVIOR benchmark is challenging 2. Realism, diversity and complexity indeed contributes to task-difficulty respectively.

strengths_and_weaknesses: Strengths:
1.	The paper is clearly-written and well-organized.
2.	The benchmark proposed in this paper requires heavy engineering to implement.
3.	The authors put forward a very challenging and meaningful benchmark, which I believe will be very helpful to the community in the future.
4.	The benchmark takes the advantage of human demonstration(VR interfaces), which is novel and helpful.
5.	The benchmark is not bounded to a specific simulator, and the authors identify the simulator-agnostic features required for deploying the benchmark, which I think this will be the trend in the future.

Weaknesses:
1.	The method evaluated in “Results in the original activities” experiment is too simple. I think authors should try to implement a more advanced end2end pipeline in embodied AI literature just as [1] does.

2.	The human demonstrations collected by VR is not evaluated in experiments. I really appreciate the idea that the platform can take advantage of human display, but I also want to see experiments prove that this idea is very helpful for challenging tasks

[1] L. Weihs, M. Deitke, A. Kembhavi, and R. Mottaghi. Visual room rearrangement. arXiv preprint arXiv:2103.16544, 2021. 

summary_of_recommendation: The proposed benchmark is meaningful and challenging. Some key ideas involved in this benchmark is thoughtful: the task definition should be comprehensive and enable unlimited instances generation, the benchmark should be simulator-agnostic and we should take advantage of human demonstrations. It will be better if authors evaluate more advanced method on this benchmark and evaluate the human demonstrations, to address the concerns I mentioned in the ‘Weakness’ part.

issues: 1.	I hope the authors can address my concerns in ‘Weaknesses’ part.
2.	The proposed benchmark seems to be the most challenging tasks I have ever seen, what do you think is the promising-direction/key-aspect to solve this problem in the future?
3.	There are built-in agents in recent embodied AI benchmarks [1] [2]. Do you consider develop built-in agent that can solve these tasks with high success rate? 

[1] L. Weihs, M. Deitke, A. Kembhavi, and R. Mottaghi. Visual room rearrangement. arXiv preprint arXiv:2103.16544, 2021.
[2] Puig, X., Shu, T., Li, S., Wang, Z., Tenenbaum, J. B., Fidler, S., and Torralba, A. Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration. arXiv preprint arXiv:2010.09890, 2020.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper introduces a simulator-agnostic predicate logic framework for defining task specifications and goal completion, as well as associated metrics. The paper demonstrates an implementation in iGibson for 100 human activities, and presents a dataset of 500 human demonstrations of such activities gathered via virtual reality with a direct mapping to simulated agent control.
The collection of human demonstrations that can be replayed, understood, and evaluated in simulation for training agent models is the primary contribution of this paper.

*Post-response*
The authors did a nice job clarifying some questions I and other reviewers raised. The paper contributions are still "too big" to be squeezed into 8 pages, but with the supplementary material available I think that's fine. The audience at CoRL will find the dataset useful, and if the tooling is easy to use off the shelf, the data may spawn a lot of follow up methods and approaches that will be fun to see.
strengths_and_weaknesses: The main strength of the paper is the contribution of human-level, VR-based task completion demonstrations.
That real-world activities curated by a third party are used is a strength that could be further emphasized in the writing. For example, the decision to use real-world activities brings this paper's data demonstrations closer in line to "Ecologically Valid" AI.

@inproceedings{devries:arxiv20,
  author = {Harm de Vries and Dzmitry Bahdanau and Christopher Manning},
  booktitle = {arXiv},
  title = {Towards Ecologically Valid Research on Language User Interfaces},
  year = {2020},
}

The main weakness of the paper is that CoRL isn't the "perfect" fit venue for it. The contributions are substantial in both data and implementation, but are limited to simulation, while CoRL as a space prefers actual robot learning.
summary_of_recommendation: The VR dataset and benchmark implementation in iGibson is substantial. While CoRL may not be the perfect venue for such a benchmark since it's currently fully in simulation, I feel the benchmark takes meaningful steps towards "Embodied AI" and eventual physical agent transfer.
issues: Questions in order of importance

Q1: What is the motivation for using sample-inefficient RL techniques only in section 7 for evaluation? With human-level demonstrations via VR, doesn't teacher forcing stand out as a cleaner and more straightforward starting point? It's somewhat unsurprising that performance is zero (Table 2 row 1, most of row 2) for RL.

Q2: While full physics simulation is touted as an important part of the instantiation for the dataset, section 5 indicates the action space is discretized into 6 actions which take object targets directly [e.g., 'open fridge']. Why use a full physics simulation if inference will be reduced to a semantics-level action space?

Q3: For Efficiency metrics, why is Displacement of Hands calculated only while in contact with an object? In particular, that metric will miss the planning time and efficiency needed for grasping the object / planning to make initial contact.

Nits (not effecting evaluation):

- The backronym title is too painfully forced; I feel like you'll mostly get backlash in reception for the awkward "O" and "R".

- Line 28 stray right paren

- Line 33 "to accommodate" -> "from accommodating"

- Line 52 "comprise of" -> "comprise" or "are composed of"

- Line 89 no closing parent for open left paren

- The close similarity of using BDDL to instantiate possible environments with ALFRED's use of PDDL to instantiate environments and create automated plans towards goal conditions could be mentioned.
review_title: null

Meta-review: **Update after rebuttal**
I thank the authors for their thorough responses, my concerns/questions have been addressed (and the reviewers concerns as well). I recommend accept.

**Initial meta-review**
**Summary**: 
This work proposes a benchmark for every day household activities/tasks, as well as presents a dataset of human (VR) demonstrations that show how to perform these tasks. The proposed framework is independent of a specific simulator, and in this work has been instantiated via iGibson. The authors identify the following contributions 
* a high-level PDDL-style language which enables the generation of arbitrary many instances of a task
* Identification of environment-agnostic functional requirements, such that the proposed framework can be instantiated in any environment/simulator that fulfills these requirements. A specific example is given through instantiation in iGibson
* Developing a comprehensive set of metrics for consistent evaluation across various tasks. Specifically a set of metrics relative to human demonstrations (collected in VR) are proposed. The human demonstrations can also be utilized for evaluating imitation learning.

**Strengths**:

* This work addresses an important and very timely problem of benchmarking and evaluating the growing literature on embodied AI research. All reviewers agree that this work could have significant impact.
* Reviewers appreciate the complexity of the proposed proposed benchmark household activities and the engineering effort that must have gone into setting this up as well as combining it with human demonstrations. Furthermore it is appreciated that the proposed framework is inherently simulation independent 
* The reviewers mostly agree that this manuscript is well written and easy to follow 

**Weaknesses**:

* One of the claimed contributions are the PDDL-style language for describing tasks. However, this manuscript does not delineate how it differs from other PDDLs. While the authors cite one such work, the authors merely mention that the one proposed in this work is different from the cited one. As one of your main contributions this part of the related work needs to be discussed more thoroughly. A lot of work exists in this area and at least some of it should be referenced. A good entry point may be “A Review and Comparison of Ontology-based Approaches to Robot Autonomy “ https://upcommons.upc.edu/bitstream/handle/2117/179204/2019_KER_A_Review_and_Comparison_of_Ontology-based_Approaches_to_Robot_Autonomy_OlicaresEtAl.pdf
* While the reviewers agree that the manuscript is easy to follow, they have  also identified several places that lack important details. The authors should address these questions/lack of details mentioned by the reviewers
* A big question is the question of realism. While realism is addressed in the manuscript, it only addresses high-level realism (at the level of task description). The paper lacks a clear description of what level of realism is present in the execution of each of the subtasks. For instance, what does grasping look like? Especially for a conference like CoRL it would be important to discuss how well results obtained simulated benchmark would transfer to the real world.

Decision: Accept (Poster)