Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents a text-based robot instruction following benchmark, where the instructions have different levels of ambiguities, and the agent needs to resolve them via understanding both human intention and the physical environment. 

There are existing work that also consider ambiguity in human instructions, but this work proposes a systematic way to create and control the level of ambiguity, via sampling based on the utility and cost of sub-tasks, and also based on the trade-off between precision and cost in instruction language.

Also, this benchmark is based on the tasks in BEHAVIOR-100, which has good variety in task types, object categories and actions, not a toy environment.

strengths: - The proposed benchmark focuses on very critical topics in human-robot interaction: the trade-off between informativity and communication cost in human language, and the challenges in understanding human intention and resolving ambiguities.
- This paper presents a systematic way to create and control the level of ambiguity in instruction following tasks. And this approach seems can be easily generalized to wider applications.
- This paper provides great details and explanations about the creation process of the benchmark, which are beneficial to researchers in this field.
weaknesses: - While this paper presents an interesting benchmark, but the experiments are not so informative. The heuristic model outperforms both learned models, while it simply looks for object categories that are involved in human action history, without understanding of human goals.
- There is no further analysis on how important it is for the robot to understand human goals in task level 2 and above.
- It's unclear how good is the proposed quest generation process. Intuitively it makes sense to mimic human behavior in communications. A further analysis of data quality would be more convincing, e.g., via comparing with real human judgements.
correctness: The data set is constructed in a sound way.
clarity: Writing is clear with plenty of details for most parts.
relation to prior work: Yes.
additional feedback: None
review_title: A novel text-based instruction following benchmark. The creation process of this benchmark is very interesting.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a new robot learning environment for instruction understanding and following within physical and social contexts. Compared with current existing methods that focus on the language grounding, the paper added instruction understanding and following. The problem is formulated as a Markov decision process in the textual world. Benchmark results use a random agent, a heuristic-based agent, a sequence-to-sequence model, and  a DRRN RL model. The reported results indicate that the new robot learning environment with consideration to instruction understanding and following in physical and social contexts is challenging and the heuristic model showed the best average performance in the fully-observable setting while the sequence-to sequence showed the best average performance in the partially-observable setting.
strengths: 1. The HandMeThat problem is interesting and well formulated. I think it can provide a useful benchmark platform for comparing RL based models in a (limited) scenario.
2. Benchmark results using RL methods and other non-learning based alternatives are provided.

weaknesses: One limitation about this paper is that human tests are missing during the comparison experiments. I assume this would be difficult to collect data from humans, but it would be nice to see if humans are able to do this at all. Within a limited time, it seems like even humans might find those tasks too difficult. I feel that learning based methods also did not prove the potential under uncertainty in the reported results.
correctness: Seems correct.
clarity: Fairly clear.

relation to prior work: The paper discussed and compared with the prior work sufficiently.

additional feedback: Nothing particular.
review_title: An interesting problem

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposed a new dataset and a text game to resolve instruction ambiguities for indoor task planning in HRI setting.  The robot is supposed to understand human utterances and propose proper actions to help humans. The tasks are from behavior challenges. However, all actions are operating in the state space through a textual game setting. 
strengths: 1. It’s very important for robots to understand human instructions under uncertainty. 
2. Paper is generally well-written for readers to understand the main contributions. 

weaknesses: 1. The proposed setting is too simplified for the real HRI use-cases. Humans use ambiguous language instructions, often accommodating those with non-verbal communication. For example, pointing/eye gazing.
2. The proposed benchmark operates in a text game. This setting is analogous to state-space,  which does not have any ambiguities. However, this benchmark aims to resolve ambiguities in human instructions. It seems this setting kind of defeats its own purpose.
3. The proposed ambiguities in the benchmark are quite narrow and limited. They are essentially hand-crated by the author. I am unsure if they truly represent the ambiguities in real HRI settings. The only justification authors present is that these templates are from behavior challenges. However, behavior challenges also have limitations (discrete states etc)



Resolving ambiguities in human instructions is very important for the HRI community. 
However, there is a lack of reasonable justification on how this proposed text-game setting will benefit HRI.  From my perspective, real-world HRI has much more than textual settings. 


correctness: Yes
clarity: Generally clear
relation to prior work: Yes
additional feedback: N/A
review_title: Bechmark for HRI ambuguity 

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper proposes a benchmark and a textual environment for instruction understanding and following.
It consists of a set of "episodes", where each episode is a trace, human's goal, and two pieces of text.
The first one corresponds to the actual meaning of the human request from the agent and the second one is the text of the ask given to the agent. 

strengths: - Originality:  While there are many similar datasets/benchmarks, none provide exactly the same data.  
- Quality: The paper is not making any claims, so it can be considered technically sound.
- Clarity: The writing can be improved (see more details below)
- Significance: The authors are not successful in making the case for the need of such a benchmark, so I would consider the benchmark as possibly nice to have.

weaknesses: - Originality: There are many similar datasets/benchmarks. Instead of providing another benchmark, it seems to be possible to just augment the existing ones (e.g., ALFworld) with the missing text.

- Quality:
    - The paper is not making any claims. There is no theoretical justification for the work done. There are 4 level of tasks created. What are the properties of the resulting tasks? What complexity classed do they belong to? 
    - The notation used through the paper is unclear and quite informal. It seems like you could mostly base your notation on the existing notation from classical planning for lifted planning tasks. 
I would suggest to take a look at, e.g.,
Malte Helmert, Concise finite-domain representations for PDDL planning tasks, AIJ 2009.
If you use planning notation, you can map your resulting problems to existing planning formalisms. Once you do that, you can use existing results from planning literature on the complexity of your problems.

- Clarity: The writing should be improved (see more details below)

- Significance: You need to better justify the need for such a benchmark. Current text does not show how this benchmark helps overcoming the difficulties in communication between humans and non-human agents. The experimental results indicate that the non-trivial levels are too difficult, and it is not clear what exactly that difficulty stems from. Is it possible that you are trying to overcome too many issues at once?
correctness: Not relevant.

clarity: The paper is not too clear. There is sometimes a confusion in text between human and non-human agents, since the word "agent" is used for both.
Some of the language sounds strange. For example, putting sliced fruit in a bottle? Sometimes you use "jar" and that might be a better choice. Frankly, I would just say a container.  


relation to prior work: Related work section indicates the differences from prior work, as perceived by the authors. However, this is done by operating with concepts that are not clearly defined (no references either), and therefore it is not clear what is the meaning of these concepts. Social context, social reasoning, etc., what is the meaning of these concepts in this context? The citation of the book on the intentional stance is not a useful source of information in this case.
It does look like ALFRED and ALFworld could easily be augmented with the same information...

additional feedback: Questions to the authors:
1. Could you talk more about the need for this particular benchmark? How would the community benefit from it?
2. Why yet another separate benchmark? Why not, e.g., augmenting ALFworld instead?
review_title: Not sufficiently well motivated, too informal

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper introduces HandMeThat, a benchmark that evaluates human instruction understanding and following in physical and social contexts. It contains a large set of trajectories in which the human first attempt N steps to accomplish a household activity from BEHAVIOR, and then it queries the robot for help in an ambiguous way. The robot needs to resolve the ambiguity based on physical and social context and then interacts with the environment via a textual interface. The authors evalaute several baseline methods on HandMeThat and show significant room for improvement.  
strengths: The HandMeThat benchmark builds upon the natural distribution of human household tasks from BEHAVIOR and further enhance it with more distinct tasks (with object and object relationship randomization) and actions that change object states/relationships in the symbolic space. The latter is missing from the original BEHAVIOR work. The main differentiator of this paper seem to be the requirement to disambiguate human instructions based on physical and social contexts, which is quite unique and can be a solid contribution to the community. The difficulty level definition in Section 3.4 is also very interesting and presumpy people can use differnet partitions of the dataset for different research purposes.

weaknesses: One weakness of the benchmark could be it’s not actually physically grounded, even though it claims to be. The trajectories in the benchmark seem to be generated purely in the symbolic space.

correctness: The dataset/benchmark seems to be constructed in a very sound way. The supplementary material (appendix) has a very comprehensive explanation of how the dataset/benchmark was constructed.

The experimental evaluation, however, seems to be rather lacking. For the seq2seq baseline, it’s pretty unintuitive that the model performance in the partially observable scenario is better than the fully observable scenario. Although the authors attributed this to the longer text contexts in the fully observable scenario, this seems to a pure optimization issue that can be significantly alleviated with proper tuning. Also, the RL-based DRRN model completely fails even for the simplest difficulty level. The authors attribute this to the sparsity of the reward. A simple thing worth trying is to do a bit of reward shaping (since the optimal plan already exists in the dataset) and encourages the RL agent to follow the optimal action sequence. For a benchmark to be widely adopted in the community, a well-tuned set of baselines that leverage the current SOTA methods is essential for users to get started. 


clarity: The paper is relatively well-written, although some of the equations / concepts (e.g. action cost, cost of specifying the subgoal, etc) are difficult to understand until I also read the appendix.

relation to prior work: The paper covers four areas of related work: household manipulation tasks, social reasoning, pragmatic inference, text-based reinforcement learning. For household manipulation tasks section, the paper is missing a citation of iGibson 2.0, which BEHAVIOR is built upon. For text-based reinforcement learning, it’s missing citations of many other text-based games/environments, such as LIGHT [1]. In fact, HandMeThat is more similar to Textworld and LIGHT than to BEHAVIOR or ALFRED (which have a full-blown 3D simulation environment underneath). The authors are advised to do a more through literature review of other text-based games/environments and explain why HandMeThat (in the household activities domain) should be preferred over others. 

[1] Urbanek, Jack, et al. "Learning to speak and act in a fantasy text adventure game." arXiv preprint arXiv:1903.03094 (2019).

additional feedback: In the appendix section B: demonstrations: it’s mentioned that “we do not generate the demonstrations by PDDL planner due to the time cost of running PDDL planners on a large environment. By contrast, we directly generate a (potentially sub-optimal) solution by hand-coded policies.” Two questions: 1) since it’s an offline dataset, why is the planning time cost a concern? 2) could you give a concrete example of a sub-optimal solution? The pick-and-place example that is included seems pretty optimal.
review_title: Interesting text-based benchmark of human instruction following and symbolic planning; room for improvement for motivation and experimental evaluation

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper looks at the problem of human-robot communication. The major contribution is a benchmark building on top of a text-based environment that evaluates the agent's ability to finish a certain task for the human given the instructions and initial human actions. 

The paper look at an important problem of service robots assisting humans with their goals, and propose a new task involving goal inference based on both explicit and implicit communication. The paper is generally well written and the authors give sufficient details on the task design and dataset generation. These are the strengths of this paper. However, there are several significant weaknesses: the text-based environment overly simplified the problem, the settings are not justified and the literature review is not complete. Thus I'm reluctant to recommend acceptance.
strengths: Strengths:
- The proposed benchmark looks at an important problem of service robots assisting humans with their goals. To infer human intents, the paper proposes a specific task which involves agent reasoning based on implicit observations and explicit communication signals. 
- The paper is generally well written and technically sound. The authors provide sufficient details in the paper to help readers understand how the benchmark is designed and how the dataset is generated. 

weaknesses: Weaknesses:
- Most importantly, I believe the text-based environment over-simplifies the problem. Many factors that prevent the agent from predicting human's goals and intention mainly exist in the visual domain (e.g. occlusion, detecting interacted objects). The textual setting fails to consider these factors. 
- Some task settings are not justified and the application is not clear. I understand that due to communication cost, the human's utterance can not completely specify the subgoal. However, during human communication, most of the time the speaker should be in favor of accuracy rather than brevity. "Hand me that" is a rather rare case for human communicators, only occurs when the common ground is clearly established. Since adding a few words can greatly reduce the search space of the listener, It is not clear whether such a tradeoff are necessary and meaningful in reality. 
- In terms of literature, some relevant references are missing and the comparison is baised (see below). 

correctness: Correctness:
- I understand the cost of subgoals are incoporated in the utterance generation to reflect a human tradeoff between communication cost and information precision. However, the cost term also exists during the meaning generation. How does an accurate sub-goal specification reduce human's internal reward? 
- It seems the values in the table of fig3(c) do not align with the equation in the main text. 


clarity: The paper can benefit from a revision to improve clarity:
- Since the proposed benchmark is mainly based on textual environment, the images displayed in figure 1 cause misunderstandings for the readers. The authors should consider adding a note in the caption for clarification. 
- More explanations are needed for section 3.3. In line 200, the authors state the assumption that the listener knows the internal goal of the human. Isn't the goal inference an essential part of the task? 
- The authors should explain how the values of hyperparameters (alpha, beta, k) are chosen and how they affect the results. 

relation to prior work: - Some relevant references are missing. The authors are recommended to check and add the references below:

Gao, X., Gao, Q., Gong, R., Lin, K., Thattai, G., & Sukhatme, G. S. (2022). Dialfred: Dialogue-enabled agents for embodied instruction following. arXiv preprint arXiv:2202.13330.

Padmakumar, A., Thomason, J., Shrivastava, A., Lange, P., Narayan-Chen, A., Gella, S., ... & Hakkani-Tur, D. (2022, June). Teach: Task-driven embodied agents that chat. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 36, No. 2, pp. 2017-2025).

Jain, U., Weihs, L., Kolve, E., Rastegari, M., Lazebnik, S., Farhadi, A., ... & Kembhavi, A. (2019). Two body problem: Collaborative visual task completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6689-6699).

- In addition, I believe the section 2 and table 1 are biased: a majority of benchmarks mentioned here focus on learning from visual information while this paper looks at low-level textual information. The authors should acknowledge this point in the paper.
additional feedback: N/A
review_title: A benchmark based on textual interface for human-robot communication

Meta-review: metareview: The authors presented a new text-based benchmark environment where agents are challenged to follow ambiguous instructions to help humans achieve their goal. 

### Review Summary

#### Strengths:

- Almost all reviewers agreed that the paper provided an important novel contribution by focusing on a benchmark for collaborative agents working under ambiguity. 
- Several reviewers praised the solid construction of the benchmark, in particular the utterances and sub-goal generation.  
- Some reviewers also praised the clarity of the paper, particularly the description of the generation process, noting it would be useful to other researchers. This clarity was further improved by significant revisions, including a more formal problem statement and description and a discussion of the different challenges posed in this task.

#### Weaknesses:

- Several reviewers criticized the experimental section, often specifically requesting additional details of human performance, and better tuning.  These were added by the authors in a subsequent revision, along with revised results for Seq2Seq model with improved tuning, and discussion of the heuristic model performance.
- Several reviewers also criticized the environment for being too simple, being a text-based environment.  This was met with a convincing argument about the value of separating visual understanding from pragmatic reasoning and planning. This ‘simplicity’ argument was also undermined by the large performance gap learned methods had compared to human performance. 
- One reviewer engaged very thoroughly with the paper, criticising its lack of formalism, interrogating the problem statement and questioning its contribution to the wider academic landscape. The discussion that ensued proved very illuminating, with substantial revisions and discussions that addressed many of these issues, though did not satisfy the reviewer enough to recommend acceptance.

### AC View

Following the reviews, discussions, and substantial improvements made to this paper I have decided to accept this paper, on account of the generally accepted importance and novelty of this work and the solid construction and clear presentation of it.  The revisions made to the paper during the discussion period made a _significant contribution_ to this decision, not least in greatly improving the clarity of the work and helping place it in the wider academic context. Indeed, this process prompted several other reviewers to raise their scores. 

This view reflects the majority of reviewers. While one reviewer recommended a narrow rejection, after a very thorough and very useful review, I did not find their remaining criticisms sufficient to reject the paper.

confidence: 3: The area chair is somewhat confident
Decision: Accept