Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper suggests ELLA (Exploration through Learned Language Abstraction), a reward shaping approach for language instruction folllowing tasks. The contributions that the authors argue in this paper can be summarized as follows:
1. propose a reward shaping approach for instruction following, which consist of two main module, (1) termination classifier and (2) relevance  classifier
2. empirically show that the proposed method is effective by experiment on BabyAI environment, which is a representative benchmark for instruction following tasks.
main_review: This paper presents an exploration strategy of reinforcement learning agents to solve instruction following tasks. Since instruction following tasks are usually specified on a sparse reward setting, reward shaping can be effective for reinforcement learning. I think the most strong point of ELLA is a policy invariance property, which is usually not guaranteed on recent intrinsic exploration approaches. 

However, I have following concerns and questions for this papers:

1. Although this paper consider on an exploration strategy for language guided tasks, I think considerations of this approach seems task-specific rather than language-specific. The authors especially only consider hierarchical tasks where a high-level instruction (goal) is able to be decomposed into low-level instructions (sub-goals). 

2. In Table 1 in Appendix A, what are criteria of assigning low-level tasks for each high-level task? 

3. Can you clarify how is low-level instructions data (used in experiments) collected? Whether low-level instruction data is collect by human  demonstration or demonstrations generated by human-crafted oracle, I understand each low-level instruction dataset should be collected individually for each low-level task. I think this requirement is also too expensive for general applications. 
 
4. Figure 4 shows ELLA is sensitive with a hyperparameter selection in that inappropriate selected $\lambda$ can make learning failure. This also can be one of limitations of this paper.

limitations_and_societal_impact: Yes, the authors adequately have addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a reward shaping approach that provides immediate rewards through the achievement of low-level sub-tasks. The authors assume a problem where a high-level task given in the natural language in a sparse reward environment consists of several low-level sub-tasks, and aim to guide the agent to effectively explore using task structure. Two components (termination classifier and relevance classifier) are proposed, and reward shaping is performed using them. Finally, the authors demonstrated the results on BabyAI, a grid world platform for instruction following tasks. 
main_review: I agree that this is an interesting and important problem. However, I think the assumptions in the paper are quite strong, and I am concerned about whether the experiments shown in the paper demonstrate the effect of language abstraction well. Comparison with additional baselines should be included, and it seems that the experiment should be modified to better reveal the effect of language abstraction. The detailed comments and questions are as follows:

1. The baseline algorithms seem to be poor with PPO and LEARN. ELLA aims to effectively perform the exploration using language abstraction, then I think it should show advantages and improvements compared to existing exploration methods that do not use the information of language abstraction (ex. uncertainty-based exploration [1], curiosity-based exploration [2], or impact-driven exploration [3]). Some tasks were compared with RIDE in the appendix, but I think that the results on other tasks should also be mainly compared and addressed in the main text.

2. ELLA uses more information than RIDE (instruction set of subtask and human-labeling of termination states), but despite this, RIDE and ELLA show similar results in terms of sample efficiency and performance. The current experimental results do not reveal the advantages of ELLA compared to the existing approaches that do not use language abstraction information. It is thought that the generalization effect of language abstraction can be shown more meaningfully by conducting experiments on unseen tasks such as meta-learning or transfer learning settings. 

3. As a weakness of this paper, I am concerned with the following two points: 1) The assumption for termination states of instructions are quite strong. In the general case, it is very expensive to label a large number of data manually. 2) It seems that performance and sample efficiency are sensitive to $\lambda$ parameters.

4. (Page 9, lines 310-313) I don't understand how the process of calculating the $\lambda$ is done. How is $\lambda$ computed from step here?

5. (Page 8 lines 281-285) The authors explain why ELLA does not increase sample efficiency in a COMBO environment, but I don't quite understand what it means.

[1] Yuri Burda et al, Exploration by Random Network Distillation, ICLR 2019

[2] Deepak Pathak et al, Curiosity-driven Exploration by Self-supervised Prediction, ICML 2017

[3] Roberta Raileanu et al, RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments, ICLR 2020

limitations_and_societal_impact: Yes, the authors have adequately addressed it.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper present ELLA, a three-part solution to instruction following tasks with a high-level instruction and a pre-defined set of low-level instructions:

1. a termination classifier is trained offline to identify the completion of a low-level goal at a state;
2. a relevance classifier is learned online to predict a binary relevance of a low-level goal to a high-level goal.
3. reward shaping is used to encourage low-level goals suggested by the relevance classifier.

ELLA is shown to improve sample complexity on 5/6 BabyAI tasks of diverse environment and instruction complexities, compared to an ablation PPO model with reward shaping, and a baseline model LEARN.
main_review: Pros:

- The method is well-motivated for the considered task setup of three levels: high-level instruction, low-level instructions, and low-level actions. Such a setup is natural for many real-world language-guided tasks, since language is hierarchical. Directly mapping the compositional high-level instruction to low-level actions is sample-inefficient, and this is a fundamental problem in instruction following. To make use of a pre-defined set of low-level instructions, a method must
A. Link low-level instructions to low-level actions. This is done by the termination classifier providing pseudo-rewards, and reward shaping to affect the low-level action policy. 
B. Link high-level instruction to low-level instructions. This is done by the relevance classifier learned online.
 
- It is infeasible to learn both A and B online, so ELLA proposes to learn the termination classifier offline and the relevance classifier online, with the assumption that the former supervision is easier to collect. The consequent empirical results look reasonable. 

- The paper is well written with good figures. 

limitations_and_societal_impact: Cons/questions:

- The baselines might be under-explained and a bit weak. According to my understanding, the PPO baseline makes use of neither high-level or low-level instructions, and the LEARN baseline just uses the high-level instruction? More detailed setups are needed. And it might be a less surprising comparison if ELLA is the only model that leverages the low-level instruction set. See below for some suggested baselines/model variants.

- A major model design is to learn termination classifier offline and relevance classifier online. Is it possible to show the opposite, e.g. learn the termination classifier online, while learning the relevance classifier offline (possible since BabyAI is synthetic)?

- Another potential concern is, learning relevance classifier totally offline abandons the fact that instructions are languages with compositional structures. For example, a high-level goal “goto box and pick it up” can be easily decomposed to “goto box” and “pickup box”, even without the game context. Currently, for each game, a new relevance classifier is learned from scratch about such decompositions, and no language priors is used. I wonder if this classifier can be somehow pre-trained as well and boost performance. 

- Another model design is reward shaping. Is it possible not to use it, but directly learn a relevance classifier plus a low-level goal-conditioned policy? Some experiments are needed to justify reward shaping compared to such an approach without it. 

- I don’t totally get Section 6.2. “However, the large number of diverse language instructions makes it difficult to estimate good decompositions via our deduplication method, and so it is difficult to learn f ρ online and offer the most targeted low-level task bonuses.” So for the PutNext task, why sample efficiency is still improved even if the relevance classifier does not learns good decompositions? 

- The relevance classifier seems to have no explicit temporal order of decompositions. Is this a concern for more practical tasks, or is it solid because only one relevant low-level goal can be achieved any time? Some discussion is needed.


**After rebuttal**

Thanks to authors for clarifying my questions. The baselines are stronger than I thought, and it is reasonable to adopt some of my advice as future work. I raise the score to 7 and hope the paper can add/clarify discussion with respect to my questions.
review_title: null

Meta-review: The paper describes an approach to training an agent to follow instructions for tasks composed of multiple low-level subtasks. The method uses reward shaping, whereby the agent receives supplementary rewards when sub-tasks are achieved. The shaped rewards are determined using two classifiers, a "terminal classifier" that classifies the completion of a low-level subtask, and a "relevance classifier" that assesses the relationship between low-level tasks and the success of the high-level task. The latter is learned online without the need to break high-level instructions into their lower-level components. Experimental results on the the BabyAI task and a grid world task demonstrate gains in sample efficiency relative to recent baselines.

The reviewers agree that the paper considers an important and challenging domain. The problem of understanding natural language instructions has long been the focus of AI and robotics research, and has recently received renewed attention within the broader machine learning community. The idea of exploiting the hierarchical nature of the tasks as a means of encouraging exploration through reward shaping is reasonable and well motivated. The reviewers point out that the approach relies on several strong assumptions (e.g., involving the existence of terminal states and access to low-level instructions). Two of the reviewers also raise concerns about the baselines and the amount of information that they are provided with compared to ELLA. The author response helped to resolve some of these concerns, but the authors are encouraged to ensure that they are addressed in any subsequent version of the paper.
Decision: Accept (Poster)