Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper explores a simple, elegant idea: the use of natural language as supervision when performing instruction following tasks. Centered around an evaluation of various tasks in the BabyAI complex grid world suite, as well as the Crafting environment introduced by Chen et. al (2021), the proposed approach takes an easy-to-digest tact: eat a sequence of observations from the environment with a Transformer, and predict two things: first, given a goal, predict the action to take (a traditional behavioral cloning objective). Second (and a core contribution of the approach) treat language prediction as an auxiliary task, predicting a sequence of language instructions that specify how to perform the task (which I liken to “subtasks” or some compact language describing an immediate outcome to optimize for, though the authors should please correct me if I’m misinterpreting!).

These language assumptions are assumed to come from an oracle (concretely, in the implementation, generated synthetically exploiting the simulated nature of the environments), though the authors argue that this sort of “guidance instructions” appears naturally in the wild; videos have captions, humans could provide this feedback cheaply, etc.

The bulk of the evaluation evaluates sample efficiency and performance on the long-horizon tasks in BabyAI/Crafting, comparing the proposed approach to several meaningful ablations (no auxiliary objectives, a hierarchical variant, a self-supervised contrastive variant, and the proposed approach). These baselines are incredibly well thought through, and serve as valuable comparison points for the proposed approach. 

The final, perhaps salient point of this work is the data regime; the main body of the paper and the appendix both argue that the true value of this approach (language as auxiliary tasks) is when (1) we are operating in a moderate data regime — too few data, and we can’t learn decent behavior, too much data, and we can just learn tasks without additional supervision, and (2) when the supplemental language information is providing something new, over the goal information itself. Experiments on the ALFRED suite show that because intermediate language instructions can be predicted from the goal itself, they don’t have inherent value. 
main_review: I believe this to be a well-written, strong paper, with an elegant and simple approach. The baselines go above and beyond, and though the evaluation is limited to BabyAI and Crafting simulated environments in 2D, I understand how these environments can be complex and multi-faceted, and serve as excellent testbeds. 

I think the argument regarding the data regime is nuanced, but very very interesting, and is worth sharing with the community as well.
However, the one weakness I have is a pretty big one, and one that I hope the authors can discuss during the rebuttal; the availability of the intermediate language instructions (those used for auxiliary language prediction), and ensuring these intermediate language instructions are of the structure that’s necessary for the approach to work! It seems to be the case that these “auxiliary” instructions need to be pretty distinct from the goal, spelling out explicit subtasks, that cannot be directly inferred.

I’m curious where the authors think this data could come from when scaled up; when I look at the suggestions in the paper (exploiting PDDL planners in simulation environments — this seems debunked by the ALFRED experiments, or video captions — unclear that these have any instructional signal at all), I don’t really see the broader applicability of this approach. This feels like a strong, perhaps breaking assumption, which is really coloring my initial review.


Typos/Style/Questions:

Page 4: Langauge —> Language 
summary_of_the_review: In general, I believe the proposed approach to be simple and elegant; the arguments about the data regime are well argued, and the evaluation is precise. However, my biggest concern has to do with the viability of obtaining the requisite auxiliary language instructions — I’m not convinced that this data can be easily obtained in the wild, which puts the value of this approach into question.

I sincerely hope that the authors can engage with me on this point!
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the usage of language prediction for imitation learning. The authors distinguish between a goal (a short description of the task) and an instruction (a detailed description of subtasks that span a sequence of observations). They show that predicting instructions at each step as an auxiliary loss improves the performance using lesser number of demonstrations. They experiment with minigrid and crafting environments to show the efficacy of the proposed approach.
main_review: I like the overall problem and conclusion that predicting detailed instructions can improve performance using much smaller data than without this auxiliary objective. I think the perspective that labeling subtasks with language instructions is more sample efficient than collecting more demonstrations is also useful. I have some questions about the approach.

1- Could you explain if "instruction as a text" or just "objects and actions in instructions" are more important? Based on Figure-3, it seems that each instruction includes an object and an action. It would clarify if your auxiliary objective is helping just the object detection, subtask understanding, even using OOD information that wouldn't otherwise be in the goal (such as a grey door is never seen in a goal but it is a part of navigating the environment), or if actual text is also important.

2- What is the main bottleneck that instruction generation is solving? Similar to my above question, it could be object detection or subtask understanding but might also be that understanding a textual goal is difficult. A pre-trained language model for goal encoding would help answer this.

3- To understand if detailed instructions are really crucial for instruction generation, an auxiliary loss baseline where goal is predicted at each time step is needed. It would help understand if text generation or detailed instructions are the most important; even for POMDPs, partially generated goals might be helpful.

4- You use only observations and goal as input to the transformer encoder but DecisionTransformer [1] showed that using action and reward was crucial for success. I think a similar baseline with and without language prediction is needed.

5- In Figure-2 and Figure-6, decoder outputs should be shifted by 1 but right now the same input is outputted at each step.

6- In Table-1, you put "=# Demos", was this intentional or a placeholder for an exact number?

7- Some grammatical errors:
* In Section 3.1, "works with our without" should be "works with or without".
* The same section, "langauge" should be "language".
* Section 3.2, "depcited" should be "depicted".
* The same section, "except it the encoder" should be "except the encoder".
* Section 4.1, "to an agents initial" should be "to an agents' initial".
* Section 4.2, "Prior work has not use" should be "Prior work has not used".
* Appendix A.2, "AFLRED" should be "ALFRED".
* Appendix B, "to select a instruction" should be "to select an instruction".

[1] Decision Transformer: Reinforcement Learning via Sequence Modeling. Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, Igor Mordatch.
summary_of_the_review: I think instruction generation as an auxiliary loss is an interesting approach and presents a new perspective that labeling subtasks with instructions is better than just collecting more demonstrations. I also think that some more baselines and clarifications are needed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors explore the use of language as auxiliary supervision for long-horizon imitation learning tasks. The authors show that using instruction modeling is able to improve the performance in planning environments when training with a limited number of
demonstrations on the BabyAI and Crafter environments. They further show that instruction modeling is most important for tasks that require complex reasoning in the experiments. 
main_review: The novelty of this paper is limited. The authors use language as an auxiliary task to improve planning accuracy. Such a method has no technical contribution. I would like to see the summarization of the technical contributions from the authors in their feedback. 

The experimental results are not very interesting. It is not surprising that using extra supervision will improve the results.

Could the authors explain the difference between the claimed "non-Markovian" and the commonly used "partially observable Markov decision process (POMDP)"? Why do the authors use "non-Markovian" instead of "partially observable Markov decision process (POMDP)" in the paper?

In the Crafting environment, the results of "Lang" are comparable with the results of "Xformer" in Table 2. Is the model of "Lang" also larger than "Xformer" as it has the extra language encoder and decoder? Could the authors give any explanation of why using extra supervision generates comparable results? 

summary_of_the_review: After reading the whole paper, I didn't find any interesting points that make me feel excited about this paper. I think the authors did good experiments to verify their thoughts, but this paper has no technical contribution and the results are not surprisingly good enough. 
This paper is easy to read and has no significant fault.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes using instruction modelling as an auxiliary objective to improve long-term planning. On BabyAI and Crafter, this method demonstrates significant improvement on longer hop levels (e.g. 3 hop BabyAI).
main_review: I think the core method of this work is quite simple (which is not inherently bad and I would even argue is inherently good). The authors add a auto-encoding objective to a transformer policy network, where during each step, an additional decoder ingests 1) the instruction for that step and 2) the observation encoder output for all previous steps and recovers the instruction for that step. This reconstruction loss is weighted with the usual policy loss and jointly optimized. On the BabyAI and Crafter experiments, this method achieve strong gains using fewer demonstrations (for imitation learning) and converges faster and to higher performance. I think it is quite clear that this method is very helpful for the tasks studied in this work.

What I find lacking is how this comparison compares to the general claim that instruction modelling is useful in the more general sense (although I hope it is and would like to believe it is). I think what would significantly improve this paper is to study instructions with more complex behaviour which truly suffers from the lack-of-annotation problem. There are a number of such environments such as Touchdown (navigation with streetview panoramas, https://arxiv.org/abs/1811.12354) and DRIF (quadcopter control, https://arxiv.org/pdf/1910.09664.pdf), where due to the complexity of the trajectories and observation space it is prohibitively expensive to annotate language instructions. This would evaluate this technique on complex language instructions. Another way to go is to do more complex grid worlds with high stochasticity and more complex dynamics that actually require detailed understanding of the language because test language-entity associations are unseen and must be constructed from language. Examples here include RTFM (7+ hop reference resolution, https://arxiv.org/abs/1910.08210) and Messenger (NL instructions, small training distribution, https://arxiv.org/abs/2101.07393). These are also good related reference for the paper, which this draft doesn't include - some of the references this work does include would also be good test beds - because they also contain more complex grounding challenges due to visual observation complexity.

All of these previous works include dynamics and language instructions more complex that the ones studied in this paper. In particular, due to construction, once instructions are broken down into atomic steps, they don't even require sequential understanding. For example the instruction "go to the red door" is perfectly recovered with the binary features (goto, red, door). As a result, the experiments leave many questions unanswered, such as 

1) does this result generalize to complex environments where language-entity and language-dynamics associations are more difficult due to complex entity behaviour or complex observation?
2) does this result generalize to complex language instructions where the instruction is more difficult to model (and therefore to understand)?

Update because I looked at appendix:
The negative results from ALFRED further raises my suspicions that these results may not generalize. For the results in the Appendix: are the authors claiming that one can do ALFRED without visual observations? My understanding is that the ALFRED work shows exactly that you cannot do the task without visual observations (it is one of the ablations in their paper). Perhaps I am misunderstanding this result. Can the authors please comment on this?

I would imagine that the result shown here is because the model is incapable of modelling the instructions, and therefore the reconstruction loss is not helpful. In particular, I would think that there is a high correlation between being able to do the task and being able to reconstruct instructions from the bottleneck representation. If the instructions are very complex to understand, then both are hard and the reconstruction loss is not helpful. One way to test this hypothesis is to plot the instruction recovery accuracy instead of loss (the authors show the latter but not the former).
summary_of_the_review: Strong results that show instruction modelling help policy learning on two simpler tasks. Evaluation leaves much to be desired in terms of substantiating main claims.
review_title: null

Meta-review: The reviewers all consider the paper to be below the acceptance bar. While the revision addressed some concerns, several critical ones remain open. This includes empirical concerns with regard to the extremely simple grid-world environments used, and with regard to the vague distinction between instructions and goal specifications. To improve the submission, the authors should seek stronger empirical foundations, and either refine or remove vague distinctions with regard to the phenomena they aim to study.

Special thanks to the reviewers for an extremely productive discussion.
Decision: Reject