Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new reward shaping method, EAGER, in Language-guided reinforcement learning (RL).  By extracting auxiliary Question Answering (QA) tasks, the agent is given intrinsic rewards based on how the generated trajectories could help answer these questions.  The authors claim that such auxiliary tasks can be designed without engineer interventions.  Specifically, the answer to these questions is the output of the pre-trained QA transformer model after feeding the generated trajectories.  For the BabyAI experiments, this QA model is pre-trained using example trajectories provided by the bot that can already solve the environment. The method is compared against two recent baselines RIDE and ELLA.  EAGER improves sample-efficiency in some tasks from BabyAI.
strengths_and_weaknesses: Strengths: 
-This seems like a novel method that is inspired by reference-less metrics in NLP. The idea is interesting, and the authors provided detailed explanations on how each component, QA module, QA module, is implemented. 
- Overall, the paper is well-structured and clearly written from the method section to the experiment section.  
- I believe the future work could be influenced by the idea of reference-less metrics proposed here.

Weaknesses: 
- I think the experiment part is weak due to EAGER still using “engineer intervention” in BabyAI.  If I understand correctly, RIDE [1] does not use any expert information at all while EAGER currently requires a bot that can already solve some of the tasks from BabyAI.  In addition,  QG assumes the answer to the generated questions is one of the words that appear in the goal sentence.  The output of the QA model also requires prior knowledge about the potential words that can appear in the goal sentence though this might be okay if you assume these  are all nouns and you could extract them in advance.  I recommend adding a table to list all the assumptions and expert knowledge required for RIDE, EAGER, and ELLA [2] in a table.  
- Although EAGER outperforms RIDE, RIDE does not use the bot from BabyAI so I think it’s not too surprising that EAGER can outperform RIDE all the time.  Since EAGER and ELLA both excel in certain tasks, it is difficult to judge whether the method is actually an improvement to ELLA given the reasons in the previous bullet point.  

Minor points:
- Title 5.2 “How does EAGER performs” → “How does EAGER perform” “Sparsity increase” → increases
- Line245: I think it’ll be helpful to explain what these 3 input values represent in BabyAI.  How are objects represented in the gridworld.  


[1] Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for 418 procedurally-generated environments. In International Conference on Learning Representations 419 (ICLR), 2020

[2] Suvir Mirchandani, Siddharth Karamcheti, and Dorsa Sadigh. Ella: Exploration through learned 406 language abstraction. In Advances in Neural Information Processing Systems (NeurIPS), 2021.

questions: - Is it possible to use behavior cloning to train an agent using those example trajectories from the bot?  This might serve a good baseline.

limitations: Yes, the authors have addressed the limitations
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a simple but novel reward shaping method for language-conditioned RL. The paper mainly proposes the auxiliary objectives for question generation and question answering models. Then they calculate the intrinsic reward which is proportional to its confidence in its answer. In the experiments, the authors showed that the proposed algorithm learns more efficiently on various tasks than the baseline algorithms.

strengths_and_weaknesses: Strengths
- The main idea of the proposed method in the paper is simple but novel. 
- The experiments include not only the results of sample efficient learning but also various analyzes.

Weaknesses
- A near expert dataset is needed for pre-training the QA module, which seems to be a quite strong assumption.

questions: - In lines 280-281, it is expressed as without using expert knowledge, but is the expert dataset not used for pre-training when learning the QA module? (The paper explains as if it used less information than the ELLA, but it requires the same or similar information as the ELLA.)
- If there is a pre-trained dataset (for the QA module), isn't it more efficient to use knowledge in the form of a prior policy learned from it? It seems that there should be an experiment about it.
- What if it is used simultaneously with intrinsic reward methods such as RIDE? It is also important to check whether the effect of both methods is still helpful even if they are used at the same time.

limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes an auxiliary reward scheme (EAGER) for instruction following RL (BabyAI framework) based on question generation (QG) and answering (QA). Concretely, QG is based on masked word modeling of the instruction, and QA is pre-trained with demo instructions and trajectories and fixed when training RL. The intuition is good and successful RL agent trajectories should make QA correct and easy, thus when QA is successful, an auxiliary reward proportional to correct answer confidence is given. The method is evaluated on several BabyAI tasks and shown on par with other methods (ELLA, RIDE) with more expert inductive bias for subgoal specification and judgement. Further analysis shows EAGER is fairly robust to QA performance, but some QA design choices are important.
strengths_and_weaknesses: Strengths: 

- The idea is neat, novel, effective, and provides a means to auxiliary reward for instruction following only using example instructions and trajectories, and minimal linguistic assumptions (what is noun/adj). In contrast, previous methods usually require domain knowledge about subgoal formulation/proposal/completion judgement. 

- Some design choices for QA are reasonably explained in theory (e.g. policy invariance) and practice (e.g. no_answer, using confidence instead of binary reward).

- The motivation and idea are well explained, and the paper is well structured and easy to read for most parts.


Weakness:

As mentioned in the last part of the paper, two main limitations (and some of my extensions) are 

- QA needs expert trajectories with instructions to pre-train. Arguably it is a weaker assumption compared to prior work, but I feel two important ablations are missing: direct PPO without the auxiliary reward (a trivial low-bound baseline), and just using the QA training trajectories to do imitation learning and warmstarting for RL agent (a trivial way to use these pre-training trajectories). 

- BabyAI is perhaps too simple, and experiments seem a bit limited. I can see some sample efficiency advantage on PutNext tasks, but ELLA seems better than EAGER on Unlock and Sequence. What about other tasks in BabyAI, let alone more complex tasks like Alfred (which I feel this method might not easily work, given the visual observation and language instruction are both much harder). Some more experiments or tasks would make me believe the method is more general and robust.

Also pointed out in the paper (line 295),'

- EAGER might not work well on Sequence or tasks with temporal subgoals. In general, I don't have an intuition what properties of env/task does EAGER work better on and what not. Some analysis or examples of agent behavior or even how auxiliary reward changes with time would give me more concrete ideas.

questions: 
Some minor points:

- Maybe in abstract, mention the domain (babyAI)?
- Is there any ablation for the policy invariance thing? 
limitations: They are discussed in the paper, see the weakness part.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In this paper, the authors proposed a reward shaping method for language-conditioned RL. The authors proposed to generate a set of questions based on natural language instructions, and then evaluate how likely the current trajectory answers the questions. The probabilities of the questions being correctly answered are used as intrinsic rewards. In the experiment, the authors worked on Baby AI benchmark and showed that the proposed reward shaping method helped to improve sample complexity and model performance.
strengths_and_weaknesses: Strengths:
The idea of using question generation and answering to do reward shaping for LC-RL is novel and interesting. It is like doing a self-check when a human follows a task instruction.

The paper showed that this idea works in the Baby AI benchmark.

Weaknesses:
While the idea is interesting, I think it is not sophisticated enough to generalized to more complex setting.

First of all, the QG module generates a questions by masking a word (such as nouns and adjectives) in the linguistic instruction. However, for complex instructions, it is likely that the question can by answered by the masked instruction. For example, an instruction "pick the [red ball], open the red door, ...., put the [red ball] on the chair". If we mask the 'red ball' at the beginning of the sentence, we can still infer the answer from the 'red ball' at the end of the sentence. Therefore, even without the trajectory, the questions can be answered. I'd assume that it requires more than just masking words to generate a valid question.

Secondly, the QA model is trained by using the full trajectory, however, when calculating intrinsic rewards, partial trajectories (trajectories up to the current timestep) are used. I would assume that when the partial trajectories are short, the QA model performance is bad. Have the authors investigated this?

questions: Section 4.1 mentioned that if a question is answered, then it is removed from the set of questions. How to determine if a question is answered? Does the QA model have to predict the correct answer with high probability?
limitations: N/A
review_title: null

Meta-review: metareview: This paper presents a language-guided auxiliary reward mechanism based on generating Q&A pairs based on agent trajectories and rewarding the agent for producing trajectories that yield correct answers from an answering model. The reviewers broadly found the paper compelling and convincing, and thus I am happy to follow their general consensus in recommending acceptance.
confidence: Certain
Decision: Accept