Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper posits the hypothesis that language abstractions are superior to raw states to guide exploration motivated by intrinsic reward. 

The authors extend two recent and performant algorithms, AMIgo and Nove1D, to use language abstractions instead of states as a means of computing intrinsic rewards. 

They show on a suite of experiments that, indeed, these language-based algorithms outperform their state-based counter-parts thereby supporting their hypothesis. 

Moreover, while previous works using language as an abstraction for exploration have focused only on comparing their methods to vanilla RL, here a useful systematic study of the performance of language abstraction vs. state based intrinsic reward exploration is presented.


strengths_and_weaknesses: **Strengths**

1. In terms of quality and soundness, I cannot find much to fault with the paper. The claims of the paper are clearly stated and quite strongly supported by empirical results (v. Figure 3). That is, indeed language abstractions appear to be better than state as a basis for generating intrinsic reward. 
2. In terms of clarity,  the paper is largely clear and concise. It is easy to follow the development of the logic throughout. The authors carefully provide all necessary details to understand the development of their new algorithms. Moreover, all figures, captions, and all plots in general are clean and easy to understand.

**Weaknesses**

1. In my opinion, the originality of this paper is limited. It could be argued that replacing state as the means of computing intrinsic reward with language amounts to simply changing the input space of AMIgo and Nove1D. I do recognize that some innovations are required for the adaptation of the base algorithms to language abstractions (e.g., the grounding network), but I am not convinced that these are large novel steps.
2. Notwithstanding the above, if there was a method to develop language abstractions for environments in general, the applicability of the algorithms developed in this paper may be rationalized. However, the L-variants are limited to environments which come pre-packaged with language abstractions thereby limiting their scope. In fact, in Zhang et al. [1], Nove1D is flexibly used in environments which have language abstractions (MiniGrid) and which don’t have language abstractions (Atari), while L-Nove1D cannot be used as flexibly.

Overall, my main contention is that the originality and novelty are somewhat limited.

[1] T. Zhang, H. Xu, X. Wang, Y. Wu, K. Keutzer, J. E. Gonzalez, and Y. Tian. NovelD: A simple yet effective 478 exploration criterion. In Advances in Neural Information Processing Systems, 2021.
questions: 1. In L-AMIgo, I’m curious about the impact of the “grounding network”. Is it critical performance? Did you by any chance do ablation study on the components?
limitations: In Section 8, the authors are up-front about the fact that the L-variants described in their paper is so far only applicable in environments that come with a language "oracle". Indeed, this seems like a significant limitation that perhaps explains why most literature focuses on state based measure for generating intrinsic rewards.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes the use of _language_ to improve exploration via intrinsic rewards. The paper proposes two methods that make use of sentences describing the goals in particular states.
strengths_and_weaknesses: 
Strengths:

- The paper is very well written, and clearly motivates the problem and solution.
- Exploration in RL is an ever-important problem, and the paper proposes a solution that improves upon state-of-the-art methods (but with some limitations, like the assumption of an "oracle language annotator")
- An impressive set of results in two significantly hard environments (however, I have some concerns about the experiments)

Weaknesses:

- L-AMIGo and particularly L-NovelD seem to be significantly similar to their respective base algorithms, and just offer a way to use state information in the form of language.
- L-NovelD uses NovelD(s_t, s_{t-1}) (eq7) which I find slightly concerning. I would hope that NovelD(l_t, l_{t-1}) would give a strong learning signal by itself, but as the authors explain it does not lead to equally high performance. This ties to my main concern (below) that the two methods mostly benefit from the (extra) data provided by the oracle and not because they are by themselves enough to provide an intrinsic reward.
- My main concern is that the baselines (AMIGo/NovelD/IMPALA) do not use the information used by their L- extensions. After reading the paper, it is still not clear to me what the main contributing factor to the improved performance is (also see questions below).

I have enjoyed reading this paper, but I believe it needs to address the issue of my last bullet point. Even a comparison to baselines that at least make use of the generated sentences in a naive way would be somewhat useful.

I would be open to increasing my score if the authors give a satisfying answer to my concerns.

** Update after rebuttal: This reviewer has increased the overall rating from 4 to 7. **
questions: - Do the baselines make any use of the descriptive sentences in the environment? Is it part of the environment state? As mentioned above, I am concerned that the increased performance stems from the high quality of the language annotations and not from the methodology itself. Could the authors elaborate on this?
- In Discussion (Sec 8), the authors mention that it might be possible to lift the restriction of an oracle language annotator by using a learned state description. Instinctively, I would say that since this is used as an intrinsic reward, such a model would require multiple visitations to that particular state anyway and would lose its novelty along the way. Am I misunderstanding something? If yes, such a result would be truly impressive and an important contribution.

limitations: Yes, the authors have addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The work proposes that the presence of (consistent) language, even if task-irrelevant and noisy, can provide a signal for exploration. They evaluate against two intrinsic exploration methods (Amigo and NovelD) by creating language abstraction variants of these methods. They run a thorough evaluation on mini-grid and minihack and find that exploration around language-based abstraction are indeed helpful. 



strengths_and_weaknesses: Strengths:

* Related work is thorough and fairly clear.
* The methods are clear and clean extensions of prior work.
* The results are thorough and well-organized


Weaknesses:

* Related work is missing exploration based on state abstractions
* The contributions of this paper could be clarified

More on contribution: 
The main finding is that exploration around language/state abstraction helps in large state spaces. We know that language abstraction helps in exploration--the authors reference many papers on this. In terms of state-abstraction helping in exploration, I think we also know this [1, 2]. The 1-hot id description conditioned student agent is like a parameterized skill that is being taught to go to abstract states. That said, I don't think state abstractions have been combined with these newer methods for exploration (Amigo/NovelD/etc.), so I think a discussion on this would clarify the contributions of this paper for me.



[1] Discovering Options for Exploration by Minimizing Cover Time
[2] Identifying useful subgoals in reinforcement learning by local graph partitioning
questions: Interesting paper. Some points:

In your intro, you state that state-based exploration would reward going to every room but the agent should stay by the stove. Changing the objects on the stove achieves novel states so I'm not sure your example is correct. If we had access to ground-truth state, state-based exploration might very well lead to exploration by the stove. While I agree with your goal of using language, I don't see how what you said really supports language.

I could but wrong but it feels like you're using "state" in multiple ways: e.g., the observation the agent sees and underlying latent state that's generated the observation. Am I wrong about?

It seems a bit restrictive to also have an oracle grounding classifier. Do you ablate this? 

Figure 5 has very similar colors. Overall, one-hot performance seems quite similar to using real-language. Does this change the story/message of your paper? The results are more-or-less the same indicating that it isn't necessarily language that's the driver of performance. Given that, I think your paper conclusions are not as well aligned with your story as you portray in the introduction. Don't get me wrong, I like this paper and think it's fairly thorough and clear but I think the findings don't align with the message. Could you clarify on this point?

The authors mention that there are several prior methods which use language for reward-shaping but they don't compare to any. How come?

Comment:
* Figure 6 has a lot going on. Challenging to make sense of.

Suggestions:
* Figure 2 is not as clear as it could be. Recommend describing it a bit more.
* Label "large" vs not large environments in Figure 3.

limitations: The limitations described seem adequate.
review_title: null

Meta-review: metareview: This paper studies an interesting problem, and overall the reviewers agreed the exposition and validation are sufficient, although there are minor concerns about novelty. However, the work is clear and studies an interesting idea for the RL community. We encourage the authors to consider the issues raised by the reviewers and further improve the work in the final version.

confidence: Certain
Decision: Accept