Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper motivates the use of language abstractions instead of raw visual state representation for designing intrinsic exploration reward. As language can provide coarser and compact description of the environment, it can prevent unnecessary actions and meaningfully incentivize the agent to achieve the task. To this end, the paper proposes to use pre-trained models on image captioning datasets to enable better task-relevant exploration in 3D simulated environments. The experiments demonstrate that (1) random network distillation on oracle captions performs better than visual state representation and (2) semantics in the oracle caption matter, especially in longer horizon tasks. Furthermore, the use of ALM, CLIP or Imagenet embeddings based NGU or RND for exploration achieves faster task success as compared to using raw visual state representations. 
strengths_and_weaknesses: The paper is overall well-written and sound. The contribution in terms of experiments showing the value of using language based abstraction in modeling intrinsic reward in 3D scenes is novel and promising step towards exploration techniques for robot learning.
However, the success of using language based representations for exploration is not itself novel and somewhat depends on the kind of. captions and the task. This is reflected in Fig 6 and the authors also suggest experimenting with different captioning strategies as future research direction. 

questions: - The motivation why language enables wider exploration is unclear in Figure 1.
- Appendix is missing?
- In line 113, the policy receives … template-based “language” goal?
- What held-out goal settings, ie. in terms of unseen objects or environments considered in evaluation for different tasks?
- Can you clarify what ALM-ND (Text) and ALM-ND (Image) mean in Fig 6(c)? It seems that ALM-ND (X) is extracted from the pre-trained X encoder to use for network distillation and generate intrinsic reward.
- ‘Find’ task in Playroom seems very similar to the maximizing coverage in object navigation [1] Chaplot et al. How does the previous work compare as a baseline in terms of achieved coverage in held-out city environments?
limitations: The authors provide detailed limitations and future scope, namely: (1) comparing different ways of captioning, (2) fine-tuning pre-trained VL models on relevant datasets for interaction tasks, and (3) having pretrained models on multi-object scenes to improve performance without oracle captions. Extending upon (1), the proposed approach seems to have an objective way of captioning any state that is relevant to solve the task effectively, whereas natural language captions are context-driven, underspecified in terms of all the objects in the seen and mostly describing main semantic takeaway. This assumption should be noted before using the proposed method to design intrinsic rewards for physical robots task.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes to use semantic representation of the visual environment to guide the agent to explore novel scenes in 3D tasks.
 Either oracle description of the environment or vision-and-text representation from a large language model can be used to calculate reward to guide a Reinforcement Learning based agent. They validate the idea via implementing Impala on Playroom and R2D2 on City. The results are promising. However, many details about the models and the experiments are missing. The authors mention appendix many times but I did not find it.
strengths_and_weaknesses: Strengths

1. The idea of using language as a guidance to encourage the agent to explore more is interesting. As is discussed in the paper, language is abstract and compact, making it great cue for the environment information.

2. The language oracle can be replaced by a vision-and-language representation.

2. Successfully incorporating it into the RL system is also important. They demonstrated how to do it in two agents.

3. Good experimental results in two different environments.

Weakness:
1. The authors are using Unity, meanwhile more 3D navigation tasks are using photo-realistic environment such as Matterport-3D.

2. Lack of theory support. For example, how the language actually influences the exploration process.

3. Missing a lot of details. I guess some details might be in the missing appendix file.
questions: N/A
limitations: N/A
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors propose to use language to meaningfully abstract the observable state of an environment. The authors argue this language abstraction can be used to improve exploration when training reinforcement learning agents. Specifically, the authors propose to use the text description/caption of a state to provide an intrinsic reward to the agent for exploring new states that are semantically different from each other. To test language-based exploration, the authors use two popular exploration methods, Never Give Up (NGU) and Random Network Distillation (RND), and augment them with language. Experiments were conducted on both on-policy (Impala) and off-policy (R2D2) algorithms on several goal-oriented tasks within two environments Playroom and City. A first set of experiments was performed with a language oracle captioner as a proof-of-concept, then a second set that uses pretrained vision-language models to provide captions instead of the oracle. Empirically, it was shown that language-based exploration yields faster convergence, and greater state-space coverage across the different exploration methods and RL algorithms in the tested environments.

strengths_and_weaknesses: **What I like about this paper**
- Using language as a useful abstraction that coarsens the state space while preserving the semantics of the environment.
- Using a language oracle as a proof-of-concept, then showing promising results when access to such oracle is not possible and have to rely on vision-language models as captioners.
- The proposed technique is modular and can be applied to different RL algorithms (both on-policy and off-policy) and exploration techniques.
- Language models are being used out-of-the-box without any finetuning on the tested environments. I like the reusability aspect of it.

**Potential weaknesses**
- Language abstraction of a state should take into account the current goal. For instance, some details about an object might be relevant in one task but not in others. That said, the authors are aware of this and allude to it in the future directions section.
- Regarding the claim about random TV noise. While I agree that language could avoid getting stuck in front of a TV with random static noise, it is still susceptible to random text/images appearing on the TV (provided the language oracle describes it). While it might be an easier issue to deal with, I would still mention it has a limitation: how do you determine the 'correct' level of abstraction?
- It is not clear how stable the vision-language models (or even the language oracle) are when generating text for two adjacent frames (i.e., a slight modification of the image). For instance, does the order in which the objects in a scene are described change?

**Originality, quality, clarity, and significance**

To the best of my knowledge, the work in this paper is novel. In particular how the authors proposed to augment NGU and RND exploration methods with language. The authors do mention concurrent work and how the proposed approach differs from theirs. The paper is clearly written, well-organized, and technically sound to me. All claims are empirically backed up by experiments. With the recent advances in language modeling and computer vision, I believe the proposed technique is of interest to the research community. The modular aspect of language-based exploration should help drive wider adoption.

Overall, I recommend this paper for acceptance.

questions: - While I agree with the following claim "The human-generated captions structure the visual embedding space to reflect features most pertinent to humans and human language.", do the authors have a reference for it?
- Is the code to reproduce the results will be made available?
limitations: The authors discussed the limitations of the proposed approach, notably that vision-language models are susceptible to domain shift and less effective on multi-object scenes. Both limitations are active areas of research in the computer vision community which would benefit indirectly the proposed approach. The authors do not mention the social impacts of their research. I believe the proposed technique shares similar social impacts as any other goal-oriented RL agents, e.g. unsafe behaviors during exploration.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: To guide RL algorithms to explore novel states, the paper proposes to leverage language based representations. From Burda, language is both compact and sufficient, thus giving strong motivations for such approaches. Authors first successfully show that using scene description language as an oracle, allows the existing algorithms to achieve better performance for accomplishing tasks (Playroom) or perform better at exploration (City). Then the authors show that using caption pretrained networks can also achieve better performance than relying on visual features alone in both environments. Technically, the authors incorporate language (or language motivated representations) into NGU and RND. In both cases, the representation is used to calculate the intrinsic rewards for the corresponding RL algorithm. 
strengths_and_weaknesses: Strength

Although Khandelwa et al. 2021 have shown that CLIP pretrained representations can greatly help goal oriented RL tasks, this paper goes one step further showing that NL representation is a very effective representation and might explain from another angle the effectiveness of methods such as CLIP. The investigation is thorough by first showing the NL representation upper bound; the proposed method has been tested using two datasets with different frameworks.

Weakness

For me, the paper hasn’t shown comparisons with other pretrained model representations, thus it is very difficult to measure the paper’s contribution compared to the SOTA method. For example, line 103, the paper indicates that their method is complementary to Khandelwa et al. 2021 and thus can combine with to achieve better performance. However, given both methods capture some semantic representation about the environment, it is not clear why a better performance can even be expected? Maybe the proposed method captures this information better than Khandelwa et al. 2021? Maybe it is the inverse? Without experiments, all the above questions remain unanswered, making it difficult to situate the paper in its area.    
questions: Did the authors further explore ND type of training ? For example, we can think of experiments such as generating captioning by a pretrained model and conditioned from the language? Such experiments might shed light on how useful language really is as a representation? Or maybe the likelihood loss is generally inappropriate when we apply end to end training? Can authors share some thoughts on this aspect please?

How effective can the language be in an end-to-end training? For example, if I give a language oracle for just several scenes, what should be the design to leverage these representations to make the model better when language is no more present at test time? I agree thatt this is slightly off-topic for the paper’s main point. 

I have two minor comments for the paper that I put in this Question section:
- I feel that mentioning Impala and R2D2 explicitly is not necessary in the abstract; after all, the paper proposes a general method that is expected to work across environments and algorithms.
- The acronyms RND and NGU introduced in the introduction looks confusing, especially they don’t correspond to acronyms that can be deduced from the text.
limitations: The limitation section in this paper lists quite some important directions and items. In addition, as I listed in the strength/weakness section, to seek what this paper brings in particular compared to other approaches to integrate text inspired representations; for example, what is the advantage to integrate it in intrinsic reward instead of solely changing the representations, etc.
review_title: null

Meta-review: metareview: The proposed work aims to address a concern in novelty-based RL exploration of weak representations for guiding exploration.  They leverage image captioning to provide a semantic categorization of states for defining novelty.  The reviewers were happy with the conceptual contributions as well as the experimental results on appropriate domains. 

The "missing piece" is a better theoretical understanding of why this approach is good.  Specifically,  what about language is helpful and how might such properties be guaranteed in future trained captioning models?
confidence: Certain
Decision: Accept