Semantic Exploration from Language Abstractions and Pretrained Representations
Authors: Allison Tam
Neil Charles Rabinowitz
Andrew Kyle Lampinen
Nicholas Andrew Roy
Stephanie C.Y. Chan
DJ Strouse
Jane X Wang
Andrea Banino
Felix Hill
Venue: NeurIPS 2022 Accept (2022)

Abstract: Effective exploration is a challenge in reinforcement learning (RL). Novelty-based exploration methods can suffer in high-dimensional state spaces, such as continuous partially-observable 3D environments. We address this challenge by defining novelty using semantically meaningful state abstractions, which can be found in learned representations shaped by natural language. In particular, we evaluate vision-language representations, pretrained on natural image captioning datasets. We show that these pretrained representations drive meaningful, task-relevant exploration and improve performance on 3D simulated environments. We also characterize why and how language provides useful abstractions for exploration by considering the impacts of using representations from a pretrained model, a language oracle, and several ablations. We demonstrate the benefits of our approach with on- and off-policy RL algorithms and in two very different task domains---one that stresses the identification and manipulation of everyday objects, and one that requires navigational exploration in an expansive world. Our results suggest that using language-shaped representations could improve exploration for various algorithms and agents in challenging environments.

Semantic Exploration from Language Abstractions and Pretrained Representations
Allison C. Tam DeepMind London, UK
Neil C. Rabinowitz DeepMind London, UK ncr@deepmind.co com
Andrew K. Lampinen DeepMind London, UK lampinen@deepmind..o com
actamddeepmind.com
Nicholas A. Roy DeepMind London, UK nroy@deepmind.com com
Stephanie C. Y. Chan
DJ Strouse DeepMind London, UK s strouse@deepmind. com
DeepMind London, UK scychan@deepmind. com
Jane X. Wang"
Andrea Banino" DeepMind London, UK abanino@deepmind.com
Felix Hill" DeepMind London, UK felixhill wwtpiiii@@eeeeemnndccmmm
DeepMind London, UK
wangj wangjaane@deepmind.comm com
Abstract
Effective exploration a challenge in reinforcement learning (RL). Novelty-based exploration methods can suffer in high-dimensional state spaces, such as continuous partially-observable environments. We address this challenge by defining novelty using semantically meaningful state abstractions, which can be found in learned representations shaped by natural language. In particular, we evaluate vision-language representations, pretrained natural image captioning datasets. We show that these pretrained representations drive meaningful, task-relevant exploration and improve performance 3D simulated environments. We also characterize why and how language provides useful abstractions for exploration by considering the impacts of using representations from pretrained model, language oracle, and several ablations. We demonstrate the benefits of our approach with and off-policy RL algorithms and in two very different task domains one that stresses the identification and manipulation of everyday objects, and one that requires navigational exploration in an expansive world. Our results suggest that using language-shaped representations could improve exploration for various algorithms and agents in challenging environments.
Introduction
Exploration is of the central challenges of reinforcement learning (RL). A popular way increase an agent's 's tendency to explore is to augment trajectories with intrinsic rewards for reaching novel environment states. However, the success of this approach depends critically which states considered novel, which can in turn depend on how environment states are represented.
The literature novelty-driven exploration describes several approaches to deriving state represen- One popular method employs random features and represents the state by embedding
tations ?.
Eqqual contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

the visual observation with a fixed, randomly initialized target network [Random Network Distilla- tion: 6.. Another method uses learned visual features, taken from an inverse dynamics model [Never Give Up: S These approaches work well in classic 2D environments like Atari, but it is less clear whether they are as effective in high-dimensional, partially-observabte ments. For instance, in 3D settings, different viewpoints of the same scene may map to distinct visual states/features, despite being semantically similar. The difficulty of identifying a good mapping between visual state and feature space is exacerbated by the fact that useful state abstractions are highly task dependent. For example, a task involving tool use requires object-affordance abstractions, whereas navigation does not. Thus, acquiring state representations that support effective exploration is a chicken-aad -adeenggg-egg problem knowing whether two states should be considered similar requires the type of understanding that agent can only acquire after effectively exploring its environment.
settings such as 3D environ-
To overcome these challenges, we propose giving agents access to prior knowledge during training, in the form of abstractions derived from large vision-language models [e.g. [1] that are pretrained on image captioning data. We use these pretrained models to derive intrinsic reward that reflects meaningful novelty. We hypothesize that representations acquired by vision-language pretraining drive effective, semantic exploration in 3D environments, because the representations are shaped by the unique abstract nature of natural language.
Several aspects of natural language suggest that it could be useful to di- rect novelty-based exploration. First, language is inherently abstract: lan- guage links superficially distinct, but causally-related situations by describ- ing them similarly, and contrasts be- tween causally-distinet states by de- scribing them differently, thus outlin- ing useful concepts 29 28 Second, humans use language to communi- cate important information efficiently, without overspecifying [20 21. Thus, human language omits distracting ir- relevant information and focuses on important aspects of the world. For example, it is often observed that an agent rewarded for seeking novel ex- perience would be attracted forever to a TV with uncontrollable and unpre- dictable random static (7]. However, a human would likely caption this scene "a TV with no signal" regardless of the particular pattern; thus an agent ex- ploring with language abstractions would quickly leave the TV behind. Figure I shows another conceptual example of how language abstractions can accelerate exploration.
S
GE
"You can see a bed and two shelves"
Figure I: Navy dashed lines delincate semantically meaning- ful states. By using representations that align well with these boundaries (i.e. language), then agents more effectively ex- plore the wider state space (orange trajectory). If the rep- resentations do not reflect these boundaries and instead are amenable visual noise (i.e. different colors, viewpoints, etc.), then agents may only focus on a visually novel, yet nar- row subset of states (red trajectory).
We first perform motivating proof-of-coneept experiments using language oracle. We show that language is useful abstraction for exploration not only because it coarsens the state space, but also because it coarsens the state space in a way that reflects the semantics of the environment. We then demonstrate that our results scale to environments without language oracle using pretrained vision encoders, which are only supervised with language during pretraining. This work strives to enhance the representations used in novelty-based exploration, rather than compare various exploration methods.
We consider two popular novelty-based exploration methods from the literature, Never Give Up (NGU; Badia et al. 3 and Random Network Distillation (RND; Burda et al. [7]). and compare them to their language-augmentéd variants, Lang-NGU/LSE-NGU and Lang-RND. We evaluate performance and sample efficiency on object manipulation, search, and navigation tasks in two challenging 3D environments simulated in Unity: Playroom (a house containing toys and furniture) and City (a large-scale urban setting). Our results show that iaguage-basse exploration with pretrained vision- language representations improves sample efficiency on Playroom tasks by 18-70% It also doubles

---

piloo dining table.
see bed, shelves, book case, tedoly and
You are holding a plane. You can soe storage trey.
(a) Example instances of Oy and O2 from the Playroom environment.
You are looking ot hardwware store red
brick conner shops corner building.
You are looking of green gross ground park.
are looking at
(b) Example instances of Ov and Oz from City. Many different scenes can be associated with the same caption.
Figure 2: Visual observations from the environment and example captions generated by the language oracle. Appendix Figure $4 contains more example captions.
the visited areas in City, compared baseline methods. show that language- based exploration effective for both on-policy (IMPALA [77] and off-policy (R2D2 [25 agents.
Related Work
Exploration in RL
Classical exploration strategies include e-greedy action selection [11] state-
counting 40
30 3. curiosity driven exploration [44], and intrinsic motivation methods (36
Our work is part this last class methods, where the agent is given an intrinsic reward for visiting diverse states over time K5
Intrinsic rewards can be derived from various measures: novelty
45
55
56), prediction-error
38 ensemble disagreement I 39
48
46
18] 50 or information
gain
2
One family of methods gives intrinsic reward for following curriculum of goals [2 40
Others use novelty measures to identify interesting states from which they can perform additional These methods encourage exploration in different ways, but they all rely visual state representations that are learned jointly with the policy. Although we focus intrinsic reward and demonstrate the benefits of language in NGU and RND, our methodology is relatively agnostic to the exploration method. We suggest that many other exploration methods could be improved by using language abstractions and pretrained embeddings to represent the state space.
learning 16 54
sovelty-based
Pretraining representations for RL Pretraining has been used in RL to improve the representa- tions of the policy network. Self-supervised representation learning techniques distill knowledge from external datasets to produce downstream features that are helpful in virtual environments 15 53
Some recent work shows benefits from pretraining on more general, large-scale datasets. Pretrained CLIP features have been used in number of recent robotics papers to speed control and naviga- tion tasks. These features can condition the policy network (26] or can be fused throughout the visual encoder to integrate semantic information about the environment h7 The goal of these works is to improve perception the policy. Pretrained language models can also provide useful initializations for training policies to imitate offline trajectories 42 127 These successes demonstrate that large pretrained models contain prior knowledge that can be useful for RL. While the existing literature uses pretrained embeddings directly in the agent, we instead allow the policy network to learn from scratchm and only utilize pretrained embeddings guide exploration during training (Figure We imagine that future work may benefit from combining both approaches.

---

Language for exploration Some recent works have used language to guide agent learning, by either using language subgoals for exploratioo//pamming or providing task-specific reward shaping Schwartz. et al. 45 use a custom semantic parser for VizDoom and show that representing states with language, rather than vision, leads to faster learning by simplifying policy inputs. Chaplot et al. [TO] tackle navigation in 3D by constructing a semantic map of the environment from pretrained SLAM modules, language-defined object categories, and agent location. This approach lends itself to navigation, but it is unclear how it would extend easily to more generic settings or other types of tasks, such as manipulation. Work concurrent to ours by Mu et al. R shows how language, in the form of hand-crafted BabyAl annotations, can help improve exploration in 2D environments. These works demonstrate the value of language abstractions: the ability to ignore extrancous noise and highlight important environment features. However, these prior methods rely on environment-specific semantic parsers or annotations, which may limit the settings to which they be applied. In contrast, by exploiting powerful pretrained vision-language models, approach can be applied to any iisuallynaturralitic environment, including 3D settings, which have not been widely studied in prior exploration work. We additionally do not require any language from the environment itself. Our method could even potentially improve exploration for physical robots, but we leave that for future work.
47
36
19
3 Method
We consider a goal-conditioned Markov decision process defined by a tuple (S,A,9, Re: 7), where S is the state space, A is the action space, G is the goal space, P S x A - S specifies the environment dynamics, R. S x G + R. is the extrinsic reward, and 7 is the discount factor. State St is presented to the agent a visual observation Ov. In some cases, in order to calculate intrinsic reward, we use a language oracle O S + C that provides natural language descriptions of the state, Oz. Note that Og is distinct from the language instruction g E G, which is sampled from goal distribution at the start of an episode- the agent never observes OL. We later remove the need for a language oracle by using pretrained models.
We use goal-conditioned reinforcement learn- ing to produce a policy Tg (- Ov) that max- imizes the expected reward EE| EE (rg Br()]. where H is the horizon, rf is the extrin- sic reward, r is the intrinsic reward, and B is a tuned hyperparameter. The intrinsic reward is goal-agnostic and is computed with access to either Oy or Oz. Note that neither On nor pre- trained embeddings are used by the policy, and thus we only use them during training to com- pute the intrinsic reward (Figure 3)
Te + Br:
Action
Value
Novelty Measure
Auxiliary Component with Pretrained Language Encoder
Agent Trained Scratch
Image
Instruction
Caption
Our approach builds on two popular exploration algorithms: Never Give Up (NGU; Badia et al. [B) and Random Network Distillation (RND; Burda et al. 7 These algorithms were cho- sen to demonstrate the value of language un- der two different exploration paradigms. While both methods reward visiting novel states, they differ on several dimensions: the novelty hori- zon (episodic versus lifetime), how the history of past visited states is retained parammaraettric versus parametric), and how states are repre- sented (learned controllable states versus ran-
Figure 3: The agent is trained from scratch using RL to optimize extrinsic and intrinsic reward. It acts using the image observation Oy and goal g- During training, the novelty- -bsed intrinsic reward calculated using an auxiliary component that does not share parameters with the agent (dashed box). The auxiliary component may incorporate a pretrained language (pictured above) or image en- coder, which may respectively require OL or Oy. The latter does not rely language provided by the environment. See Figure S2 for more details.
dom features).
3.1
Never Give Up (NGU)
To more clearly isolate our impact, we focus only on the episodic novelty component of the NGU agent G. State representations along the trajectory are written to non-parametrie episodic memory

---

buffer. The intrinsic reward reflects how novel the current state is relative to the states visited so far in the episode. Novelty is a function of the L2 distances between the current state and the k-nearest neighbor representations stored in the memory buffer. Intrinsic reward is higher for larger distances.
Full details can be found in the original paper; however, we make two key simplifications. While Badia et al. B proposes learning a family of policy networks that are capable of different levels of exploration, we train one policy network that maximizes reward r Te + .r; for a fixed hyperparameter 3. We also fix the long-term novelty modulator a to be I, essentially removing it.
The published baseline method, which refer Vis-NGU, uses controllable state taken from an inverse dynamics model. The inverse dynamics model is trained jointly with the policy, but the two networks do not share any parameters.
Table I: Summary of NGU variants.
Table 2: Summary of family of RND-inspired methods. Intrinsic reward is derived from the prediction error be- tween the trainable network and frozen target function.
Name
Embedding Type
Required Input
Vis-NGU
Controllable State
Vision
Name
Trainable Network
Target Function
NGU
BERT 1
Language
Language
Vis-NND
Sv Oy + R
randomly initialized, fixed f pretrained [LM(inage, text]
Language
ND
(V..) : Oqv.,) - R
LSE-NGU
CLIP image
Vision
Lang-RND
fe: 0, + B
randomly initialized. fixed f Or from language orucle
ALMMinage
Vision
LD
0v - O2
The intrinsic reward relies on directly comparing state representations from the buffer, so our approach focuses on modifying the embedding function to influence exploration (Table Lang- NGU uses frozen pretrained language encoder embed the oracle caption OL- We compare language embeddings from BERT 11, CLIP 41, Small-ALM, and Med-ALM. The ALMs (ALign Models) are trained with a contrastive loss the ALIGN dataset 24.. Small-ALM uses 26M parameter ResNet-50 image encoder [22]: Med-ALM uses a 71M parameter NFNet [5] The language backbones are based on BERT and are all in the range of 70-90M parameters. We do not finetune on environment-speed specific data; this preserves the real world knowledge acquired during pretraining and demonstrates its benefit without requiring any cerironment-sppeiife captions.
LSE-NGU does not use the language oracle. Instead, it uses frozen pretrained image encoder to embed the visual observation Or. We use the image encoder from CLIP or ALM, which are trained on captioning datasets to produce outputs that are close to the corresponding language embeddings. The human-generated captions structure the visual embedding space to reflect features most pertinent to humans and human language [3]], so the resulting representations can be thought of as Language Supervised Embeddings (LSE). The primary benefit of LSE-NGU is that it can be applied to environments without a language oracle or annotations. CLIP and ALM are trained on world data, they would work best on realistic 3D environments. However, imagine that in future work the pretraining process or dataset could be tailored to maximize transfer to a desired target environment.
3.2 Random Network Distillation (RND)
Our RND-inspired family of methods rewards lifetime novelty. Generically, the intrinsic reward is derived from the prediction error between a trainable network and some target value generated by a frozen function (Table 2 The trainable network is learned jointly with the policy network, although they do not share any parameters. As the agent trains over the course of its lifetime, the prediction error for frequently-visiieee states decreases, and the associated intrinsic reward consequently diminishes. Intuitively, the weights of the trainable network implicitly store the state visitation counts.
For clarity, we refer to the baseline published by Burda et al. 7 as Vis-RND. The trainable network fv : Oy - R maps the visual state to random features. The random features are produced by a fixed, randomly initialized network fv. Both fy and fv share the same architecture: a ResNet followed by a MLP. The intrinsic reward is the mean squared error (Ovv - (Ov)ll.
In network distillation (ND), the target function is not random, but is instead a pretrained text or image encoder from CLIP/ALM. The trainable network f learns to reproduce the pretrained representations. To manage inference time, f is simpler network than the target (see Appendix A.2). The intrinsic loss the mean squared error between f and the large pretrained network. Like the respective Lang-NGU and LSE-NGU counterparts, rext-based ND requires a language oracle, but image-based ND does not.
5

---

In Section 5.1 we compare against two additional methods to motivate why language is useful abstraction. The first, Lang-RND, a variant in which the trainable network fr. : Or. + R maps the oracle caption to random features. The intrinsic reward is the mean squared error between the outputs of fr. and fixed ft with random initialization. Both fr. and fr. networks are of the same architecture.
The second method, language distillation (LD), is loosely inspired by RND in that the novelty signal comes from a prediction error. However, instead of learning to produce random features, the trainable network learns caption the visual state, i.e. fc: Oy - O2. The network architecture consists of a CNN encoder and LSTM decoder. The intrinsic reward is the negative log- oogliikeihood of the oracle caption under the trainable model fc. LD,, the exploration dynamics not only depend on how frequently states are visited but also the alignment between language and the visual world. We test whether this caption-denoter alignment is necessary for directing semantic exploration by comparing LD to variant with shuffled image-language alignment (S-LD) in Section 5.1
4 Experimental Setup
4.1
Environments
Previous exploration work benchmarked algorithms on video games, such as 2D grid-world MiniHack and Montezuma's 's Revenge, or 3D first-person shooter Vizdoom. In this paper, we focus on first-person Unity-based 3D environments that are meant to mimic familiar scenes from the real world (Figure 2
Playroom Our first domain, Playroom 11 52 is a randomly--ennerate house containing everyday household items (e.g, bed, bathtub, tables, chairs, toys). The agent's 's action set consists of 46 discrete actions that involve locomotion primitives and object manipulation, such as holding and rotating.
We study two settings in Playroom. In the first setting, the agent is confined to single room with 3-5 objects and is given a 1ift or put instruction. At the start of an episode, the set of objects are sampled from a larger set of everyday objects (i.e. a candle, cup. hairdryer). Object colors and sizes are also randomized, adding superficial variance to different semantic categories. The instructions take the form: "Lift <object>" or "Put a <object> on a {bed, tray3". With a lift goal, the episode ends with reward 1 or 0 whenever any object lifted. With put goal, the episode ends with reward I when the condition is fulfilled. This setting tests spatial rearrangement skills.
In the second setting, the agent is placed in a house with 3-5 different rooms, and is given a find instruction of the form "Find a {teddy bear, rubber duck)". Every episode, the house is randomly generated with the teddy and duck hidden amongst many objects, furniture, and decorations. The target objects can appear in any room- either on the floor, on top of tables, or inside bookshelves. The agent is randomly initialized and can travel throughout the house and freely rearrange objects. The episode ends with reward 1 when the agent pauses in front of the desired object. The find task requires navigation/search skills and tests the ability to ignore the numerous distractor objects.
City Our second domain, City, is an expansive, large-scale urban environment. Each episode, a new map is generated; shops, parks, and buildings are randomly arranged in city blocks. Daylight is simu- lated, such that the episode starts during the morning and ends at nighttime. The agent is randomly initialized and is instructed to "explore the city." It is trained to maximize its intrinsic reward and can take the following actions: move (forward, backward, left, right), look_ {left, right), and move arreefoowwrdd__m _orward_and___
{left, right). We divide up the map into a 32 x 32 grid and track how many unique bins are visited in an episode.
Additionally, City does not provide explicit visual or verbal signage to disambiguate locations. As such, systematic exploration is needed to maximize coverage. In contrast to Playroom, City tests long horizon exploration. A Playroom episode lasts only 600 timesteps, whereas City episode lasts 11 11,250 and requires hundreds of timesteps to fully traverse the map even once. The City covers 270-by-270 meter square area, which models a 2-by-2 grid of real world blocks.
4.2 Captioning Engine
We equip the environment with a language oracle that generates language descriptions of the scene, OL, based on the Unity state, s (Figure In Playroom, the caption describes if and how the agent

---

interacts with objects and lists what is currently visible to it. In City, OL generally describes the object that the agent is directly looking at. but the captions alone do not disambiguate the agent's locations. Since these captions are generated from a Unity state, these descriptions may not be as varied or rich as a human's, s, but they can be generated accurately and reliably, and at scale.
4.3 Training Details
At test time, the agent receives image observation Oy and language-specified goal g. The policy network never requires caption Or to act. During training, the exploration method calculates the intrinsic reward from O2 or Ov.
We show that language-based exploration is compatible with both policy gradient and Q-learning algorithms. We use Impala 17 on Playroom and R2D2 on City 25 Q-learning is more suitable for the City, because the action space is more restricted compared to the one needed for Playroom tasks.
For both environments, the agent architecture consists of image ResNet encoder and language LSTM encoder that feed into a memory LSTM module. The policy and value heads are MI.s that receive the memory state as input. If the exploration method requires additional networks, such as the trainable network in RND or inverse dynamics model in NGU, they do not share any parameters with the policy or value networks. Figure S2] is visualization of an Impala agent that uses language- augmented exploration. Hyperparameters and additional details are found in Appendix A
5 Results
5.1
Motivation: Language is a Meaningful Abstraction
Comparison of Lang-RND to Vis-RND
Task
Va-RND
0.8
0.8
0.8
Lang AND
0.6
0.6
0.6
0.2
6.0g
0.0
0.5
1.0
1.5
2.0
0.0g
Leamner Updates
Learner Updates
Learner Updates
(a) Lang-RND outperforms Vis-RND by creating a coarser, more compact state space.
Comparison of S-LD to LD and Vis-RND
1.0
Uitt Task
1.0
Task
1.0
Find Task
0.6
0.8
0.8
S-LD
0.2
0.2
0.2
0.0
0.8
0.5
1.0
1.5
2.0
Leamer Updates
1ed
Learner updates
1e5
Learner Updates
(b) LD outperforms S-LD. It is important how language abstractions carve up the state space.
Figure 4: Our comparisons demonstrate that language is useful for exploration, because it outlines a more abstract, semantically meannnggu state space. Results are shown with 95% confidence band.
We share the intuition with other work [c.g. 3d that language can improve exploration. We design set of experiments to show how and why this may be the case. Our analysis follows the desiderata outlined by Burda et al. [6 aee-preddciion-eetieee exploration ought to use feature space that filters irrelevant information (compact) and contains necessary information (sufficient). Burda et al. 6 specifically studies RND and notes that the random feature space, the outputs of the random network, may not fully satisfy either condition. As such, we use the language variants of RND to frame this discussion.
We hypothesize that language abstractions are useful, because they (1) create coarser state space and (2) divide the state space in a way that meaningfully aligns with the world (i.e. using semantics). First, if language provides coarser state space, then the random feature space becomes more compact,
7

---

leading to better exploration. We compare Lang-RNN RIN to Vis-RND to test this claim. Lang Lang-RND learns the lift task 33% faster and solves the put task as Vis-RND starts to lear (Figure 4a).
Second, we ask whether semantics - that is how language divides up the state space is critical for effective exploration. We use LD to test this hypothesis, precisely because the exploration in LD is motivated by modeling the semantic relationship between language and vision.
We compare LD to a shuffled variant S- LD, where replace the particular seman- tie state abstraction that language offers with statistically-matches
randomized abstraction (Figure 5) S-LD is similar to LD; the intrin- sic reward the prediction errоr of the caption- ing network. However, instead of targeting the language oracle output, the S-LD trainable net- work produces different target caption On that may not match the image. Or, is produced by fixed, random mapping fs Ov Oz. fs is constrained such that the marginal distributions P(OL) R P(OL) are matched under trajecto- ries produced by policy TLD. See Appendix A..4 for full details on the construction of S-LD.
RF States
see cup"
Figure 5: The dotted lines correspond to state ab- stractions given by the shuffled fs used in S-LD. The states are grouped together based on similar- ities in the visual random feature space and as- signed label. Exploring in this shuffled space is less effective than exploring with the semantically- meaningful abstractions shown in Figure
Thus, whereas the LD captions parcel up state space a way that reflects the abstractions that language offers, the randomized mapping fs parcels up state space in way that abstracts over random features of the visual space (Figure 5 We control for the compactness and coarseness of the resulting representation by maintaining the same marginal distribution of captions.
semantics crucial for exploration, then we expect to see LD outperform S-LD. This indeed holds experimentally (Figure 4b). We also view these results under the Burda al. [6] framework. The S-LD abstractions group together visually similar, but semantically distinct states. A single sampled caption likely fails capture the group in manner that is representative of all the encompassing states. In other words, fs produces a compact feature space that may not be sufficient. This may explain why S-LD learns faster than Vis-RND on the simpler 1ift task but fails on the more complex put and find tasks. The S-LD experiments imply that language abstractions because they expose not only more compact, but also more semantically meaningful state space.
helpful for exploration
5.2 Pretrained Vision-Language Representations Improve Exploration
Having shown how language be helpful for exploration, language representations into NGU and RND to improve exploration. Such representations (e.g. from the image encoder in CLIP/ALM) offer the benefits of explicit language abstractions, without the need to rely on language oracle. We also compare language-shaped representations to pretrained ImageNet embeddings to isolate the effect of language. To keep the number of experiments tractable, we only perform a full comparison on the Playroom tasks.
incorporate pretrained vision-
City We first compare how representations affect performance a pure exploration setting. With extrinsic reward, the agent is motivated solely by the NGU intrinsic reward explore the City. We report how many unique areas the agent visits in an episode in Figure g While optimizing coverage only requires knowledge of agent's global location rather than generic vision-language representations are still useful simply because meaningful exploration inherently semantic. Lang-NGU, which uses text embeddings of Or, visits area up times larger. LSE- NGU achieves 2 times the coverage even without querying language oracle (Appendix Figure S5).
understanding,

---

Playroom We next show that pretrained vision-language representations significantly speed up learning across all Playroom tasks (Fig- ure 77. The LSE--GU and Lang-NGU agents improve sample efficiency by 50-70% on the lift and put tasks and 18-38% on the find task, depending
1.-
0.8
Coverage
0.6
the pretraining model used. The ND agents are significantly faster than Vis- RND, learning 41% % faster the find task. We also measure agent-object interactions. Nearly all LSE-NGU and Lang-NGU NGU agents learn to foveate on and hold objects within 40k learn- ing updates, whereas Vis-NGU agent takes at least 60k updates to do so with the same fre- quency (Appendix Figure [S7]. Although LSE- NGU and image-based ND agents do not access a language oracle, they are similarly effective as their annotation-dependen counterparts in the Playroom tasks (Appendix Figure (S6), suggest- ing that our method could be robust to the avail- ability of a language oracle.
0.4
0.2
0.0
NGU Baseline
LSE-wGu (meggatet)
Lang NGU (CUP)
Lang Nou (ALM)
LSE LSe-NGu (CUP)
NGU (ALM)
Figure 6: Coverage of City (number of bins reached on map) by NGU variants using different state representations for exploration, normalized by coverage of a ground-truth agent. The ground- truth agent represents state in NGU as the global coordinate of the agent location. The dashed line indicates coverage of uniform random policy. Er- ror bars indicate standard error of the mean, over 5 replicas. See Appendix Table S4 for absolute cov- erage numbers.
To demonstrate the value of rich language, we compare LSE-NGU agents to control agent that instead uses pretrained ImageNet embed- dings from 70M NENet [5]. ImageNet embed- dings optimize for single-object classífication, so they confer some benefit the most object- focused tasks, lift and put. However, Ima- geNet embeddings hurt exploration the find task, where agents encounters more complex scenes (Figure /6). By contrast, the language-shaper representations are well-suited for not only describing simple objects, but also have capacity for multi-object, complex scenes. Of course, current CLIP- style models can be further improved in their ability to understand multi-ohject scenes, which may explain why the benefits are less pronounced for the find task. However, the performance of pre- trained aision-language models improve, we expect to see those benefits transfer to this method and drive even better exploration.
Discussion
We have shown that language abstractions and pretrained vision-language representations improve the sample efficiency of existing exploration methods. This benefit is seen across on-policy and off-policy algorithms (Impala and R2D2), different exploration methods (RND and NGU), different 3D domains (Playroom and City), and various task specifications (lifting/puttng, searching, and intrinsically motivated navigation). Furthermore, we carefully designed control experiments to understand how language contributes to better exploration. Our results are consistent with cognitive perspectives human
is powerful because groups together situations according to semantic similarity. In terms of the desiderata that Burda al. K present, language is both compact and sufficient. Finally, we note that using pretrained vision-languee observations enables more effective exploration even if language not available during agent training. This is vital for scaling to environments that do not have a language oracle or annotations.
representations to embed image
Limitations and future directions We highlight several avenues for extending our work. First, additional research could provide a more comprehensive understanding of how language abstractions affect representations. This could include comparing different types of captions offering varying levels of detail, or task-dependent descriptions. These captions could be dynamically generated at scale by prompting a large multimodal model 2 Second, it would be useful to investigate how to improve pretrained vision-language representations for exploration by finetuning on relevant datasets. The semantics of a dataset could even be tailored to pask-specific abstractions increase the quality of the learnt representations. Such approaches would potentially allow applying our method to virtual

---

Performance of Lang-NGU Variants
Lift Task
Task
Find
1.0
0.8
0.6
0.6
0.6
Va NGU
0.4
BERT Lang NGU
0.2
0.2
0.2
ALM - Lang NGU
0.0
8.00
0.25
0.50
0.75
1.00
1,25
1.50
0.0
Learner Updates
1o4
Learnrr Updates
105
Learner Updates
(a) Lang-NGU rewards novel pretrained text embeddings of Oz.
Performance LSE-NGU Varlants
Lift Task
Put Task
Find Task
1.0
1.0
1.0
0.8
0.0
0.0
Roward
0.6
0.6
0.4
0.4
0.4
Imagellet LSE-NGU
CUP LSE MgU
0.0
8800
0.25
0.50
0.75 1.00
1.25
1.50
0.0g
Learner Updates
Learnr Updates
le5
Learner Updates
1c4
(b) LSE-NGU rewards novel pretrained image embeddings Oy and does not use oracle language.
Performance of RND Variants
Lift Task
Task
Find Task
2.0
1.0
Vi RND (teet)
0.8
0.8
0.8
ALM-ND HD
Reward
0.6
0.6
0.6
0.2
0.2
0.2
0.0
9800
0.25
0.75
1.00
1.25
0.0g
Learner Updates
Learner Updates
Learner Updates
(c) ND intrinsic rewards derive from the prediction error of the representations from pretrained ALM network.
Figure 7: Agents that use pretrained sanuagge-happe representations to explore (ALM-ND, Lang- NGU, LSE-NGU) learn faster than baseline agents. ALM-ND (Text/Image) refer to the ND variants Table d
Results shown with a 95% confidence interval.
environments that are farther from the pretraining distribution, such as Atari. In contrast, compared to our experiments, we believe that the current pretrained representations would deliver even more benefit for entirely photorealistic, visually rich environments, such as Matterport3D (]. Finally, we note that a limitation of this approach is that current pretrained vision-language models may be less effective on multi-object scenes. Future pretraining innovations or larger models would presumably produce more robust representations and thus lead to even more effective exploration.
Acknowledgments and Disclosure of Funding
We would like to thank lain Barr for ALM models and Nathaniel Wong and Arthur Brussee for the Playroom environment. For the City environment, we would like to thank Nick Young, Tom Hudson, Alex Platonov, Bethanie Brownfield, Sarah Chakera, Dario de Cesare, Marjorie Limont, Benigno Uria, Borja Ibarz and Charles Blundell. Moreover, for the City, we would like to extend our special thanks to Jayd Matthias, Jason Sanmiya, Marcus Wainwright, Max Cant and the rest of the Worlds Team. Finally, we thank Hamza Merzic, Andre Saraiva, and Tim Scholtes for their helpful support and advice.
References
[1] Abramson, A. Ahuja, I. Barr, A. Brussee, Carnevale, M. Cassin, R. Chhaparia, S. Clark, B. Damoc, A. Dudzik, et al. Imitating interactive intelligence. arXiv preprint arXhv:2012.0567.
10

---

2020.
[2] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: visual language model for few-shot learning, arXiv preprint arXiv:2204. 14198, 2022.
[3] A. P. Badia, P. Sprechmann, A. Vitvitskyi, D. Guo, B. Piot, S. Kapturowski, O. Tieleman, M. Arjovsky, Pritzel, Bolt, et al. Never give up: Learning directed exploration strategies. arXiv preprint wrXiv:2002.06033 2020.
[4] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[5] A. Brock, S. De, S. Smith, and K. Simonyan. perhhppeffrmmace large-scale image recogmition without normalization, In International Conference on Machine Learning, pages 1059-1071. PMLR, 2021.
[6] Y. Burda, H. Edwards, D. Pathak, A. Storkey, Darrell, and A. A. Efros. Large-scale study of curiosity learning. arXiv preprint arXiv: 1808. 04355, 2018.
[7] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. arXiv preprint 1810v:1810.128800 2018
[8] A. Campero, R. Raileanu, H. Küttler B. Tenenbaum, T. Rocktäschel, and E. Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals. arXiv preprint arXiv:2006. 12122, 2020.
[9] A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, Song, A. Zeng, and Y. Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv: 1ee.11109966511 2017.
[10] D. Chaplot, D. Gandhi, A. Gupta, and R. R. Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems, 33: 4247-4258 2020.
[11] R. Y. Chen, Sidor. P. Abbeel, and Schulman. UCB exploration via Q-ensembles. arXiv preprint arXiv: 015000 2017.
[12] C. Colas, P. Fournier, M. Chetouani, O. Sigaud, and P.-Y. Oudeyer. Curious: intrinsically motivated modular multi-goal reinforcement learning. In International conference on machine learning, pages -331- PMLR, 2019.
[13] C. Colas, T. Karch, N. Lair, J.-M. Dussoux, C. Moulin-Frier, P. Dominey, and P.-Y. Oudeyer. Language cognitive tool to imagine goals in curiosity driven exploration. Advances in Neural Information Processing Systems, 33:3761-3774, 2020.
[14] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 18X00:8/00048800 2018
[15] Y. Du, C. Gan, and P. Isola. Curious representation learning for embodied intelligence. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10408- 10417, 2021.
[16] A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune. First return, then explore. Nature, 590(7847):580-556, 2021.
[17] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Doron, V. Firoiu, Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, pages 1407-1416. PMLR, 2018.
[18] S. Flennerhag, J. X. Wang, P. Sprechmann, F. Visin, A. Galashov, S. Kapturowski, D. L. Borsa, N. Heess, A. Barreto, and R. Pascanu. Temporal difference uncertainties as a signal for 2020.
exploration. arXiv preprint

---

[19] P. Goyal, S. Niekum, and R. J. Mooney. Using natural language for reward shaping in reinforce- ment learning, arXiv preprint 1903.0:900.0220 2019.
[20] H. P. Grice. Logic and conversation. In Speech acts, pages 41-58. Brill, 1975.
[21] M. Hahn, D. Jurafsky, and R. Futrell. Universals of word order reflect optimization of grammars for efficient communication. Proceedings of the National Academy of Sciences, 117(5)22347- 2353, 2020.
[22] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[23] R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel. Vime: Variational information maximizing exploration. Advances in neural information processing systems, 29, 2016.
[24] C. Jia, Yang. Y. Xia, Y.-T. Chen, Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-langueen representation learning with noisy text supervision. In International Conference on Machine Learning. pages 4904 4916. PMLR, 2021.
[25] S. Kapturowski, G. Ostrovski, J. Quan, R. Munos, and W. Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018.
[26] A. Khandelwal, L. Weihs, R. Mottaghi, and A. Kembhavi. Simple but effective: Clip embeddings for embodied ai. arXiv preprint arXiv:21 1.09888, 2021.
[27] S. Li, X. Puig. Y. Du, C. Wang, E. Akyurek, A. Torralba, J. Andreas, and I. Mordatch. Pre-trained language models for interactive decision making. arXiv preprint arXi::2202.01771, 2022.
[28] G. Lupyan. What do words do? toward theory of Inguage-auumentte thought. In Psychology of leurning and motivation, volume 57, pages 255-297. Elsevier, 2012.
[29] G. Lupyan, D. H. Rakison, and J. L. McClelland. Language is not just for talking: Redundant labels facilitate learning of novel categories. Psychological science, 18(12):0077-1083 2007.
[30] M. C. Machado, M. G. Bellemare, and M. Bowling. Count-based exploration with the successor representation. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):5125- 5133, 2020.
[31] R. Marjieh, P. van Rijn, I. Sucholutsky, T. R. Sumers, H. Lee, T. L. Griffiths, and N. Jacoby. Words are all you need? capturing human sensory similarity with textual descriptors. arXiv preprint arXiv:2206.088 0400, 2022.
[32] J. Martin, S. N. Sasikumar, T. Everitt, and M. Hutter. Count-based exploration in feature space for reinforcement learning. arXiv preprint arXiv: 1706.08090. 2017.
[33] S. Mirchandani, S. Karamcheti, and D. Sadigh. Ella: Exploration through learned language abstraction. Advances in Neural Information Processing Systems, 34, 2021.
[34] Mu, V. Zhong, Raileanu, M. Jiang, N. Goodman, T. Rocktäschel, and E. Grefenstette. Improving intrinsic exploration with language abstractions. arXiv preprint arXiv:2202.089335 2022.
[35] P.YY Oudeyer and F. Kaplan. What intrinsic motivation? a typology of computational approaches. Frontiers in neurorobotics, 1:6, 2009.
[36] P.-Y. Oudeyer, F. Kaplan, and V. V. Hafner. Intrinsic motivation systems for autonomous mental development. IEEE transactions on evolutionary computation, 11(2):265-28) 2866, 2007.
[37] S. Parisi, A. Rajeswaran, S. Purushwalkam, and A. Gupta. The unsurprising effectiveness of pre-trained vision models for control. arXiv preprint 233.502203.03338, 2022.
12

---

[38] D. Pathak, P. Agrawal, A. A. Efros. and T. Darrell. Curiosity-drive driven exploration by self- supervised prediction. In International conference on machine learning, pages 2778-2787. PMLR, 2017.
[39] D. Pathak, D. Gandhi, and A. Gupta. Self-supervised exploration via disagreement. In International conference on machine learning, pages 5062-5071, PMLR, 2019.
[40] S. Racaniere, A. Lampinen, A. Santoro, D. Reichert, V. Firoiu, and T. Lillicrap. Automated curriculum generation through setter-solver interactions. In International conference learning representations, 2019.
[41] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748 8748-8763. PMLR, 2021.
[42] M. Reid, Y. Yamada, and S. S. Gu. Can wikipedia help offline reinforcement learning? arXiv preprint arXiv:2201. 12122, 2022.
[43] N. Savinov, A. Raichuk, R. Marinier, D. Vincent, M. Pollefeys. T. Lillicrap, and S. Gelly. Episodic curiosity through reachability. arXiv preprint arXiv:1810.02277 2018.
[44] Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals animatx, pages 222-227, 1991.
[45] E. Schwartz, G. Tennenholtz, C. Tessler, and S. Mannor. Language is power: Representing states using natural language in reinforcement learning. arXiv preprint arXiv: 2019.
[46] R. Sekar, O. Rybkin, K. Daniilidis, P. Abbeel, D. Hafner, and D. Pathak. Planning to explore via self-supervised world models. In International Conference on Machine Learning (ICML), 2020.
[47] M. Shridhar, X. Yuan, M.-A. Côté, Y. Bisk, A. Trischler, and M. Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020.
[48] P. Shyam, W. Jaskowski, and F. Gomez. Model-based active exploration. In International Conference Machine Learning (ICMI), 2019.
[49] A. L. Strehl and M. L. Littman. An analysis of model- interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8):1309-1333) 2008.
[50] D. Strouse, K. Baumli, D. Warde-Farley, V. Mnih, and S. Hansen. Learning more skills through optimistic exploration. International Conference on Learning Representations (ICLR), 2022.
[51] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018
[52] D. I. A. Team, J. Abramson, A. Ahuja, A. Brussee, F. Carnevale, M. Cassin, F. Fischer, Georgiev, A. Goldin, T. Harley, et al. Creating multimodal interactive agents with imitation and self-supervised learning. arXiv preprint 2rXiv:21 2021.
[53] T. Xiao, I. Radosavovic, T. Darrell, and J. Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06173, 2022.
[54] D. Zha, W. Ma, L. Yuan, X. Hu, and Liu. Rank the episodes: A simple approach for exploration in procedurallyy-generaeed environments. arXiv preprint arXiv:2101.08152. 2021.
[55] T. Zhang, P. Rashidinejad, J. Jiao, Y. Tian, J. E. Gonzalez, and S. Russell. Made: Exploration via maximizing deviation from explored regions. Advances in Neural Information Processing Systems, 34, 2021.
[56] T. Zhang, H. Xu, X. Wang, Y. Wu, K. Keutzer, J. E. Gonzalez, and Y. Tian. Noveld: A simple yet effective exploration criterion. Advances in Neural Information Processing Systems, 34, 2021.
13


Reviews:
[{"review_id": "zqI_qV4abr", "reviewer": "Reviewer_kb5q", "summary": "In this paper, the authors propose to use language to meaningfully abstract the observable state of an environment. The authors argue this language abstraction can be used to improve exploration when training reinforcement learning agents. Specifically, the authors propose to use the text description/caption of a state to provide an intrinsic reward to the agent for exploring new states that are semantically different from each other. To test language-based exploration, the authors use two popular exploration methods, Never Give Up (NGU) and Random Network Distillation (RND), and augment them with language. Experiments were conducted on both on-policy (Impala) and off-policy (R2D2) algorithms on several goal-oriented tasks within two environments Playroom and City. A first set of experiments was performed with a language oracle captioner as a proof-of-concept, then a second set that uses pretrained vision-language models to provide captions instead of the oracle. Empirically, it was shown that language-based exploration yields faster convergence, and greater state-space coverage across the different exploration methods and RL algorithms in the tested environments.\n", "questions": "- While I agree with the following claim \"The human-generated captions structure the visual embedding space to reflect features most pertinent to humans and human language.\", do the authors have a reference for it?\n- Is the code to reproduce the results will be made available?", "limitations": "The authors discussed the limitations of the proposed approach, notably that vision-language models are susceptible to domain shift and less effective on multi-object scenes. Both limitations are active areas of research in the computer vision community which would benefit indirectly the proposed approach. The authors do not mention the social impacts of their research. I believe the proposed technique shares similar social impacts as any other goal-oriented RL agents, e.g. unsafe behaviors during exploration.\n", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**What I like about this paper**\n- Using language as a useful abstraction that coarsens the state space while preserving the semantics of the environment.\n- Using a language oracle as a proof-of-concept, then showing promising results when access to such oracle is not possible and have to rely on vision-language models as captioners.\n- The proposed technique is modular and can be applied to different RL algorithms (both on-policy and off-policy) and exploration techniques.\n- Language models are being used out-of-the-box without any finetuning on the tested environments. I like the reusability aspect of it.\n\n**Potential weaknesses**\n- Language abstraction of a state should take into account the current goal. For instance, some details about an object might be relevant in one task but not in others. That said, the authors are aware of this and allude to it in the future directions section.\n- Regarding the claim about random TV noise. While I agree that language could avoid getting stuck in front of a TV with random static noise, it is still susceptible to random text/images appearing on the TV (provided the language oracle describes it). While it might be an easier issue to deal with, I would still mention it has a limitation: how do you determine the 'correct' level of abstraction?\n- It is not clear how stable the vision-language models (or even the language oracle) are when generating text for two adjacent frames (i.e., a slight modification of the image). For instance, does the order in which the objects in a scene are described change?\n\n**Originality, quality, clarity, and significance**\n\nTo the best of my knowledge, the work in this paper is novel. In particular how the authors proposed to augment NGU and RND exploration methods with language. The authors do mention concurrent work and how the proposed approach differs from theirs. The paper is clearly written, well-organized, and technically sound to me. All claims are empirically backed up by experiments. With the recent advances in language modeling and computer vision, I believe the proposed technique is of interest to the research community. The modular aspect of language-based exploration should help drive wider adoption.\n\nOverall, I recommend this paper for acceptance.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "o_mN1_y6aO", "reviewer": "Reviewer_p9aE", "summary": "The paper motivates the use of language abstractions instead of raw visual state representation for designing intrinsic exploration reward. As language can provide coarser and compact description of the environment, it can prevent unnecessary actions and meaningfully incentivize the agent to achieve the task. To this end, the paper proposes to use pre-trained models on image captioning datasets to enable better task-relevant exploration in 3D simulated environments. The experiments demonstrate that (1) random network distillation on oracle captions performs better than visual state representation and (2) semantics in the oracle caption matter, especially in longer horizon tasks. Furthermore, the use of ALM, CLIP or Imagenet embeddings based NGU or RND for exploration achieves faster task success as compared to using raw visual state representations. ", "questions": "- The motivation why language enables wider exploration is unclear in Figure 1.\n- Appendix is missing?\n- In line 113, the policy receives … template-based “language” goal?\n- What held-out goal settings, ie. in terms of unseen objects or environments considered in evaluation for different tasks?\n- Can you clarify what ALM-ND (Text) and ALM-ND (Image) mean in Fig 6(c)? It seems that ALM-ND (X) is extracted from the pre-trained X encoder to use for network distillation and generate intrinsic reward.\n- ‘Find’ task in Playroom seems very similar to the maximizing coverage in object navigation [1] Chaplot et al. How does the previous work compare as a baseline in terms of achieved coverage in held-out city environments?", "limitations": "The authors provide detailed limitations and future scope, namely: (1) comparing different ways of captioning, (2) fine-tuning pre-trained VL models on relevant datasets for interaction tasks, and (3) having pretrained models on multi-object scenes to improve performance without oracle captions. Extending upon (1), the proposed approach seems to have an objective way of captioning any state that is relevant to solve the task effectively, whereas natural language captions are context-driven, underspecified in terms of all the objects in the seen and mostly describing main semantic takeaway. This assumption should be noted before using the proposed method to design intrinsic rewards for physical robots task. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is overall well-written and sound. The contribution in terms of experiments showing the value of using language based abstraction in modeling intrinsic reward in 3D scenes is novel and promising step towards exploration techniques for robot learning.\nHowever, the success of using language based representations for exploration is not itself novel and somewhat depends on the kind of. captions and the task. This is reflected in Fig 6 and the authors also suggest experimenting with different captioning strategies as future research direction. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "TZPGZilLxE2", "reviewer": "Reviewer_jixR", "summary": "To guide RL algorithms to explore novel states, the paper proposes to leverage language based representations. From Burda, language is both compact and sufficient, thus giving strong motivations for such approaches. Authors first successfully show that using scene description language as an oracle, allows the existing algorithms to achieve better performance for accomplishing tasks (Playroom) or perform better at exploration (City). Then the authors show that using caption pretrained networks can also achieve better performance than relying on visual features alone in both environments. Technically, the authors incorporate language (or language motivated representations) into NGU and RND. In both cases, the representation is used to calculate the intrinsic rewards for the corresponding RL algorithm. ", "questions": "Did the authors further explore ND type of training ? For example, we can think of experiments such as generating captioning by a pretrained model and conditioned from the language? Such experiments might shed light on how useful language really is as a representation? Or maybe the likelihood loss is generally inappropriate when we apply end to end training? Can authors share some thoughts on this aspect please?\n\nHow effective can the language be in an end-to-end training? For example, if I give a language oracle for just several scenes, what should be the design to leverage these representations to make the model better when language is no more present at test time? I agree thatt this is slightly off-topic for the paper’s main point. \n\nI have two minor comments for the paper that I put in this Question section:\n- I feel that mentioning Impala and R2D2 explicitly is not necessary in the abstract; after all, the paper proposes a general method that is expected to work across environments and algorithms.\n- The acronyms RND and NGU introduced in the introduction looks confusing, especially they don’t correspond to acronyms that can be deduced from the text.", "limitations": "The limitation section in this paper lists quite some important directions and items. In addition, as I listed in the strength/weakness section, to seek what this paper brings in particular compared to other approaches to integrate text inspired representations; for example, what is the advantage to integrate it in intrinsic reward instead of solely changing the representations, etc.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\nAlthough Khandelwa et al. 2021 have shown that CLIP pretrained representations can greatly help goal oriented RL tasks, this paper goes one step further showing that NL representation is a very effective representation and might explain from another angle the effectiveness of methods such as CLIP. The investigation is thorough by first showing the NL representation upper bound; the proposed method has been tested using two datasets with different frameworks.\n\nWeakness\n\nFor me, the paper hasn’t shown comparisons with other pretrained model representations, thus it is very difficult to measure the paper’s contribution compared to the SOTA method. For example, line 103, the paper indicates that their method is complementary to Khandelwa et al. 2021 and thus can combine with to achieve better performance. However, given both methods capture some semantic representation about the environment, it is not clear why a better performance can even be expected? Maybe the proposed method captures this information better than Khandelwa et al. 2021? Maybe it is the inverse? Without experiments, all the above questions remain unanswered, making it difficult to situate the paper in its area.    ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2lx1LTFRAa3", "reviewer": "Reviewer_mRW6", "summary": "This paper proposes to use semantic representation of the visual environment to guide the agent to explore novel scenes in 3D tasks.\n Either oracle description of the environment or vision-and-text representation from a large language model can be used to calculate reward to guide a Reinforcement Learning based agent. They validate the idea via implementing Impala on Playroom and R2D2 on City. The results are promising. However, many details about the models and the experiments are missing. The authors mention appendix many times but I did not find it.", "questions": "N/A", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The idea of using language as a guidance to encourage the agent to explore more is interesting. As is discussed in the paper, language is abstract and compact, making it great cue for the environment information.\n\n2. The language oracle can be replaced by a vision-and-language representation.\n\n2. Successfully incorporating it into the RL system is also important. They demonstrated how to do it in two agents.\n\n3. Good experimental results in two different environments.\n\nWeakness:\n1. The authors are using Unity, meanwhile more 3D navigation tasks are using photo-realistic environment such as Matterport-3D.\n\n2. Lack of theory support. For example, how the language actually influences the exploration process.\n\n3. Missing a lot of details. I guess some details might be in the missing appendix file.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept