Improving Intrinsic Exploration with Language Abstractions
Authors: Jesse Mu
Victor Zhong
Roberta Raileanu
Minqi Jiang
Noah Goodman
Tim Rocktäschel
Edward Grefenstette
Venue: NeurIPS 2022 Accept (2022)
Keywords: reinforcement learning
intrinsic motivation
exploration
language
deep rl
language-guided rl

Abstract: Reinforcement learning (RL) agents are particularly hard to train when rewards are sparse. One common solution is to use intrinsic rewards to encourage agents to explore their environment. However, recent intrinsic exploration methods often use state-based novelty measures which reward low-level exploration and may not scale to domains requiring more abstract skills. Instead, we explore natural language as a general medium for highlighting relevant abstractions in an environment. Unlike previous work, we evaluate whether language can improve over existing exploration methods by directly extending (and comparing to) competitive intrinsic exploration baselines: AMIGo (Campero et al., 2021) and NovelD (Zhang et al., 2021). These language-based variants outperform their non-linguistic forms by 47-85% across 13 challenging tasks from the MiniGrid and MiniHack environment suites.

Improving Intrinsic Exploration with Language Abstractions
Jesse Mul; Victor Zhong*. Roberta Raileanu', Minqi Jiang
Noah Goodman', Tim Rocktäschel*, Edward Grefenstette Stanford University, UUiiersity of Washington, Meta AI, UUniversity College London, 'Cohere
Abstract
Reinforcement learning (RL) agents particularly hard train when rewards are sparse. One common solution is to use intrinsic rewards to encourage agents to explore their environment. However, recent intrinsic exploration methods often use state-based novelty measures which reward low-level exploration and may not scale domains requiring more abstract skills. Instead, we explore language as a general medium for highlighting relevant abstractions in environment. Unlike previous work, we evaluate whether language can improve over existing exploration methods by directly extending (and comparing to) competitive intrinsic exploration baselines: AMIGG (Campero et al., 2021) and NovelD (Zhang et al., 2021). These language-based variants outperform their non-linguistic forms by 47-85% across challenging tasks from the MiniGrid and MiniHack environment suites.
1 Introduction
A central challenge in reinforcement learning (RL) is designing agents that solve complex, long-how long-horizon tasks with sparse rewards. In the absence of extrinsic rewards, one popular solution is provide intrinsic rewards for exploration [4 35
This invariably leads to the challenging question: how should one measure exploration? One common answer that an agent should be rewarded for attaining "novel" states in the environment, but naive measures of novelty have limitations. For example, consider an agent that starts in the kitchen of a large house and must make an omelet. Simple state-based exploration will reward an agent for visiting every room in the house, but a more effective strategy would be to stay put and use the stove, Moreover, like kitchens with different-colored appliances, states can look cosmetically different but have the same underlying semantics, and thus are not truly novel. Together, these constitute two fundamental challenges for intrinsic exploration: first, how can we reward true progress in the environment over meaningless exploration? Second, how can we tell when a state is not just superficially, but semantically novel?
42 [R
Fortunately. humans are equipped with a powerful tool for solving both problems: language. As comnrrtone of human intelligence, language has strong priors over the features and hehaviors needed for exploration and skill acquisition. It also describes rich and compositional set of meaningful behaviors as simple as directions (c.g. move left) and as abstract as conjunctions of high level tasks (e.g. retrieve the ring and defeat the wizard) that can categorize and unify many possible world states.
Our aim to see whether language abstractions can improve existing state- state-based exploration methods in RL. While language-gut guided exploration methods exist in the literature [3] 5
[2]
[3]
21
24 B1
51 53 we make two key contributions over prior work. First, existing methods assume access
to high-level linguistic instruction for reward shaping, or otherwise assume that any intermediate language annotations encountered are always helpful for learning. Instead, we study settings without
Work done while at Meta AI. Correspondence to muj@stanford.cm
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

instructions, with more diverse intermediate messages (Figure 1 that may or may not be useful, but may nonetheless be a more effective measure of novelty than raw states.
Second, past work often compares only to vanilla RI., while ignoring competitive intrinsic exploration baselines. This leaves the true utility of language over simpler state-based exploration unclear. To rem- edy this issue, we conduct controlled evaluation on the effect of language on competitive approaches exploration by extending two recent, state-of-the-art methods: AMIGo 7l where a teacher pro- poses intermediate location-based goals for a student, and NovelD [54] which rewards an agent for visiting novel regions of the state space. Building upon these methods, we propose L-AMIGo, where the teacher proposes goals expressed via language instead of coordinates, and L-NovelD, variant of NovelD with an additional exploration bonus for visiting linguistically- states. Across 13 challenging,
genneratly-
MiniGrid
MiniHack
sparse-reward tasks the MiniGrid [8] and MiniHack 41 environment suites, we show that aanguage-parameterizae exploration meth- ods outperform their non-linguistic counterparts by 47-85%, especially in more abstract tasks with larger state and action spaces. We also show that language improves the interpretabil- ity of the training process, either by developing a natural curriculum of semantic goals (in L- visualize the most
There You here crystal wand.
the green door.
the yellow
you went zap? [f or
key.
Put the key the
direction? [h]k1]
AMIGo) by allowing novel language during training (in L-NovelDD. Finally, we show when and where the fine- grained compositional semantics the language improves agent exploration, when compared to non-compositional baselines.
yellow door
You kill minotaur!
Figure 1: Language conveys meaningful environ- ment abstractions. Language state annotations in the MiniGrid KeyCorridorS4R3 [8] and MiniHack
Wand of Death (Hard) 41 tasks.
2 Related Work
Exploration in RL.
Exploration has long history in RL, from e-greedy
48 or count-based
exploration 4
to intrinsic motivation 33]33 and curiosity-based learning [52] More recently, deep neural networks have been used to measure novelty with changes in state
29
30
47
50
representations 6 50
5d
or prediction errors in world models
36
46
Another long tradition
generates curricula of intrinsic goals to encourage learning 7
18 20, 37-33
In this paper,
we explore the potential benefit of language on these approaches exploration.
Language for Exploration. The observation that Linguage-guided exploration can improve RL is not new: language has been used to shape policies [4 [1] and rewards K
5
21 23
52 53 and
set intrinsic goals 22 [ Crucially, our work differs from prior work in two ways: first, instead of reward shaping with high-level instructions, we use noisier, intermediate language annotations for exploration; second, we directly extend and compare to competitive intrinsic exploration baselines.
L-AMIGo, our variant of AMIGo with language goals, is similar to the IMAGINE agent of Colas et al. 13 which also sets intrinsic language goals. However, IMAGINE is built for instruction following, and requires a perfectly compositional space of language goals, which the agent tries to explore so that it can complete novel goals at test time. Instead, we make no assumptions on the language and explore to maximize extrinsic reward, using an alternative goal difficulty metric to measure progress.
511 5X reward an agent for actions associated with linguistic descriptions. but again are primarily designed for instruction following, where an extrinsic goal is available to help shape intermediate rewards. In our setting, however, we have not high-level extrinsic goals but low-level intermediate language annotations. Extrinsic reward shaping methods such as LEARN 22 could be naively applied by simply doing reward shaping with every intermediate language annotation, and a few of these methods are designed for low-level language subgoals (24. B1] However, a shared assumption of these approaches is that language always helpful: either because we have expert-curated messages (as in Harrison et al. 24).. or because we have goal descriptions that let us identify subgoals relevant to the extrinsic goal (as in ELLA;; Mirchandani et (.). In our tasks, however, most language unhelpful for progress in the environment, and we have extrinsic goals. Consequently, past methods reduce to
Meanwhile, reward shaping and inverse RI. methods

---

T
(5,4)
T (uround)
to the red doar up bell open the gray
T (policy)
go to the red door
plok
open the gray door
(a) AMIGi (Campero et al., 2021)
(b) L-AMIGo
Figure 2: L-AMIGo. (a)
go the yellow door
AMIGo. (b) L-AMIGo
T (ground)
go
to the green door
teacher first predicts
go
the
achievable goals, then
to the (old)
What longvoge was ochieved?
(it) Train grounding network
samples goal. (i-iii)
L-AMIGo teacher training steps: updating the goal set and training grounding and policy networks.
pick up the key (new)
reward goal achieved in i' steps
Ipolisyl
reward otherwise
go red door
(i) Add new goals
(iil) Train policy network
simply giving fixed reward for every intermediate message encountered, which (we will show) fails to learn. Finally, work concurrent to ours by Tam et al. [49] tackles similar ideas in photorealistic environments that permit transfer from foundation models. Instead, explore domain-specifer symbolic games with built-in language where such models are not readily available.
A final distinguishing contribution of our work is that prior work often neglects non-linguistic exploration baselines. For example, Harrison et al. (24) and LEARN (22) compare to vanilla RI. only; ELLA 31 compares to LEARN and RIDE 40 with limited improvements over RIDE. Prior work can thus be summarized as showing that linguistic rewards improve over extrinsic rewards alone. Instead, we provide novel evidence that linguistic rewards improve upon state-based intrinsic rewards, using the same exploration methods and challenging tasks typical of recent work in RL.
Problem Statement
We explore RL in the setting of an augmented Markov Decision Process (MDP) defined as the tuple (S,A, A,A,T, T, R2 7, L), where S and A are the state and action spaces, T' S A S is the environment transition dynamics, R S A R is the extrinsic reward function, where rtt R(st, az) is the reward obtained at time by taking action ae in state 8t, and y is the discount factor. To add language, assume access to an annotator L that produces language descriptions for states: & L(se). such as those in Figure D Note that not every state needs a description (which we model with a null description 2) and the set of descriptions need not be known ahead of time We ultimately seek policy that maximizes the expected discounted (extrinsic) reward R. E = Trre+ where H is the finite time horizon. During training, however, we maximize an augmented reward r = r + Art. where of is an intrinsic reward and À is scaling hyperparameter.
Like past work 26 B1 [53 we make the simplifying assumption of access an oracle language annotator L provided by the environment. Note that the language annotator is "oracle" in that it always outputs messages that are true of the current state, but not "oracle" in that it indiscriminately outputs messages that are not necessarily relevant for the extrinsic goal. Many modern RL environments are pre-equipped with language, including NetHaac/MiinHack and in fact most video games in general. In the absence of an oracle annotator, one common approach though such datasets are often generated from oracles that are simply run offline instead 31 Concurrent work
41 text-based games 15 451 52)
learn an annotator model from a dataset of language-annnn annotated states [3] 22
49 uses pretrained foundation models to automatically provide annotations in 3D environments, though such models are not readily available in the symbolic 2D games we explore. Since this idea has been well-proven, we assume oracle access to L. but as an example, one could straightforwardly adapt the annotator model trained on Baby AI by Mirchandani et al. 31 to setting.
For presentational simplicity, the annotator here outputs a single description per state, but in practice, we allow an annotator to produce multiple descriptions: e.g. in MiniGrid, open the door and open the red door describe the same state. This requires two minor changes in the equations, described in Footnotes 4 and
3

---

4 L-AMIGo
We now describe our approach to jointly training a student and a gooo-proposing teacher, extending AMIGo 7 to arbitrary language goals.
4.1 Adversarially Motivated Intrinsic Goals (AMIGo)
AMIGo ? augments an RL student policy with goals generated by a teacher, which provide intrinsic reward when completed (Figure 2)). The idea is that the teacher should propose intermediate goals that start simple, but grow harder to encourage an agent to explore its environment.
Student. Formally, the student S is goal-conditioned policy parameterized as (a; | 8t. St: 0s), where Si is the goal provided by the teacher, and the student receives an intrinsic reward r of 1 only the teacher's goal that timestep completed. The student receives goal from the teacher either at the beginning of an episode, or mid-episode, if the previous goal has been completed.
Teacher. Separately, AMIGo trains an adversarial teacher policy Tr(ae s8; (r) to propose goals to the student given its initial state. The teacher is trained with a reward r that depends on difficulty threshold t": the teacher is given a positive reward of +o for proposing goals that take the student more than t timesteps to complete, and -3 for goals that are completed sooner, or never completed within the finite time horizon. To encourage proposing harder and harder goals that promote exploration, t* is increased linearly throughout training: whenever the student completes 10 goals in row under the current difficulty threshold, it is increased by 1, up to some tunable maximum difficulty. Finally, to encourage intermediate goals that are aligned with the extrinsic goal, the teacher is also rewarded with the extrinsic reward when the student attains it.
This teacher is updated separately from the student at different time intervals. Formally, its training data is batches of (so, 9t, It,r?") tuples collected from student trajectories for nonzero rf where 80 is the initial state of the student's trajectory and 9: is the goal that led to reward r!!
The original paper implements AMIGo for MiniGrid only, where the goals gr are (x, y) coordinates to be reached. The student gets the goal embedded directly in the M N environment, and the teacher is a dimensiionality-prsseee environment into an M N distribution over coordinates, from which single goal is selected.
convolutional network which encodes the student's M x N
4.2 Extension L-AMIGo
Student. The L-AMIGo student is a policy conditioned not on (z. y) goals, but on language goals 4; (a(a 8t, 0g). Given the "goal" ly the student is now rewarded if it reaches state with the language description l, i.e. f L(s,) Typically this student will encode the goal with learned language model and concatenate the goal representation with its state representation.
Teacher. Now the L-AMIGo teacher selects goals from the set of possible language descriptions in the environment. Because the possible goals are initially unknown, the teacher maintains a running set of goals G that is updated as the student encounters new state descriptions (Figure 2)
This move to language creates challenge: not only must a teacher choose a goal to propose, must also determine which goals are achievable at all. For example, the goal go to the red door only makes sense environments with red doors. L-AMIGo, these tasks are factorized into policy network, which produces the distribution over goals given a student's 's state, and a grounding network, which predicts the probability that goal is likely to be achieved in the first place (Figure 2b):
TV (fe 81; 0T) x Per 81; OT) Ppolicy (fe) 81; 1115 0T) Pground (c, s1 0T) = a (f(le; 0T) (si 0T))
Prolicy (fe (1;0r) x fl(;;Tr) hpoticy (s1; 0r)
We can treat language goals and state descriptions equivalently, even the wordings are slightly different across environments. In MiniGrid, messages (e.g. go to the red door) look like goals but can also be interpreted as state descriptions: (in this state, you have] go[ne] to the red door In MiniHack, messages are description like (e.g. you kill the minotaur!), but imagine the teacher's goal (reach state where] you kill the minotaur!

---

Equation describes the policy network as producing a probability for a goal by computing the dot
product between goal and state representations f(e;; 0T) and mpolicy (sy; 0T), normalizing over possible goals; this policy is learned identically to the standard AMIGo teacher (Figure ii.. Equation 2 specifies the grounding network as predicting whether goal is achievable in an environment, by applying the sigmoid function to the dot product between the goal representation f(e;; 0();;T) and (possibly separate) state representation Rggroun 0r). Given an oracle grounding classifier, which outputs only 0 or I, this is equivalent to restricting the teacher to proposing only goals that are achievable in a given environment. In practice, however, we learn the classifier online (Figure ii). Given the initial state So of an episode, we ask the grounding network to predict the first language description encountered along this trajectory: fist L(se), where t' is the minimum where Ø. This is formalized as multilabel binary cross entropy loss,
L(s:) +
Cgroand (Sp, (1s) = log log'Pgrounal (fas S0s 0r))
log(1 - Perouna (e' |00 0r)), (4)
where the second term noisily generates negative samples of (start state, unachieved description) pairs based on the set of descriptions G known to the teacher at the time, similar to contrastive learning Note that since G updated during training, Equation grows to include more terms over time.
To summarize, training the teacher involves three steps: (1) updating the running set of descriptions seen the environment, (2) learning the policy network based on whether the student achieved goals proposed by the teacher, and (3) learning the grounding network by predicting descriptions encountered from initial states. Algorithm ST Appendix A describes how L-AMIGo trains in an asynchronous actor-critic framework, where the student and teacher are jointly trained from batches of experience collected from separate actor threads, as used in our experiments (see Section
L-NovelD
Next, we describe NovelD [54]. which extends simpler tabular- 47 or pseudo- 4 6 count-based intrinsic exploration methods, and our language variant, L-NovelD. Instead of simply rewarding an
agent for rare states, NovelD rewards agents for Iransitioning from states with low novelty to states with higher novelty. Zhang et al. [54] show that NovelD surpasses Random Network Distillation (6)
another popular exploration method, on a variety of tasks including MiniGrid and Atari.
5.1 NovelD
NovelD defines the reward rf to be the difference in novelty between state st and previous state st-1;
r = NovelD 11 A
- - (()))))))))))((((sss)))))))))))) (s))=
(5)
Term (NovelD)
(ERIR)
In the first NovelD term, N(s)) ) is the novelty of state 8,; this quantity describes the difference in novelty between successive states, which is clipped >> 0 so the agent is penalized from moving back to less novel states. a is a hyperparameter that scales the average magnitude of the reward. The second term is the Episodic Reduction on Intrinsic Reward (ERIR); constraint that the agent only receives reward when encountering a state for the first time in an episode. (o():) is an episodic state counter that tracks exact state visitation counts, as defined by (z. y) coordinates.
Measuring novelty with RND. In smaller MDPs, it is possible to track exact state visitation counts, which case the novelty is typically the inverse square root of visitation counts 47). However, in larger environments where states are rarely revisited, we (like NovelD) use the popular Random Network Distillation (RND) 6 technique as an approximate novelty measure. Specifically, the novelty of state is measured by the prediction enror of a state embedding network that is trained jointly with the agent to match the output of fixed, random target network. The intuition that states which the RND network has been trained will have lower prediction error than novel states.
'If multiple "first" descriptions are found, the teacher predicts for each description, and 0 for all others.
5

---

5.2 Extension to L-NovelD
Our incorporation of language is simple: we add an additional exploration bonus based on novelty defined according to the language descriptions of states:
(eve 4-1) - max(((ee) - -.N(44-1),) ((N((()) =).
(6)
This bonus is identical to standard NovelD: (()) is the novelty of the description as measured by separately parameterized RND network encoding the description and Ne(4)) = 1 when the language description has been encountered for the first time this episode. We keep the original NovelD exploration bonus, language rewards may be sparse and a basic navigation bonus can encourage the agent to reach language-annotated states. The final intrinsic reward for L-NovelD is
L-NovelDD(s; St-1, la 4 - NovelD, (8t, 8t-1) + AeNovelD
where de controls the trade-off between Equations and
One might ask why we do not simply include the language description & as input into the RND network, along with the state. While this can work in some cases, decoupling the state and language novelties allow us to precisely control the trade-off between the two, with a hyperparameter that can be tuned different tasks. In contrast, a combined input obfuscates the relative contributions of state and language to the overall novelty. Appendix F2 has ablations that show that (I) combining the state and language inputs or (2) using the language novelty term alone leads to worse performance.
Experiments
We evaluate L-AMIGo, AMIGo, L-NovelD, and NovelD, implemented in the TorchBeast 27 imple- mentation of IMPALA 17, common asynchronous actor-critic method. Besides vanilla IMPALA,
we also compare to a naive (fixed) message reward given for any message in the environment, which is similar doing extrinsic reward shaping for all messages (e.g. LEARN [2] also B 5 21
2
44 (33
or prior approaches that assume that messages are always helpful (Harrison et al. 24 ELLA B1); see Appendix B for more discussion
this bascline and its equivalencies to prior work We
each model times across 13 tasks within two challenging procedurally-generateed RI. environments,
MiniGrid [ and MiniHack [41). and adapt baseline models provided for both environments 7
for full model, training, and hyperparameter details, see Appendix
6.1 Environments
MiniGrid. Following Campero et al. 7 we evaluate on the most challenging tasks in MiniGrid S which involve navigation and manipulation tasks in gridworlds: KeyCorridorS(3,455)) and ObstructedMazze. (aaeeedddMaeeeDD..DDDM (1DI,2DIhb,1Q)
(Figure ! These tasks involve picking up a ball in a locked room, with the key to the door hidden in boxes or other rooms and the door possibly obstructed. The suffix indicates the size of the environment, in increasing order. See Appendix G. for more details.
To add language, we use the complementary Baby Al platform 1 which provides grammar of 652 possible messages, involving goto, open, pickup, and putnext commands applied to variety of objects qualified by type (e.g. box, door) and/or color (e.g. red, blue). The oracle language annotator emits a message when the corresponding action is completed, On average, only 6 to 12 messages (1-2% of all 652) are needed to complete each task (see Appendix G.I for all messages).
Note that since Al messages are not included in the original environment from which we adapt baseline agents 77 none of our MiniGrid agents encode language observations directly into the state. While it can be tempting and beneficial to use language in this way, one priori benefit of using language solely for exploration is that language is only needed during training. and not evaluation. Regardless, see Appendix D for additional experiments with MiniGrid agents that encode language into the state representation; while this boosts performance of baseline models, the experiments show that innuagg--augmenter exploration methods still outperform non-linguistic ones.
$ Foor multiple messages, we average NovelD of each messsage. 'For an implementation of a message reward with simple novelty-based decay, see the message-only L- NovelD ablation results in Appendix F2 which underperforms full L-NovelD and L-AMIGo.

---

KeyCoridorS3RB
KeyComidorS4R3
KeyComdorSSR3
WoD-Medium
WoD-Hard
1.00
0.75
0.50
0.25
0.00
2e7
4e7
6e7
5e7
1e6
1.5e6
ObstructedMaze. D
OostructedMazze 20thb
ObstructedMaze_11
River
Quest-Easy
Extrinsic Reward
1.00
0.75
0.50
0.25
0.00
107
3e70
5e7
1e8
1.5e8
1e8 1.5e8
Quest-Medium
MultiRcom-N--- N2-Exxreme
MullRcoo-NV4Exxreemm
1.00
0.75
IMPALA AMIGo
L-AMIGo NovelD
L-NoveiD
0.50
Naive Reward
0.00
4e7 Frames
Figure 3: Training curves. Mean extrinsic reward (# std eIT) across independent runs for each model and environment. In general, linguistic variants outperform their non non-linguistic forms.
MiniHack. MiniHack ! is a suite of seneeduurlll-genertte tasks of varying difficulty set the roguelike game NetHack 28 MiniHack contains diverse action space beyond simple MiniGrid- esque navigation, including planning, inventory management, tool use, and combat. These actions cannot be expressed by (2, y) positions, but instead are captured by in-game messages (Figure We evaluate our methods on a representative suite of tasks of varying difficulty: River, Wand of Death (WoD)-(Medium,,Ha) Hard), Quest-rEasy,Mediing
and (N2,
For space reasons, we describe the WoD-Hard environment here, but defer full descriptions of tasks (and messages) to Appendices G.2 and H In WoD-Hard, depicted in Figure [ the agent must learn to use a Wand of Death, which can zap and kill enemies. This involves a complex sequence of actions: the agent must find the wand, pick it up. choose zap item, select the wand in the inventory, and finally choose the zapping direction (towards the minotaur which is pursuing the player). must then proceed past the minotaur to the goal to receive reward. Taking these actions out of order (e.g. trying to zap with nothing in the inventory, or selecting something other than the wand) has no effect.
It is difficult to enumerate all MiniHack messages, as they are hidden in low-level game code which has many edge cases. As an estimate, we can examine expert policies: agents which have solved WoD tasks encounter around 60 messages, of which only 5-10 (8-16%) are needed for successful trajectories, including inventory (f a metal wand, what do you want to zap?, in what direction?) and combat (You kill the minotaur!, Welcome level 2.) messages; most are irrelevant (e.g. picking up and throwing stones) or nonsensical (There is nothing to pick up, That a silly thing to zap). In the other tasks, only 8-18% of the hundreds of unique messages needed for success (Appendix G.2).
Unlike the MiniGrid environments, we adapt baseline models from [4] which all already encode the in- mn-game message into the state representation. Despite this, as we will show, using language as an explicit target for exploration outperforms using language as a state feature alone.
7 Results
Figure 3 shows training curves with AMIGo, NovelD, language variants, and the IMPALA and naive message reward baselines. Following Agarwal et al. [2]. we summarize these results with the interquartile mean (IQM) of all methods Figure 4 with bootstrapped 95% confidence intervals constructed from 5k samples per model/env combination We come to the following conclusions:
7 "See Appendix for numeric tables and area under the curve (AUC)probability of improvement plots. SSee Appendix for ablation studies of L-AMIGo's grounding network and the components of L-NovelD.
7

---

KeyCorddrSSSRS
KayCoriderS383
KoydardaeSS4RS
keyCorrido-4In3
KeyCoridorS5n3
ObeinuctedMsse 1DI
ObstuceedMaze_ 10
ObstructedMaaz 2Dhb
ObstuctedMaaze_200 20hhb
ObstructedMaze_ 10
Obstructed 10
River
Medium-
WaD-Medium
WaD-Hard
Queet-Easy
Quest-Ear
Quest Medium
Quest Medium
MutiiRoom-NZ NZ--Extemee
Muti Ex-iimmme
MuttiRoom-NA4
Exxrame-
MutiRoom-Mee
Extreme
MiniGrid-
MiniGrid
MiniHack
MiniHack
Overall
Overall
.75
5
77
Extrinsic Reward
Extrinsic Reward
L-
L-NovelD
L-AMiGo
L-VoveID
One-hat
Figure 4: Aggregate performance. Interquar- tile mean (IQM) of models across tasks. Dot is median; error bars are 95% bootstrapped Cls.
Figure 5: One-hot performance. Models com- pared to variants with one-hot non-compositional goals. Plot elements same as Figure 47
Linguistic exploration outperforms non-linguistic exploration. Both algorithms,, L-AMIGo and L-NovelD,, outperform their nonlinguistic counterparts. Despite variance runs and across environ- ments, averaged across all environments (Overall) we see statistically significant improvement of L-AMIGo over AMIGo (.27 absolute, 47% relative) and of L-NovelD over NovelD (.35 absolute, 85% relative). In some tasks, Figure B shows shows that L-AMIGo and L-NovelD reach the same asymptotic performance as their non-linguistic versions, but with better sample efficiency and stability (e.g. KeyCorridorS3R3 L.-AMGGo, Quest-Fasy L-NoveeID; see Appendix D.3 AUC plots). Lastly, the failure of the naive message reward shows that indiscriminate reward shaping fails tasks with sufficiently diverse language; instead, some notion of novelty or difficulty is needed to make progress.
Linguistic exploration excels in larger environments. Our tasks include sequences environments with the same underlying dynamics, but larger state spaces and thus more challenging exploration problems. general, larger environments result in bigger improvements of linguistic over non- linguistic exploration, since the space of messages remains relatively constant even as the state space grows. For example, there no difference in ultimate performance for Sanguageennon-angguags variants on KeyCorridorS3R3, yet the gaps grow as the environment size grows to KeyCorridorS3R3, especially in L.-NovelD, A similar trend can seen the WoD tasks, where AMIGo actually outperforms L-AMIGo in WoD-Medium, but in WoD-Hard is unable to learn at all.
7.1 Interpretability
One auxiliary benefit of our language-based methods is that the language states and goals can provide insight into an agents' training and exploration process. We demonstrate how L-AMIGo and L-NovelD agents can be interpreted in Figure
Emergent L-AMIGo Curricula. Campero et al. showed that AMIGo teachers produce an interpretable curriculum, with initially easy (z. y) goals located next to the student's start location, and later harder goals referencing distant locations behind doors. In L-AMIGo, we can see a similar curriculum emerge through the proportion of language goals proposed by the teacher throughout training. In the KeyCorridorS4R33 environment (Figure (1). the teacher first proposes the generic goal open (any) door before then proposing goals involving specific colored doors (where <C> is a color). Next, the agent discovers keys, and the teacher proposes pick[ing) up the key and putting it in certain locations. Finally, the teacher and student converge on the extrinsic goal pick up the ball.
Due to the complexity of the WoD-Has -ard environment, the curriculum for the teacher is more exploratory (Figure Go). The teacher proposes useless goals at first, such as finding staircases and slings. At one point, the teacher proposes throwing stones at the minotaur (an ineffective strategy) before devoting more time towards wand actions (you see here wand, the wand glows and fades). Eventually, as the student grows more competent, the teacher begins proposing goals that involve directly killing the minotaur (you kill the minotaur, welcome to experience level 2) before converging on the message you see a minotaur corpse- the final message needed to complete the episode.
L--NoveID Message Novelty. Similarly, L-NovelD allows for interpretation by examining the messages with highest intrinsic reward as training progresses. In KeyCorridorS4R3 (Figure op), the

---

100%
open the
pick
0-
door
action
Teacher Proposed Goals
open the
-2-
prokuup
key
the
pick
put the key
open
g0to
<C> doorgo the
put the
next to the ball
picup
<C> door
key next.
puinext
to the key ug
<C door
the
object
kej
bell
open the door
kay
-6-
mulliple
0%
207
367
2e7
4e7
6e7
Be7
Frames
Frames
(a) L- L-MIGGo, KcyCorridorS4R3
(b) NovelD, KeyCorridor4R33
you
you see here Tunnursed finl stornss.
you have
Teacher Proposed Gools
78%
stalecase up hero.
sing.
minotaur
M1
experience level
welcome to
you 600
the death ray. bouncess
you
liting minolasur corpes.
storee
werdd
minotour porpoe.
what da
log NovelD))
80%
you Wal
zap7
menotaur
glowes and bouneesl
the wand
the death ray
-2.5
you
the fhat stone hits minotaur.
rades.
-5.0
25%
what do you
want
-7.5
- that a silly thing
0%
367
707
167
267
4e7
Frames
Frames
(c) AMMG,G, WoD-Hard
(d) NovelD, WoD-Hard
Figure Interpretation of language-guided exploration. For the KeyCorridorS4R3 and WoD-Hard environments, shown are curricula of goals proposed by the L-AMIGo teacher (a,c) and the intrinsic reward of messages (some examples labeled) for L-NovelD (b,d).
novelty of easy goals such as open the door decreases fastest, while the novelty of the true extrinsic goal (pick up the ball) and even rarer actions (put the key next the ball) remains high throughout training. In WoD-Hard (Figure G(),, messages vary widely in novelty: simple and nonsensical messages like that a silly thing to zap and it's wall quickly plummet, while more novel messages are rarer states that require killing the minotaur (you have trouble lifting minotaur corpse).
7.2 Do semanties matter?
Language not only denotes meaningful features in the world; its lexical and compositional semantics also explain how actions and states relate to each other. For example, in L-AMIGo, an agent might more easily go to the red door if it already knows how to go to the yellow door. Similarly, in L-NovelD, training the RND network on the message go to the yellow door could lower novelty of similar messages like go the red door, which might encourage exploration of semantically broader states. While our primary focus not on whether agents can generalize to new language instructions or states, we are still interested in whether these semantics improve exploration for extrinsic rewards.
To check this hypothesis, in Figure semantics of the language annotations are hidden: each message is replaced with a one-hot identifier (e.g. go the red door + 1, go to the blue door + 2) but otherwise functions identically to the original message. We make two observations. (1) One-hot goals actually perform quite competitively. demonstrating that the primary benefit of language these tasks abstract over the state space, rather than provide fine-gre -raiind semantic relations between states. (2) Nevertheless, L-AMIGo is able to exploit semantics, with a significant improvement (.20 absolute, 32% relative) in aggregate performance over one-hot goals, in contrast to L-NovelD, which shows no significant difference. We leave for future work a more in-depth investigation into what kinds of environments and models might benefit more from language semantics.
we run "one-hot" variants of L-AMIGo and L-NovelD where the

---

8 Discussion
The key insight in this paper is that language, even if noisy and often unrelated to the goal, is a more abstract, efficient, and interpretable space for exploration than state representations. To support this, we have presented variants two popular state-of oh-e----the-rr art exploration methods, L-AMIGo and L-NovelD, that outperform their non- non-linguistic counterparts by 47-85% across 13 hunuaage-annttate tasks in the challenging MiniGrid and MiniHack environment suites.
Despite their success here, our models have some limitations. First, as is common in work like ours, it will important to alleviate the restriction on oracle language annotations, perhaps by using learned Furthermore, L-AMIGo specifically cannot handle tasks such as the full NetHack game which have unbounded language spaces and many redundant goals (e.g. go to/appraach/arrive at the door), since it selects single goal which must be achieved verbatim. An exciting extension to L-AMIGo would propose abstract goals (e.g. kill [any] monster or find new item), possibly in continuous semantic space, that can be satisfied by multiple messages.
state description models [31] 1
More general extensions include better understanding when and why language semantics benefits exploration (Section 72 and using pretrained models to imbue semantics into the models beforehand Additionally, although the agents in this work are able to explore even when not all language is useful, we must take caution in adversarial settings where the language is completely unrelated to the extrinsic task (and thus useless) or even describes harmful behaviors. Future work should measure how robust these methods are to the noisiness and quality of the language. Nevertheless, the success of L-AMIGo and L-NovelD demonstrates the power even noisy language in these domains, underscoring the importance of abstract and semantically-meeninggii measures of exploration in RL.
40
Acknowledgments and Disclosure of Funding
We thank Heinrich Küttler, Mikael Henaff, Andy Shih, Alex Tamkin, and anonymous reviewers for constructive comments and feedback, and Mikayel Samvelyan for help with MiniHack. JM is supported by Open Philanthropy AI Fellowship.
References
[1] J. Achiam and S. Sastry. Surprise- intrinsic motivation for deep reinforcement learning. arXiv preprint arXiv: (70300/77337732. 2017.
[2] R. Agarwal, M. Schwarzer, P. S. Castro, A. Courville, and Bellemare. Deep reinforeement learning at the edge the statistical precipice. In Advances in Neural Information Processing Systems, 2021.
[3] D. Bahdanau. F. Hill, J. Leike, E. Hughes. A. Hosseini, P. Kohli, and E. Grefenstette. Learning to understand goal specifications by modelling reward. In International Conference on Learning Representations, 2018
[4] Bellemare, S. Srinivasan, G. Ostrovski, Schaul, D. Saxton, and Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, 2016.
[5] V. Blukis, Y. Terme, E. Niklasson, R. A. Knepper, and Y. Artzi. Learning to map natural language instructions to physical quadcopter control using simulated flight. In Proceedings of the 3rd Conference on Robot Learning, 2019.
[6] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018
A. Campero, Raileanu, H. Kuttler, J. B. Tenenbaum, Rocktäschel, and E. Grefenstette. Learning with AMIGo: Adversarially motivated intrinsic goals. In International Conference on Learning Representations, 2021.
[8] M. Chevalier-Boisver., L. Willems, and S. Pal. Minimalistic gridworld environment for OpenAl gym. minigrid 2018
https://github. htttss///iituuu..co////iiiieiiii
[9] M. Chevalier Boisverr, D. Bahdanau, S. Lahlou, L. Willems, C. Saharia, T. H. Nguyen, and Y. Bengio. BabyAl: A platform to study the sample efficiency of grounded language learning. In International Conference Learning Representations, 2019. 2019.
10

---

[10] K. Cho, B. van Merriënboer, C. Gulcehre. D. Bahdanaau, F. Bougares. H. Schwenk, and Y. Bengio. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods Natural Language Processing, pages 1724-1734, 2014.
[11] D.-A Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by exponential linear units (ELUs). In International Conference on Learning Reprexentations, 2016.
[12] C. Colas, A. Akakzia, P.-.. Oudeyer, M. Chetouani, and O. Sigaud. new approach to language grounding for RL, arXiv preprint arXiv:2006. 07044, 2020.
goal generation:
[13] C. Colas. T. Karch, N. Lair. J.-M. Dussoux. C. Moulin-Friett P. Dominey, and P.-Y. Oudeyer. Language as a cognitive tool imagine goals curiosity driven exploration. In Advances Neural Information Processing Systems, 2020.
[14] C. Colas, T. Karch, Sigaud, and P.-Y. Oudeyer. Autotelic agents with intrinsically motivated goal- conditioned reinforcement learning: short survey. arXiv preprint arXiv:20/2.09830, 2020.
[15] M.-A. Côté, A. Kádár, X. Yuan, B. Kybartas, Barnes, Fine, J. Moore, M. Hausknecht, L. EI Asri, Adada, W. Tay, and A. Trischler. TextWorld: A learning environment for text-based games. In Workshop on Computer Games, pages 4 41-75, 2018.
[16] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT; Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguisticx; Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186, 2019.
[17] L. Espeholt. H. Soyer, R. Munos. K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, 1. Dunning. et al. IMPALA: Scalable distributed deep-rl with importance weighted actor-learner architectures. In Proceedings of the 35th International Conference on Machine Learning, pages 1407-1416, 2018.
[18] M. Fang, T. Zhou, Y. Du, L. Han, and Z. Zhang. Curriculum Curriculum-uiiedd hindsight experience replay. In Advances in Neural Information Processing Systems, 2019.
[19] C. Florensa, D. Held, M. Wulfmeier, M. Zhang, and Abbeel. Reverse curriculum generation for reinforcement learning. In Proceedings of the 1st Conference on Robor Learning, pages 482-495, 2017.
[20] S. Forestier, R. Portelas, Y. Mollard, and P.-Y. Y. Oudeyer. Intrinsically motivated goal exploration processes with automatic curriculum learning. arXiv preprint 708. 1900 2017.
[21] J. Fu, A. Korattikara, S. Levine, and S. Guadarrama. From language to goals: Inverse reinforcement learning for vision- based instruction following. In International Conference on Learning Representations. 2019.
[22] P. Goyal, S. Nickum, and R. J. Mooney. Using natural language for reward shaping in reinforcement learning. In Proceedings of the Twenty-Eightn International Joint Conference on Artificial Intelligence (ICCAI-19), 2019.
[23] P. Goyal, S. Nickum, and R. Mooney. Pixl2r: by mapping pixels to rewards. In Proceedings of the 4th Conference on Robot Learning, pages 485-497, 2020.
Guiding reinforcement learning using natural language
[24] B. Harrison, U. Ehsan, and M. O. Riedl. Guiding reinforcement learning exploration using natural language. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 1956 1958, 2018
[25] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 98): 1997.
[26] Y. Jiang. S. Gu, K. Murphy, and C. Finn. Language as an abstraction for hierarchical deep reinforcement learning. In Advances in Neural Information Processing Systems. 2019.
[27] H. Küttler. N. Nardelli, T. Lavril, M. Selvatici, V. Sivakumar, T. Rocktäschel, and E. Grefenstette. Torch- Beast: A PyTorch platform for distributed RL. arXiv preprint 1910.091 2019.
[28] H. Küttler. N. Nardelli, A. H. Miller, R. Raileanu, M. Selvatici, E. Grefenstette, and T. Rocktäschel. The NetHack learning environment. In Advances in Neural Information Processing Systems, 2020.
[29] M. C. Machado, M. G. Bellemare, and M. Bowling. Count based exploration with the successor represen- tation. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, pages 5125 5133, 2020.

---

[30] J. Martin, S. N. Sasikumar, T. Everitt, and M. Hutter. Count-based exploration in feature space for reinforcement learning, In Proceedingy of the Twenty-Sixth International Joint Conference on Artificial Intelligence (ICCI-117, pages 2471-2478, 2017.
[31] S. Mirchandani, S. Karamcheti, and D. Sadigh. ELLA: Exploration through learned language abstraction. In Advances in Neural Information Processing Systems, 2021.
[32] G. Ostrovski, M. G. Bellemare, A. Oord, and R. Munos. fount-based exploration with neural density models. Proceedings of the 34th International Conference on Machine Learning, pages 2721-277 1721-2730, 2017.
[33] P.-Y. Oudcyer and F. Kaplan. How can we define intrinsic motivation? Proceedings of the 8th International Conference on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems, 2008.
[34] P.-Y. Oudeyer and Kaplan. What is intrinsic motivation? typology of computational approaches. Frontiers in Neurorobotics, 1:6, 2009.
[35] P.-Y. Oudeyer, F. Kaplan, and V. V. Hafner. Intrinsic motivation systems for autonomous mental develop- ment. IEEE Transactions on Evolutionary Computation, 1 11(2):265-866, 2007.
[36] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, Curiosity-driven exploration by self-supervised prediction. In Proceedings of the 34th International Conference on Muchine Learning, pages 2778-2787, 2017.
[37] R. Portelas, C. Colas, K. Hofmann, and P.-Y. Oudeyer. Teacher algorithms for curriculum learning of deep RL in continuously parameterized environments. In Proceedings of the 4th Conference on Robot Learning. pages 835-853, 2020.
[38] R. Portelas. C. Colas, L. Weng. K. Hofmann, and P.-Y. Oudeyer. Automatic curriculum learning for deep RL: A short survey. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (ICCA-220) Survey Track, pages 4819 4825, 2020.
[39] S. Racaniere, A. K. Lampinen, A. Santoro, D. P. Reichert, V. Firoiu, and T. P. Lillicrap. Automated curricula through setter-solver interactions. In International Conference on Learning Representations, 2020.
[40] R. Raileanu and T. Rocktäschel. RIDE: Rewarding impact-driven exploration for procedurally-generaaed environments. International Conference on Learning Representations, 2020.
[41] Samvelyan, R. Kirk, V. Kurin, J. Parker-Holder, M. Jiang, E. Hambro. Petroni, H. Kuttler, E. Grefen- stette, and Rocktäschel. MiniHack the planet: A sandbox for open-ended reinforcement learning research. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), 2021.
[42] J. Schmidhuber. A possibility for implementing curiosity and boredom in bodel-bbilding neural controllers. In Proceedings of the International Conference on Simulation of Adaptive Behavior: From Animals to Animats, pages 222-227, 1991.
[43] J. Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Transactions on Autonomous Mentul Development, 2(3):230-22 247. 2010.
[44] E. Schwartz, G. Tennenholtz, C. Tessler, and S. Mannor. Language is power: Representing states using natural language in reinforcement learning. arXiv preprint arXiv:1910.022 2019.
[45] M. Shridhar, X. Yuan, M.-A. Côté. Y. Bisk, A. Trischler, and M. Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021.
[46] B. C. Stadie, S. Levine, and P. Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.00814, 2015.
[47] A. L. Strehl and M. L. Littman. An analysis of model based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8): 1309 1331, 2008
[48] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT Press, 1998.
[49] A. C. Tam, N. C. Rabinowitz, A. K. Lampinen, N. A. Roy, S. C. Chan, D. Strouse, J. X. Wang, A. Banino, and F, Hill, Semantic exploration from language abstractions and pretrained representations. arXiv preprint arXiv:2204.0 05080, 2022.
12

---

[50] H. Tang, R. Houthooft, D. Foote, A. Stooke, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel. #exploration: A study of count-based exploration for deep reinforcement learning. In Advances in Neural Information Procexsing Systems, 2017.
[51] I. Tasrin, M. S. A. Nahian, H. Petera, and B. Harrison. Influencing reinforcement learning through natural language guidance. arXiv preprint arXiv:2104.01500 2021.
[52] J. Urbanek, A. Fan, S. Karamcheti, S. Jain, S. Humeau, E. Dinan, T. Rocktäschel, D. Kiela, A. Szlam, and J. Weston. Learning to speak and act fantasy text adventure game. Proceedings of 2019 Conference on Empirical Methods in Natural Longuage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 673 683, 2019.
[53] N. Waytowich, L. Barton, V. Lawhern, and G. Warnell, A narration- based reward shaping approach using grounded natural language commands. arXiv preprint 19Xiv:9111.00499 2019.
[54] T. Zhang. H. Xu, X. Wang. Wu, K. Keutzer, J. E. Gonzalez, and Y. Tian. NovelD: A simple yet effective exploration criterion. In Advances in Neural Information Processing Systems, 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations your work? [Yes] In Section S we discuss limitations
of oracle annotator, and in Section g discuss broader limitations.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] In Section g we discuss caution in adversarial settings where the language signal is useless for learning or, worse, encourages harmful behavior.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]] Code included with supplementary material and will be made public upon acceptance, with link in Appendix (currently anonymized)
C
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] In Appendices B C
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] We follow best practices as suggested by Agarwal et al.
2 for reporting IQM with 95% boostrapped CIs in Figures
S2
S3
S6
S7
and
Table ST Standard error bars are shown in main curves in Figure
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] Appendix C.3
4. If you using existing assets (e.g., code, data, models) curating/releusing
assets....
(a) If your work uses existing assets, did you cite the creators? [Yes] Yes, we cited
Campero al. 77. Zhang al. [54], Samvelyan et [41, Chevalier-Boisver et al.
88 Chevalier-Boiste the supplementary material
et al. [ and codebases we have used are included and credited
(b) Did you mention the license of the assets? [Yes] Yes, code licenses are included in the supplementary material.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] See checklist ((a).
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] No human data
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] No human data
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "ljuTUBiKL3r", "reviewer": "Reviewer_c6cu", "summary": "This paper proposes the use of _language_ to improve exploration via intrinsic rewards. The paper proposes two methods that make use of sentences describing the goals in particular states.", "questions": "- Do the baselines make any use of the descriptive sentences in the environment? Is it part of the environment state? As mentioned above, I am concerned that the increased performance stems from the high quality of the language annotations and not from the methodology itself. Could the authors elaborate on this?\n- In Discussion (Sec 8), the authors mention that it might be possible to lift the restriction of an oracle language annotator by using a learned state description. Instinctively, I would say that since this is used as an intrinsic reward, such a model would require multiple visitations to that particular state anyway and would lose its novelty along the way. Am I misunderstanding something? If yes, such a result would be truly impressive and an important contribution.\n", "limitations": "Yes, the authors have addressed the limitations and potential negative societal impact of their work.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n\n- The paper is very well written, and clearly motivates the problem and solution.\n- Exploration in RL is an ever-important problem, and the paper proposes a solution that improves upon state-of-the-art methods (but with some limitations, like the assumption of an \"oracle language annotator\")\n- An impressive set of results in two significantly hard environments (however, I have some concerns about the experiments)\n\nWeaknesses:\n\n- L-AMIGo and particularly L-NovelD seem to be significantly similar to their respective base algorithms, and just offer a way to use state information in the form of language.\n- L-NovelD uses NovelD(s_t, s_{t-1}) (eq7) which I find slightly concerning. I would hope that NovelD(l_t, l_{t-1}) would give a strong learning signal by itself, but as the authors explain it does not lead to equally high performance. This ties to my main concern (below) that the two methods mostly benefit from the (extra) data provided by the oracle and not because they are by themselves enough to provide an intrinsic reward.\n- My main concern is that the baselines (AMIGo/NovelD/IMPALA) do not use the information used by their L- extensions. After reading the paper, it is still not clear to me what the main contributing factor to the improved performance is (also see questions below).\n\nI have enjoyed reading this paper, but I believe it needs to address the issue of my last bullet point. Even a comparison to baselines that at least make use of the generated sentences in a naive way would be somewhat useful.\n\nI would be open to increasing my score if the authors give a satisfying answer to my concerns.\n\n** Update after rebuttal: This reviewer has increased the overall rating from 4 to 7. **", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "eK4qJkDffII", "reviewer": "Reviewer_auiq", "summary": "The work proposes that the presence of (consistent) language, even if task-irrelevant and noisy, can provide a signal for exploration. They evaluate against two intrinsic exploration methods (Amigo and NovelD) by creating language abstraction variants of these methods. They run a thorough evaluation on mini-grid and minihack and find that exploration around language-based abstraction are indeed helpful. \n\n\n", "questions": "Interesting paper. Some points:\n\nIn your intro, you state that state-based exploration would reward going to every room but the agent should stay by the stove. Changing the objects on the stove achieves novel states so I'm not sure your example is correct. If we had access to ground-truth state, state-based exploration might very well lead to exploration by the stove. While I agree with your goal of using language, I don't see how what you said really supports language.\n\nI could but wrong but it feels like you're using \"state\" in multiple ways: e.g., the observation the agent sees and underlying latent state that's generated the observation. Am I wrong about?\n\nIt seems a bit restrictive to also have an oracle grounding classifier. Do you ablate this? \n\nFigure 5 has very similar colors. Overall, one-hot performance seems quite similar to using real-language. Does this change the story/message of your paper? The results are more-or-less the same indicating that it isn't necessarily language that's the driver of performance. Given that, I think your paper conclusions are not as well aligned with your story as you portray in the introduction. Don't get me wrong, I like this paper and think it's fairly thorough and clear but I think the findings don't align with the message. Could you clarify on this point?\n\nThe authors mention that there are several prior methods which use language for reward-shaping but they don't compare to any. How come?\n\nComment:\n* Figure 6 has a lot going on. Challenging to make sense of.\n\nSuggestions:\n* Figure 2 is not as clear as it could be. Recommend describing it a bit more.\n* Label \"large\" vs not large environments in Figure 3.\n", "limitations": "The limitations described seem adequate.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* Related work is thorough and fairly clear.\n* The methods are clear and clean extensions of prior work.\n* The results are thorough and well-organized\n\n\nWeaknesses:\n\n* Related work is missing exploration based on state abstractions\n* The contributions of this paper could be clarified\n\nMore on contribution: \nThe main finding is that exploration around language/state abstraction helps in large state spaces. We know that language abstraction helps in exploration--the authors reference many papers on this. In terms of state-abstraction helping in exploration, I think we also know this [1, 2]. The 1-hot id description conditioned student agent is like a parameterized skill that is being taught to go to abstract states. That said, I don't think state abstractions have been combined with these newer methods for exploration (Amigo/NovelD/etc.), so I think a discussion on this would clarify the contributions of this paper for me.\n\n\n\n[1] Discovering Options for Exploration by Minimizing Cover Time\n[2] Identifying useful subgoals in reinforcement learning by local graph partitioning", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MAFHpbc1Ra", "reviewer": "Reviewer_dbwv", "summary": "This paper posits the hypothesis that language abstractions are superior to raw states to guide exploration motivated by intrinsic reward. \n\nThe authors extend two recent and performant algorithms, AMIgo and Nove1D, to use language abstractions instead of states as a means of computing intrinsic rewards. \n\nThey show on a suite of experiments that, indeed, these language-based algorithms outperform their state-based counter-parts thereby supporting their hypothesis. \n\nMoreover, while previous works using language as an abstraction for exploration have focused only on comparing their methods to vanilla RL, here a useful systematic study of the performance of language abstraction vs. state based intrinsic reward exploration is presented.\n\n", "questions": "1. In L-AMIgo, I’m curious about the impact of the “grounding network”. Is it critical performance? Did you by any chance do ablation study on the components?", "limitations": "In Section 8, the authors are up-front about the fact that the L-variants described in their paper is so far only applicable in environments that come with a language \"oracle\". Indeed, this seems like a significant limitation that perhaps explains why most literature focuses on state based measure for generating intrinsic rewards.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. In terms of quality and soundness, I cannot find much to fault with the paper. The claims of the paper are clearly stated and quite strongly supported by empirical results (v. Figure 3). That is, indeed language abstractions appear to be better than state as a basis for generating intrinsic reward. \n2. In terms of clarity,  the paper is largely clear and concise. It is easy to follow the development of the logic throughout. The authors carefully provide all necessary details to understand the development of their new algorithms. Moreover, all figures, captions, and all plots in general are clean and easy to understand.\n\n**Weaknesses**\n\n1. In my opinion, the originality of this paper is limited. It could be argued that replacing state as the means of computing intrinsic reward with language amounts to simply changing the input space of AMIgo and Nove1D. I do recognize that some innovations are required for the adaptation of the base algorithms to language abstractions (e.g., the grounding network), but I am not convinced that these are large novel steps.\n2. Notwithstanding the above, if there was a method to develop language abstractions for environments in general, the applicability of the algorithms developed in this paper may be rationalized. However, the L-variants are limited to environments which come pre-packaged with language abstractions thereby limiting their scope. In fact, in Zhang et al. [1], Nove1D is flexibly used in environments which have language abstractions (MiniGrid) and which don’t have language abstractions (Atari), while L-Nove1D cannot be used as flexibly.\n\nOverall, my main contention is that the originality and novelty are somewhat limited.\n\n[1] T. Zhang, H. Xu, X. Wang, Y. Wu, K. Keutzer, J. E. Gonzalez, and Y. Tian. NovelD: A simple yet effective 478 exploration criterion. In Advances in Neural Information Processing Systems, 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept