ELLA: Exploration through Learned Language Abstraction
Authors: Suvir Mirchandani
Siddharth Karamcheti
Dorsa Sadigh
Venue: NeurIPS 2021 Poster (2021)
Keywords: instruction following
reward shaping
reinforcement learning

Abstract: Building agents capable of understanding language instructions is critical to effective and robust human-AI collaboration. Recent work focuses on training these agents via reinforcement learning in environments with synthetic language; however, instructions often define long-horizon, sparse-reward tasks, and learning policies requires many episodes of experience. We introduce ELLA: Exploration through Learned Language Abstraction, a reward shaping approach geared towards boosting sample efficiency in sparse reward environments by correlating high-level instructions with simpler low-level constituents. ELLA has two key elements: 1) A termination classifier that identifies when agents complete low-level instructions, and 2) A relevance classifier that correlates low-level instructions with success on high-level tasks. We learn the termination classifier offline from pairs of instructions and terminal states. Notably, in departure from prior work in language and abstraction, we learn the relevance classifier online, without relying on an explicit decomposition of high-level instructions to low-level instructions. On a suite of complex BabyAI environments with varying instruction complexities and reward sparsity, ELLA shows gains in sample efficiency relative to language-based shaping and traditional RL methods.

ELLA: Exploration through Learned Language Abstraction
Suvir Mirchandani
Siddharth Karamcheti
Computer Science
Computer Science
Stanford University
Stanford University
suvir@cs.startee edu
skaramcheti wtteeeeiissssssttfteeeeeeeeeeeee stanford.edu
Dorsa Sadigh
Computer Science and Electrical Engineering
Stanford University
dorsa0cs crrraaocssstaffrd..co
Abstract
Building agents capable of understanding language instructions is critical to effec- tive and robust human-Al AI collaboration. Recent work focuses on training these agents via reinforcement learning environments with synthetic language; how- ever, instructions often define long-horizon, sparse-reward tasks, and learning policies requires many episodes of experience. We introduce ELLA: Exploration through Learned Language Abstraction, a reward shaping approach geared towards boosting sample efficiency in sparse reward environments by correlating high-level instructions with simpler low-level constituents. ELLA has two key elements: 1) A termination classifier that identifies when agents complete low-level instructions, and 2) A relevance classifier that correlates low-level instructions with success high-level tasks. We learn the termination classifier offline from pairs of instruc- tions and terminal states. Notably, in departure from prior work in language and abstraction, we learn the relevance classifier online, without relying on an explicit decomposition of high-level instructions low-level instructions. On suite of complex BabyAI [11] environments with varying instruction complexities and reward sparsity, ELLA shows gains in sample efficiency relative to language-based shaping and traditional RL methods.
Introduction
long-standing goal for robotics and embodied agents is to build systems that can perform tasks specified in natural language [1, 20, 25, 38, 41, 42]. Central to the promise of language is its ability to cleanly specify complex, multi-step instructions. Instructions like make a cup of coffee define long-horizon tasks abstractions over lower-level components-sinn simple instructions like pick up cup or turn on the coffee maker. Leveraging these abstractions can help amplify the sample efficiency and generalization potential of autonomous agents.
One way to do this is through the lens of instruction following. which can be framed in several ways. One common framing-ann the one we use in this work-is via reinforeement learning (RL): an agent is given a start state, a language instruction, and corresponding reward function to optimize that usually denotes termination [20, 30]. While RL can be useful framing, such approaches often not sample efficient [11, 20]. Especially the case of complex, highly compositional language instructions, RL agents can fail make progress quickly-oo
all. There are several reasons for poor performance in these settings; paramount is that in many environments, these instructions are
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

ab Expert
Low-Level
Termination Dataset
RL Agant
Reward Shaping
High-Level put
next
g green key
Sahtnak Completion
Task Completion
Task Failure
W
-Vy+ 17e
go
X
subtnols
go to the blue ball
Has tecminated for micccnt??
red ball
Which subtasks did complete?
go to the red ball
No Change
X
Subiak Rewaed
Bewand Subiakk
Contrestioe Learning
Qwerr
Query
Update Detrent
Termination Classifier
Relevance Classtfict,
Relevance Dataset,
Contrastize Learning
(Onlinc)
In this example, an expert provides examples of when low-level go to tasks are and are not solved, and the agent trains termination classifier. During policy learning, the agent rewards itself for completing go to subtasks- specificcsse those which are relevant to its current high-level task. stores successful experiences learn correlate low-level go tasks with high-level put next tasks.
Figure 1:
tied to sparse reward functions that only provide signal upon completion of the high-level task, which drastically hurts sample efficiency. For example, seeing reward for make a cup of coffee would require already having turned on the coffee maker, which could itself be a difficult task. Such "bottleneck" states, which have intermediate reward, complicate exploration and learning [29, 39].
Our goal is to improve sample efficiency for instruction following agents operating in sparse reward settings. Driving approach is using the principle of abstraction the fact that complex instructions entail simpler ones-to guide exploration. Consider our coffee example; we would like to guide the agent explore in structured fashion, learning low-level behaviors first (pick up cup, turn the coffee machine) and building up to solving the high-level task. To do this, we frame our problem via reward shaping, the general technique of supplying auxiliary rewards to guide learning [31, 35].
Our approach, Exploration through Learned Language Abstraction (ELLA) provides intermediate rewards to an agent for completing relevant low-level behaviors as it tries to solve complex, sparse reward task. Notably, our approach 1) learns to identify the low-level primitives helpful for high-level language task online, and 2) does not require strict hierarchical decomposition of language instructions to these primitives. Rather, ELLA uses low-level instructions to support agents performing complex tasks, bonusing agents they complete relevant behaviors. In contrast to prior work [3, 23] that assumes strict contracts over instruction decomposition, our contract simple and general. This prior work assumes each high-level task comprised of exact series of low-level policies an assumption that only possible when the full set primitives is known ahead of time, and fails when new actions further exploration necessary.
In order to bonus agents as they complete relevant subtasks, ELLA assumes access to set of low- level instructions and corresponding termination states, similar to the data assumed in prior work [6]. While collecting such data for complex tasks may require time and effort, annotations for low-level instructions more tractable. Humans quickly annotate instructions like pick up the cup or open the cupboard, or even build tools for generating such examples. This increasingly apparent in prior work [5, 22, 25, 43]; low-level instructions are used to simplify the problem of interpreting high-level instructions, through a variety of mechanisms. In our case, these low-level instructions provide the basis for reward shaping.
empirically validate our abstraction based reward shaping framework on series tasks via the Baby AI platform [11] We compare against standard RI baseline well to strong language- based reward shaping approach [17], and find that our method leads to substantial gains in sample efficiency across variety of instruction following tasks.

---

2
Related Work
We build on a large body of work that studies language in the context of reinforcement learning (RL) [27] with a focus on methods for instruction following, leveraging hierarchy, and reward shaping.
Instruction Following. Instruction following focuses on the automated execution of language commands in simulated or real environments [1, 16, 26, 41, 44]. We focus on the RL setting [11, 30], where agents learn
policies in a Markov decision process (MDP).
Even simple instruction following problems are difficult learn [28], which has made 2D envi- ronments important test-beds for new learning methods [3, 11, 46]. Such environments, which are procedurally- aroeddurally-generaaedd, are useful because they decouple exploration from the problem of perception, and demand policies that do not overfit to narrow regions in the state space [10, 13].
We use BabyAI [1 I] the platform for our tasks. BabyAl's language is synthetie, similar to prior work examining RL for instruction following at scale [20, 21]. These environments are scalable starting points for new instruction following approaches, since deep RL remains sample-nnefficient but has the potential to flexibly deal with complex environments [21]. Simplified environments also make investigating concepts such as language abstraction possible, and these concepts can generalize to real auman-Al interactions [25, 43]. We describe our experimental setup in Section 5.
Hierarchical and Language-Structured Policies. A large body of work in hierarchical RI. studies temporal abstractions of MDPs, where policies are designed or learned at multiple levels of hierarchy [40]. Recent works have made the connection to language: for example, high-level policies can act over the space of low-level instructions [23]. Other hierarchical language techniques include learning models that predict symbolic representations of reward functions at different granularities [5, 24], and compositional approaches like policy sketches [3] and modular control [15], methods that explicitly decompose high-level instructions into sequences of predefined low-level instructions, or subtasks.
In this work, we use language hierarchically, but without the strict decomposition assumption made by prior work: we do not assume high-level tasks are composed of an exact series of low-level tasks. Without this assumption, it is infeasible to use explicitly modular approaches [3, 5, 15]. A further distinguishing factor of our method relative to prior work that we do not require high-level task decompositions be provided a priori. We expand on these features in Section 3.
Language and Reward Shaping. Reward shaping is general technique for supplying auxiliary rewards in order to guide an agent's 's learning process. Certain types of reward transformations, such as potential-based rewards, do not change the optimal policy but can impact sample efficiency [31]. Prior work has applied reward shaping to language-basse instruction following settings. Goyal et al. [17] train a classification network on (trajectory, language) data to predict if a trajectory (parameterized by action frequencies) matches language description, and evaluate the network at every time step to form a potential-based reward shaping function. They evaluate their method on Montezuma's ' Revenge, which is a complex task, but with static, deterministic layout that does not speak to the generalization potential of such methods. Relatedly, Waytowich et al. [45] use narration- based approach, where high-level tasks are explicitly broken down into low-level tasks as narrations. Finally, Misra et al. [30] use a shaping function based on spatial distance to goals specified in language, but require the location of the goal to be known. In this work, we use the same high level principle- language can yield useful intermediate rewards-bre do not make the restrictive assumptions of prior work about the environment or the way tasks are structured.
Other Forms of Guidance. Several other methods exist for improving the sample efficiency of RL in sparse reward settings. These methods reward an aspect of exploration that is orthogonal to our approach, and we see potential for combining these methods with ours. One popular technique is hindsight experience replay (HER), in which failed trajectories are stored in a replay buffer and relabeled with new goals that get nonzero reward [4]. Jiang et al. [23] extend the method to hierarchical language settings, assuming a mapping from states to corresponding goals. Cideron et al. [12] learn this mapping alongside policy training using the environment reward as supervision. Another technique for improving sample efficiency is through intrinsic motivation [9, 32, 34, 36). In general, these methods instantiate reward shaping to incentivize accessing novel, diverse, or unpredictable parts of the state space via intrinsic rewards, and they can be extended to language-
3

---

conditioned settings as well [14]. In Appendix D, we evaluate the performance of RIDE [34], an intrinsic motivation method that rewards actions that cause significant changes to the state, and discuss potential synergies with ELLA.
3 Problem Statement
We consider an augmented MDP M defined by the tuple (S, R, where A. and yy standard. A consists of primitive actions-in Baby these navigation primitives (forward, pick up, etc.). 9 is a set of language instructions, from which a high-level task instruction is drawn, and R S A x G + [0. 1] represents the state-action reward given some g. Via RL, wish to find some policy и : x g + A that maximizes the expected discounted return.
We specifically consider cases where M has a finite horizon H and R is sparse with respect to goal-completion, making exploration difficult. Our aim to construct a reward transformation R + Rt' which is policy invariant with respect to the original MDP (i.e. an optimal policy for M' = (S, A, T, R', 9,7) is also optimal in M and vice versa), while providing the agent strong signal conducive to sample-efficient exploration.
We assume access a of low-level instructions Ge such that every g E supported by some Ie E Ge. Note that we do not require that high-level tasks fully factorize into low-level tasks. To clarify the distinction, say that GE consists go to "x" instructions: go the red ball, go to the blue ball, etc. The instruction go to the red ball and then go to the blue ball can be fully factorized into low-level tasks Gr. On the other hand, the instruction pul the red ball next to the green key is supported by go to the red ball, but it also requires picking up and putting down the ball-action not covered by Gr. Our framing permits exploration using mixture of low-level instructions as well as primitive actions in A (such pick up and put down).
We assume that examples of the corresponding termination states of instructions in Ge are easy to Abtain-via a human expert or automation. This is reasonable for simple low-level tasks like the go to tasks in Figure However, it is less feasible in domains where data collection is costly and the environment is hard to simulate; we address this further in Section 7.
The environments in our experiments are partially observable, so ingest sequences of observations (01, 02, ..., 0+) rather than a single state [19]. Though our notation describes the fully observable setting, the extension the partially observable case is straightforward.
recurrent networks that
ELLA
We present ELLA, our reward shaping framework for guiding exploration using learned language abstractions." Figure provides a graphical overview of ELLAA. Our approach rewards an agent when it has completed low-level tasks that support given high level task. To do this, it predicts 1) when a low-level task terminates and 2) when a low-level task is relevant. Section 4.1 describes our low-level termination classifier. Section 4.2 describes our relevance classifier, and how we learn it online during RL. Finally, Section 4.3 details our reward shaping scheme, which bonuses an agent for exploring states that satisfy relevant low-level tasks.
4.1 Learning the Low-Level Termination Classifier
We train binary termination classifier h Ge {0, 1} parameterized by & predict if low-level task ge E Ge terminates in a particular state s E S. We assume access to Ge as well as positive and negative examples of low-level task termination states. This dataset could be annotated by a human or created automatically. While providing demonstrations of high-level tasks (e.g., make cup of coffee) across varied environments is costly, is more feasible do for low-level tasks (e.g., pick up the mug) which are shorter, simpler, and more generalizable. To represent h we adapt the architecture from [11] which merges state and language representations using feature-wise linear modulation (FiLM) [33].
OOu code is available https: com wwt./ooo/nt/eeeeeeee hom/Stanford---o ILIAD/ELAL /ELLA.

---

Algorithm 1 Reward Shaping via ELLA
Initial policy parameters 8. relevance classifier parameters Po. update rate n, low- level bonus X, and RI. optimizer OPTIMIZE Initialize D <- {(g 9e) for all g in g} for k = 0, do
Input:
function SHAPE(T)
Set S + 0 for de E Gr do
for g. (8t, vo) E do
if (s,, 9t) = and 9 & $ then
Collect trajectories Dx using T for D3 do
Update S t $ u {gr]
Set t length of
Set + rt + X [g g6)
Set (1 $) f SHAPE(T)
return (r(.N= S)
if (T) 0 then
function NEUTRALIZE(''
Set 'N < NEUTRALIZE(''.N
Set Is +- {t St S N, 0}
Set D[g] + UPDATEDECOMP(D, 8)
return ''N - ELETs -
Update 0k+1 f OPTIMIZE(('.N) if k: is a multiple of n then
function UPDATEDECOMPCDD, $)
Update Pk+11 by optimizing cross entropy loss on a balanced sample from D.
Set S + Dig return & ns (or 3 nr = 0)
4.2 Learning the Relevance Classifier
Our reward shaping approach also needs to evaluate relevance of low-level instruction in addition to being able to classify termination. The relevance mapping from G to P(Ge) (the power set of Go) initially unknown to the agent. We refer to the output of this mapping as a decomposition of g, but note again that it need not be a strict decomposition. We propose that this mapping is represented by a separate binary relevance classifier fo G Ge + {0, 1} learned during policy training from a dataset of decompositions 'D collected online. The output of f indicates whether particular Je E Ge relevant to some g E G. We use a Siamese network to represent f..
Training the Relevance Classifier. Suppose D already contains key-value pairs (g, S) where E G and $ P(G)). Each & is estimate of the oracle decomposition S of particular g. For every g in D, we enumerate negative examples of relevant subtasks from G. S and accordingly oversample positive examples from S; then optimize cross entropy objective on this balanced dataset.
Collecting Relevance Data Online. We describe any trajectory that solves the high-level task as successful, regardless of whether they do so optimally, and any other trajectories as unsuccessful. For every successful trajectory of some g we encounter during RL training, we record the low- level instructions which terminated. That is, we relabel the steps in the trajectory and use hd to determine which low-level instructions were completed (if any), yielding an estimate $ of the true decomposition S. If already contains an estimate for g, we simply deduplicate the multiple decomposition estimates by intersecting and S. As the agent completes more successful trajectories, decompositions in D are more likely to get revisited.
Intuition for Deduplication. Intuitively, this method for deduplication denoises run-specific experiences, and distills the estimate $ in D for a given to only those Je that are shared among multiple successful runs of g. Consider g = put the red ball next to the blue box; successful runs early in training could highlight go to the red ball along with irrelevant go to low-level instructions. As nd improves, so do the decomposition estimates. Later runs may highlight only go to the red ball. Taking the intersection would yield only go to the red ball as our final estimate S. This approach also helps to deal with false positives of h by effectively boosting the estimate across runs. If deduplication produces a null intersection, which may happen due to false negatives of h we eIT on the side of using the latest estimate. If we assume h has perfect accuracy, the intersection will never null, and the continual intersections will reflect the set of completed gr common to all successful runs of For an initialization conducive to deduplication and that provides T² strong incentive to complete low-level tasks from the beginning of training, we initialize f" to predict that S Ge for all g.
5

---

Low-Level Task
Example
-OTO-RoOM (G1)
go to a purple ball go purple ball open the red door pick up the blue box
GOTo-MAZE (G2)
OPEN-MAZE (02)
PICK-MAZE (P2)
High-Level Task
Example(s)
Gz
PUTNEXT-ROOM
put the purple ball next to the blue key
GI
PUTNEXT-MAZE
put the purple ball next the blue key
G2
UNLOCK-MANE MAZE
open the green door
P2
OPEN&PICK-MAN
open the red door and pick up the blue box
02, G2
COMBO-MAZE
put next, open, pick
G2
MAAE
pick up purple key and then put the grey box next to the red box
G2
Table 1:
Sample low- and high-level instructions.
4.3 Shaping Rewards
We now describe the shaped reward function R' which relies on h and fP. R' provides bonus whenever relevant low-level task terminates. We do this by relabeling the current (st, Gt, Ft, 8t+1) with every Ge and using h to predict low-level task termination, and f" to predict relevance.
Critically, we do not want the shaping transformation from R - Rt' to be vulnerable to the agent getting "distracted" by the shaped reward (reward hacking), as in Randløv and Alstrøm [35] where an agent learns ride bicycle in circle around a goal repeatedly accruing reward. To this end, cap the bonus per low-level instruction per episode to À to prevent cycles. More generally, we want R' to be policy invariant with respect M: it should not introduce any optimal policies, and it should retain all the optimal policies from M. Policy invariance is a useful property [31]: we do not want reward suboptimal trajectory (such more than an optimal trajectory in M, and policy invariance guarantees this.
that repeatedly completes low-level instruction)
We establish two desiderata for R' [7]: (a) The reward transformation should be policy invariant with respect to M for states from which the task is solvable, and (b) R' should improve sample efficiency by encouraging subtask-base that successful trajectories get the same return under M and M', and that unsuccessful trajectories get lower returns under M than trajectories optimal in M. We will describe and intuitively justify each these choices with respect to (a) and (b).
exploration. To satisfy our desiderata,
choose R' such
Neutralization in Successful Trajectories. We neutralize shaped reward in successful trajectories that is, subtract the shaped reward at the final time step- so that successful trajectories gets the same return under both R' and R.
More specifically, let U(T) == N successful trajectory under R is (TT = N r- Under R', Tg the set of the time steps at which a bonus applied, set r'y to TN E ELeT * X. This is the value required to neutralize the intermediate rewards, such that (') y NNN TN = U(r). (Note that we cap the bonus per time step to X-if multiple low-level language instructions terminate at single state, only a bonus of À is applied.)
==== 7²t R(se, at), the cumulative discounted return under R of Let U'(T)) be likewise defined for R'. If T has N steps, the sparse reward
Theoretically, could apply neutralization to all trajectories, not just successful ones, and we would satisfy property (a) [18]. However, this is harmful to property (b), because unsuccessful trajectories would result in zero return: negative reward at the last time step would negate the subtask rewards, potentially hurting boosts in sample efficiency.

---

Tuning À to Limit Return in Unsuccessful Trajectories. Any successful trajectory gets the same return under R' as under R because of neutralization. By choosing carefully, we can additionally satisfy the property that any unsuccessful trajectory gets a lower return under R'' than any trajectory selected by an optimal policy TM in M.
To achieve this, we need À to be sufficiently small. Assume that a trajectory T selected by TM takes no longer than M time steps to solve any g. In the worst case, M = H., and (T) would be y TH- If T' is the set of the time steps at which It' provides À bonuses, ("(7) would be À Eier, 7 We can upper bound Eier, ? with (Gel. Then, the following inequality sufficient for maintaining (a):
where En is the value of a sparse reward if it were attained at time step H. We provide a more thorough justification of this choice of À in Appendix F. Note that À and the sparse reward can be both scaled by a constant if À is otherwise too small to propagate as a reward.
An important feature of our work is the realization that we can make this bound less conservative through minimal knowledge about the optimal policy and the environment (e.g., knowledge about the expected task horizon M, or a tighter estimate of the number of subtasks available in an environment instance). Such reasoning yields a feasible range of values for X, and these can empirically lead to faster learning (Section 6. 6. At minimum, reasoning over the value of À using via (1) provides a generalizable way to incorporate this technique across a variety of different settings in future work.
We summarize our shaping procedure in Algorithm 1, with an expanded version in Appendix C.
5 Experiments
Experimental Setup. We run our experiments in BabyAI [11], grid world platform for instruction following, where an agent has limited range of view and receives goal instructions such as go to the red ball or open blue door. Grid worlds can consist of multiple rooms connected by closed/locked door (e.g.. the UNLOCK-MAzZR environment in Figure 2). The action space A consists of several navigation primitives (forward, pi pickup, etc.). Every task instance includes randomly placed distractor objects that agents can interact with. Rewards in BabyAl are sparse: agents receive reward of 1 0.9 H where i is the time step upon succeeding at the high-level goal. If the goal is not reached, the reward is By default, all rewards are scaled up by a constant factor of 20.
We evaluate our reward shaping framework using Proximal Policy Optimization (PPO) [37], but note that ELLA is agnostic to the RL algorithm used. We compare PPO without shaping. as well as to LEARN, prior method on language-basees based reward shaping [17] that provides rewards based on the predicted relevance of action frequencies in the current trajectory to the current instruction.
We focus on several high- and low-level tasks. Table describes each task with examples. ROOM levels consist of single 7 7 grid, while MAZE levels contain two such rooms, connected by door; agents may need to open and pass through the door multiple times to complete the task. In the UNLOCK environment, the door is "locked," requiring the agent to hold a key of the same color as the door before opening it, introducing significant bottleneck states [29, 39]. We choose six high- and low-level task pairs in order differ along three axes: sparsity of the high-level task, similarity of the low- and high-level tasks, and compositionality of the tasks in G-the the number of ge E Ge relevant to some g. We use these axes to frame our understanding of how ELLA performs in different situations.
We used a combination of NVIDIA Titan and Tesla T4 GPUs to train our models. We ran seeds for each of the 3 methods in each environment, with runs taking 1 to 6 days.
Results. Figure 2 presents learning curves for ELLA, LEARN, and PPO (without shaping) across the six environments. We explain our results in the context of the three axes described above.
+ How does ELLA perform on tasks with different degrees of sparsity?
In both single room (PUTNEXT-ROMM) and two room (PUTNEXT-MAZE) environments, ELLA induces gains in sample efficiency, using GoTo as Ge. Relative gains are larger for the bigger
7

---

PuTNexI-ROOM
UNLOCR-MAZE
10
10-
66-
0.-
64-
go
additional open pick pick the
yellone key
00
0.0
0.0
0.5
ROLLOLT ral bull wet in grva iry
0.0
0.5
1.0
<BOLLOCT: open the yoller done
Frames
X
Prantes
X10
PUTNEXT-MAZE
OPEN&PICK
COMBO
SEQUENCE
10
688
0.-
0.5
08-
0.4
0.4
0.-
0.2
0.2
0.7
Frames
Frames
Prames
Frames
PPO [Baseline]
LEARN [Baseline]
ELLA [Oun)
Figure 2: Average reward for ELLA and baselines in six environments, with error regions indicate standard deviation over three random seeds. Example rollouts in the PUTNEXT-ROOM environment and UNLOCK-MAZA environment illustrate the final policy learned via ELLA, where the agent completes relevant low-level subtasks in order to solve the high-level task.
environment because reward signals are more spatially sparse and so spatial gubgoals--isitinnn relevant objects- the agent navigate.
Another degree of sparsity comes from bottleneck states: for example, in the UNLOCK environment, the agent must pick up a key of the corresponding color (the bottleneck) before it can successfully open colored door and see reward. Without shaping, random exploration rarely passes such bottlenecks. However, in this experiment, ELLA rewards picking up keys, via the PICK low-level instruction, quickly learning correlate picking up keys of the correct color with successfully unlocking doors, allowing agents to navigate these bottlenecks and improving sample efficiency.
+ How does ELLA perform when the low-level tasks are similar to the high-level task?
Tasks in the COMBO environment consist of multiple instruction types (e.g., put the red ball next to the green box, open the green door, and pick up the blue box). The latter instructions require minimal exploration beyond "going to" an object such as executing up primitive actions. That is, the low-level task GoTo more similar to this high-level task set than in the other environments such as PUTNEXT. As result, ELLAA does not increase in sample efficiency in COMBO. However, it notably does not perform worse than basclines: the exploration encourages is not harmful, but is simply not helpful. This is expected as achieving the COMBO tasks about as difficult as exploring the relevant GOTO subtasks.
+ How does ELLA perform when the high-level tasks are compositional and diverse?
We test ELLA on &PEN&PICK using two low-level instruction families, OPEN and GoTo. The instruction open the yellow door and pick up the blue box abstracts over go to the yellow door, open the yellow door, and go to the blue box. Although learning f becomes harder with more compositional instructions, the boost in sample efficiency provided by ELLA remains.
Similarly, SEQUENCE has combinatorially-lrre number compositional instructions: requires two of pul nexi, open, and pick up in the correct order. It has over 1.5 million instructions compared to the other high-level tasks with 300 to 1500 instructions. Although the exploration problem is very difficult, we see marginal gains from ELLA: discuss this further in Scction 7.
8

---

Learning Curves for Various A
Average Decomposition Length
Relevance Classifier Accuracy
10
c015
08-
125
0.6
10.0
0.4
0.4
PUrNExT-MAZE SiquENcE-Mazi
02-
0.
0.0
0.0
0.2
0.6
1.0
Frames
X10P
Frames x10
Frames
Figure 3: Average reward in PUTNEXT-ROOD for various values of Å. Smaller values are more conservative but potentially less helpful than larger values. = 0 indicates no shaping: for this case, show error region over three random seeds.
Figure 4: Two metrics to analyze ELLA's performance over an RL run are the average number of low-level instructions per high-level instruction in 'D and the accuracy of the relevance classifier f on balanced validation set. For PUTNEXT-MAZE, fo improves as decomposition estimates get less noisy. For the challenging SEQUENCE-NV -MAZEE task, ELLA still incentivizes subtask completion, but the decomposition estimates are noisy and fP is less accurate.
Analyzing ELLA
In this section, we analyze two aspects of ELLA: the effect of different choices for the low-level reward hyperparameter (Section 6. I), and the performance of the relevance classifier (Section 6.2).
6.1 Effect of the Low-Level Task Reward
The magnitude of the low-level reward is a critical hyperparameter. Instead of using ad hoc methods for tuning A, Section 4.3 provides guidelines for choosing a reasonable range for X. To summarize, we want to pick À small enough that any unsuccessful trajectory does not receive greater return than any optimal trajectory. In this section, we examine PUTNEXT-ROOM and evaluate various values determined using the multi-tier assumptions alluded to in Section 4.3. For this task, we have that H 128 and Gel 36. With no knowledge of the environment optimal policy, the loosest bounds for M and [Tsl (H and [Ge] respectively), as in (1) yield À = 0.015. With the minimal assumption that optimal policy solves the task in under 100 steps, we arrive at À 0.05. For 40 steps, we arrive at À = 0.25. Figure compares learning curves for X-the smallest being more conservative and not having a great effect on sample efficiency, the middle values illustrating the value rewarding relevant subtasks, and the largest values chosen specifically to demonstrate that egregious violations of policy invariance can "distract" the agent from the high-level task and lead to unstable learning. The results in Section 5 use 00.25 for PUTNEXT, UNLOCK, and COMBO, and 0.5 for OPEN& PICK and SEQUENCE (which have longer horizons H).
6.2 Progress of the Relevance Classifier
The online relevance dataset and classifier provide transparency into the type of guidance that ELLA provides. Figure focuses on PUTNEXT-MAZE and SEQUENCE- the average number of subtasks per high-level instruction in DD which we expect to decrease as decomposition estimates improve; and the validation accuracy of f in classifying a balanced oracle set of subtask dccompositions, which we expect to increase if ELLA learns language abstractions.
and shows two metries:
For PUTNEXT-MAZE, the number of subtasks per high-level instruction decreases and the relevance classifier becomes very accurate. This happens as, from Figure 2, the policy improves in terms of average return. For the SEQUENCE-MAZEE task, ELLA has marginal gains in performance compared to baselines (Figure 2). This is likely due to ELLA incentivizing general subtask completion. evidenced by the growing average length of the decompositions. However, the large number of diverse language instructions in SEQUENCE- MAZE makes it difficult to estimate good decompositions in
via our deduplication method (as is uncommon to see the same instruction multiple times). difficult to learn f online and offer the most targeted low-level bonuses in this casc.

---

7 Discussion
We introduce ELLA,, reward shaping approach for guiding exploration based on the principle of abstraction in language. Our approach incentivizes low-level behaviors without assuming a strict hierarchy between high- and low-level language, and learns to identify relevant abstractions online.
As noted in Section 2, several other methods exist for addressing sample inefficiency in sparse reward settings. While we focus on using language the basis for reward shaping, intrinsic rewards can also be based on the dynamics of the environment. For example, RIDE [34] provides rewards for actions that produce significant changes to the state. In Appendix D, we discuss how ELLA can synergize with such methods to reward different aspects of exploration.
Limitations and Future Work. Our framework requires termination states for each low-level instruction. This assumption is similar to that made by Jiang et [23] and Waytowich al. [45]. and is weaker than the assumption made by the LEARN baseline of Goyal et al. [17], where full demonstrations are required. Even so, training accurate termination classifiers can require many examples-in this work, we used 15K positive and negative pairs for each low-level task. In Appendix E, we provide an ablation on the data budget for the termination classifier. Our approach is less feasible in domains where collecting examples of low-level terminating conditions is costly.
Highly compositional and diverse instructions, as in the SEQUENCE-MAZE environment, remain a challenge. Data augmentation (e.g., GECA [2]) on the online relevance dataset could potentially improve performance on these large instruction sets. Additional boosts performance and gener- alization could emerge from instilling language priors via offline pre-training. More sophisticated relevance classifiers may also prove helpful: instead of deduplicating and training on D, we could maintain a belief distribution over whether high- and low-level instructions are relevant conditioned on trajectories observed online. Furthermore, the relevance classifier could be extended to model the temporal order of the low-level tasks.
Our approach shows strong performance in several sparse reward task settings with synthetic language. Extensions to natural language could model broad spectrum of abstraction beyond two tiers.
Acknowledgments and Disclosure of Funding
This project supported by NSF Award 1941722 and 2006388, the Future of Life Institute, the Office of Naval Research, and the Air Force Office of Scientific Research. Siddharth Karamcheti is additionally supported by the Open Philanthropy AI Fellowship.
Furthermore, we would like to thank Dilip Arumugam, Suneel Belkhale, Erdem Bıyık, Ian Huang, Minae Kwon, Ben Newman, Andrea Zanette, and our anonymous reviewers for feedback on earlier versions of this paper.
References
[1] P. Anderson, Q. Wu, D. Teney, Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. van den Hengel. Vision-and- laiin-and-lannuage navigation: Interpreting visually-grounded navigation instructions in real environments. In Computer Vision and Pattern Recognition (CVPR), 2018.
[2] Andreas. enouhh compositional data augmentation. In Association for Computational Linguistics (ACL), 2020.
[3] J. Andreas, D. Klein, and Levine. Modular multitask reinforcement learning with policy sketches. In International Conference on Machine Learning (ICML), 2017.
[4] M. Andrychowicz, F. Wolski, A. Ray. J. Schneider, R. Fong, P. Welinder, B. McGrew. J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. arXiv preprint arXiv: 2017.
[5] D. Arumugam, S. Karamcheti, N. Gopalan, L. L. S. Wong, and S. Tellex. Accurately and efficiently interpreting human-robot instructions of varying granularities. In Robotics: Science and Systems (RSS), 2017.
[6] D. Bahdanau, E. Hill, J. Leike, E. Hughes, S. A. Hosseini, P. Kohli, and E. Grefenstette. Learning to under- stand goal specifications by modelling reward. In International Conference on Learning Representations (ICLR), 2019.
10

---

[7] P. Behboudian, Y. Salsangi. M. E. Taylor. A. Harutyunyan, and M. Bowling. Useful policy invariant shaping from arbitrary advice. arXiv preprint arXiv:2011.0129) 2020.
[8] Y. Bisk, A. Holtzman, J. Thomason, J. Andreas, Y. Bengio, J. Chai, M. Lapata, A. Lazaridou, J. May.
A. Nisnevich, N. Pinto, and J. Turian. Experience grounds language. In Empirical Methods in Natural Language Processing (EMNLP), 2020.
[9] Y. Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros. Large-scale study of curiosity- driven learning. In International Conference on Learning Representations (ICLR), 2019.
[10] A. Campero, R. Raileanu, H. Küttler, J. B. Tenenbaum. T. Rocktäschel. and E. Grefenstette. Learning with AMIGo: Adversarially motivated intrinsic goals. arXiv preprint arXiv:2006. 12122, 2020.
[11] M. Chevalier-Boisvert, D. Bahdanau, S. Lahlou, L. Willems, C. Saharia, T. H. Nguyen, and Y. Bengio. BabyAl: A platform to study the sample efficiency of grounded language learning. In International Conference on Learning Representations (ICLR), 2019.
[12] G. Cideron, M. Seurin, F. Strub, and O. Pietquin. Self-educated language agent with hindsight experience replay for instruction following. In Advances in Neural Information Processing Systems (NeurlPS), 2019.
[13] K. Cobbe, C. Hesse, Hilton, and Schulman. Leveraging procedural generation to benchmark reinforce- ment learning. In International Conference on Machine Learning (ICML), 2020.
[14] C. Colas, T. Karch, N. Lair, J.-M. Dussoux, C. Moulin-Frier, P. F. Dominey, and P.-Y. Oudeyer. Language as a cognitive tool to imagine goals in curiosity driven exploration. In Advances in Neural Information Processing Systems (NeurlPS), 2020.
[15] A. Das, G. Gkioxari, S. Lec, D. Parikh, and D. Batra, Neural modular control for embodicd question answering, In Conference on Robot Learning (CORL), 2018.
[16] J. Fu, A. Korattikara, S. Levine, and S. Guadarrama. From language to goals: Inverse reinforcement learning for vision-based instruction following. In International Conference Learning Representations (ICLR), 2019.
[17] P. Goyal, S. Niekum, and R. J. Mooney. Using natural language for reward shaping in reinforcement learning. In International Joint Conference on Artificial Intelligence (IJCAI), 2019.
[18] M. Grzes. Reward shaping in episodic reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2017.
[19] M. Hausknecht and P. Stone. Deep recurrent Q-learning for partially observable MDPs. arXiv preprint 1rXiv.1507.06527, 2015.
[20] K. M. Hermann, F. Hill, S. Green, F. Wang, R. Faulkner, H. Soyer, D. Szepesvari, W. Czarnecki, M. Jader- berg, D. Teplyashin, M. Wainwright, C. Apps, D. Hassabis, and P. Blunsom. Grounded language learning in simulated 3D world. arXiv preprint
2017.
[21] F. Hill, S. Mokra, N. Wong, and T. Harlcy. Human instraction-folloiinng with deep reinforcement learning via transfer-lcaming from text. arXiv preprint arXiv:2005.09:: 2020.
[22] R. Jia and Liang. Data recombination for neural semantic parsing. In Association for Computational Linguistics (ACL., 2016.
[23] Y. Jiang, S. S. Gu, K. P. Murphy, and C. Finn. Language as an abstraction for hicrarchical deep reinforce- ment learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[24] S. Karamcheti, E. C. Williams, Arumugam, M. Rhee, N. Gopalan, L. Wong, and Tellex. tale of two draggns: A hybrid approach for interpreting action-oriented and goal-oriented instructions. In First Workshop on Language Grounding for Robotics @ ACI, 2017.
[25] S. Karamcheti, D. Sadigh, and P. Liang. Learning adaptive language interfaces through decomposition. In EMNLP Workshop for Interactive and Executable Semantic Parsing (IntEx-SemPanl 2020.
[26] T. Kollar, S. Tellex, D. Roy, and N. Roy. Toward understanding natural language directions. In Human- Robot Interaction, pages 259-266, 2010.
[27] J. Luketina, N. Nardelli, G. Farquhar, J. Foerster. J. Andreas, E. Grefenstette, S. Whiteson, and T. Rock- täschel. A survey of reinforeement learning informed by natural language. In International Joint Conference Artificial Intelligence (ICCAI), 2019.
11

---

[28] C. Lynch and P. Sermanet. Grounding language in play. arXiv preprint arXiv:2005.07645 2020.
[29] A. McGovern and A. G. Barto. Automatic discovery of subgoals in reinforcement leanmin using diverse density. In International Conference on Machine Learning (ICMI.), 2001.
[30] D. K. Misra, J. Langford, and Y. Artzi. Mapping instructions and visual observations to actions with reinforcement learning. In Empirical Methods in Natural Language Processing (EMNLP), 2017.
[31] A. Y. Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference Machine Learning (ICMI.), volume 99, pages 278-287, 1999.
[32] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self-supervised prediction. In Computer Vision and Pattern Recognition (CVPR), pages 16-17, 2017.
[33] E. Perez, F. Strub, H. D. Vries, V. Dumoulin, and A. C. Courville. FiLM: Visual reasoning with general conditioning layer. In Association for the Advancement of Artificial Intelligence (AAAT), 2018.
[34] R. Raileanu and T. Rocktäschel. RIDE: Rewarding impact-driven exploration for procedurally-geeeraaed environments. In International Conference on Learning Representations (ICLR), 2020.
[35] J. Randløv and P. Alstrom. Learning drive a bicyele using reinforcement learning and shaping. In International Conference on Machine Learning (ICML), 1998.
[36] J. Schmidhuber. Adaptive confidence and adaptive curiosity. Technical report, Institut fur Informatik, Technische Universitat Munchen, Arcisstr. 21, 800 Munchen 2, 1991.
[37] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint 10.X:::777.0644444 2017.
[38] M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox. Alfred:
A benehmark for interpreting grounded instructions for everyday tasks. Computer Vision and Pattern Recognition (CVPR), 2020.
[39] M. Stolle and D. Precup. Learning options in reinforcement leamiing. In Proceedings of the 5th International Symposium on Abstraction, Reformulation and Approximation, 2002.
[40] R. Sutton, D. Precup, and Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction reinforcement learning. Articial intelligence, 1 112:181-211, 1999.
[41] S. Tellex, T. Kollar, Dickerson, M. R. Walter, A. G. Banerjec, Teller, and N. Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Association for the Advancement of Artificial Intelligence (AAAI), 2011.
[42] Thomason, S. Zhang, R. J. Mooney, and P. Stone. Learning to interpret natural language commands through human-robot dialog. In International Joint Conference on Artificial Intelligence (IJCAI), 2015.
[43] S. I. Wang. S. Ginn, P. Liang, and C. D. Manning. Naturalizing programming language via interactive learning. In Association for Computational Linguistics (ACL), 2017.
[44] X. E. Wang, Q Huang, A. Celikyilmaz, J. Gao, D. Shen, Y.-F Wang, W. Y. Wang, and L. Zhang. Reinforced modal matching and self-supervised imitation lcarning for vision-language navigation. In Computer Vision and Pattern Recognition (CVPR), 2019.
[45] N. Waytowich, S. L.. Barton, V. Lawhern, and G. Warnell, A narration- reward shaping approach using grounded natural language commands. In International Conference on Machine Learning (ICMI.), 2019.
[46] H. Yu, H. Zhang, and W. Xu. A deep compositional framework for human- human-like language acquisition in virtual environment. arXiv preprint
2017.
12


Reviews:
[{"review_id": "xOOUmwHwQL", "reviewer": "Reviewer_3GCC", "summary": "The paper present ELLA, a three-part solution to instruction following tasks with a high-level instruction and a pre-defined set of low-level instructions:\n\n1. a termination classifier is trained offline to identify the completion of a low-level goal at a state;\n2. a relevance classifier is learned online to predict a binary relevance of a low-level goal to a high-level goal.\n3. reward shaping is used to encourage low-level goals suggested by the relevance classifier.\n\nELLA is shown to improve sample complexity on 5/6 BabyAI tasks of diverse environment and instruction complexities, compared to an ablation PPO model with reward shaping, and a baseline model LEARN.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n- The method is well-motivated for the considered task setup of three levels: high-level instruction, low-level instructions, and low-level actions. Such a setup is natural for many real-world language-guided tasks, since language is hierarchical. Directly mapping the compositional high-level instruction to low-level actions is sample-inefficient, and this is a fundamental problem in instruction following. To make use of a pre-defined set of low-level instructions, a method must\nA. Link low-level instructions to low-level actions. This is done by the termination classifier providing pseudo-rewards, and reward shaping to affect the low-level action policy. \nB. Link high-level instruction to low-level instructions. This is done by the relevance classifier learned online.\n \n- It is infeasible to learn both A and B online, so ELLA proposes to learn the termination classifier offline and the relevance classifier online, with the assumption that the former supervision is easier to collect. The consequent empirical results look reasonable. \n\n- The paper is well written with good figures. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Cons/questions:\n\n- The baselines might be under-explained and a bit weak. According to my understanding, the PPO baseline makes use of neither high-level or low-level instructions, and the LEARN baseline just uses the high-level instruction? More detailed setups are needed. And it might be a less surprising comparison if ELLA is the only model that leverages the low-level instruction set. See below for some suggested baselines/model variants.\n\n- A major model design is to learn termination classifier offline and relevance classifier online. Is it possible to show the opposite, e.g. learn the termination classifier online, while learning the relevance classifier offline (possible since BabyAI is synthetic)?\n\n- Another potential concern is, learning relevance classifier totally offline abandons the fact that instructions are languages with compositional structures. For example, a high-level goal “goto box and pick it up” can be easily decomposed to “goto box” and “pickup box”, even without the game context. Currently, for each game, a new relevance classifier is learned from scratch about such decompositions, and no language priors is used. I wonder if this classifier can be somehow pre-trained as well and boost performance. \n\n- Another model design is reward shaping. Is it possible not to use it, but directly learn a relevance classifier plus a low-level goal-conditioned policy? Some experiments are needed to justify reward shaping compared to such an approach without it. \n\n- I don’t totally get Section 6.2. “However, the large number of diverse language instructions makes it difficult to estimate good decompositions via our deduplication method, and so it is difficult to learn f ρ online and offer the most targeted low-level task bonuses.” So for the PutNext task, why sample efficiency is still improved even if the relevance classifier does not learns good decompositions? \n\n- The relevance classifier seems to have no explicit temporal order of decompositions. Is this a concern for more practical tasks, or is it solid because only one relevant low-level goal can be achieved any time? Some discussion is needed.\n\n\n**After rebuttal**\n\nThanks to authors for clarifying my questions. The baselines are stronger than I thought, and it is reasonable to adopt some of my advice as future work. I raise the score to 7 and hope the paper can add/clarify discussion with respect to my questions. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "kAtWO_gScd", "reviewer": "Reviewer_HPzX", "summary": "This paper suggests ELLA (Exploration through Learned Language Abstraction), a reward shaping approach for language instruction folllowing tasks. The contributions that the authors argue in this paper can be summarized as follows:\n1. propose a reward shaping approach for instruction following, which consist of two main module, (1) termination classifier and (2) relevance  classifier\n2. empirically show that the proposed method is effective by experiment on BabyAI environment, which is a representative benchmark for instruction following tasks.", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents an exploration strategy of reinforcement learning agents to solve instruction following tasks. Since instruction following tasks are usually specified on a sparse reward setting, reward shaping can be effective for reinforcement learning. I think the most strong point of ELLA is a policy invariance property, which is usually not guaranteed on recent intrinsic exploration approaches. \n\nHowever, I have following concerns and questions for this papers:\n\n1. Although this paper consider on an exploration strategy for language guided tasks, I think considerations of this approach seems task-specific rather than language-specific. The authors especially only consider hierarchical tasks where a high-level instruction (goal) is able to be decomposed into low-level instructions (sub-goals). \n\n2. In Table 1 in Appendix A, what are criteria of assigning low-level tasks for each high-level task? \n\n3. Can you clarify how is low-level instructions data (used in experiments) collected? Whether low-level instruction data is collect by human  demonstration or demonstrations generated by human-crafted oracle, I understand each low-level instruction dataset should be collected individually for each low-level task. I think this requirement is also too expensive for general applications. \n \n4. Figure 4 shows ELLA is sensitive with a hyperparameter selection in that inappropriate selected $\\lambda$ can make learning failure. This also can be one of limitations of this paper.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors adequately have addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "NqkIyeMNOhI", "reviewer": "Reviewer_Yaq9", "summary": "This paper presents a reward shaping approach that provides immediate rewards through the achievement of low-level sub-tasks. The authors assume a problem where a high-level task given in the natural language in a sparse reward environment consists of several low-level sub-tasks, and aim to guide the agent to effectively explore using task structure. Two components (termination classifier and relevance classifier) are proposed, and reward shaping is performed using them. Finally, the authors demonstrated the results on BabyAI, a grid world platform for instruction following tasks. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I agree that this is an interesting and important problem. However, I think the assumptions in the paper are quite strong, and I am concerned about whether the experiments shown in the paper demonstrate the effect of language abstraction well. Comparison with additional baselines should be included, and it seems that the experiment should be modified to better reveal the effect of language abstraction. The detailed comments and questions are as follows:\n\n1. The baseline algorithms seem to be poor with PPO and LEARN. ELLA aims to effectively perform the exploration using language abstraction, then I think it should show advantages and improvements compared to existing exploration methods that do not use the information of language abstraction (ex. uncertainty-based exploration [1], curiosity-based exploration [2], or impact-driven exploration [3]). Some tasks were compared with RIDE in the appendix, but I think that the results on other tasks should also be mainly compared and addressed in the main text.\n\n2. ELLA uses more information than RIDE (instruction set of subtask and human-labeling of termination states), but despite this, RIDE and ELLA show similar results in terms of sample efficiency and performance. The current experimental results do not reveal the advantages of ELLA compared to the existing approaches that do not use language abstraction information. It is thought that the generalization effect of language abstraction can be shown more meaningfully by conducting experiments on unseen tasks such as meta-learning or transfer learning settings. \n\n3. As a weakness of this paper, I am concerned with the following two points: 1) The assumption for termination states of instructions are quite strong. In the general case, it is very expensive to label a large number of data manually. 2) It seems that performance and sample efficiency are sensitive to $\\lambda$ parameters.\n\n4. (Page 9, lines 310-313) I don't understand how the process of calculating the $\\lambda$ is done. How is $\\lambda$ computed from step here?\n\n5. (Page 8 lines 281-285) The authors explain why ELLA does not increase sample efficiency in a COMBO environment, but I don't quite understand what it means.\n\n[1] Yuri Burda et al, Exploration by Random Network Distillation, ICLR 2019\n\n[2] Deepak Pathak et al, Curiosity-driven Exploration by Self-supervised Prediction, ICML 2017\n\n[3] Roberta Raileanu et al, RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments, ICLR 2020\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have adequately addressed it.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper describes an approach to training an agent to follow instructions for tasks composed of multiple low-level subtasks. The method uses reward shaping, whereby the agent receives supplementary rewards when sub-tasks are achieved. The shaped rewards are determined using two classifiers, a "terminal classifier" that classifies the completion of a low-level subtask, and a "relevance classifier" that assesses the relationship between low-level tasks and the success of the high-level task. The latter is learned online without the need to break high-level instructions into their lower-level components. Experimental results on the the BabyAI task and a grid world task demonstrate gains in sample efficiency relative to recent baselines.

The reviewers agree that the paper considers an important and challenging domain. The problem of understanding natural language instructions has long been the focus of AI and robotics research, and has recently received renewed attention within the broader machine learning community. The idea of exploiting the hierarchical nature of the tasks as a means of encouraging exploration through reward shaping is reasonable and well motivated. The reviewers point out that the approach relies on several strong assumptions (e.g., involving the existence of terminal states and access to low-level instructions). Two of the reviewers also raise concerns about the baselines and the amount of information that they are provided with compared to ELLA. The author response helped to resolve some of these concerns, but the authors are encouraged to ensure that they are addressed in any subsequent version of the paper.
Decision: Accept (Poster)