Improving Long-Horizon Imitation Through Language Prediction
Authors: Donald Joseph Hejna III
Pieter Abbeel
Lerrel Pinto
Venue: ICLR 2022 Submitted (2022)
Keywords: imitation learning
language
planning

Abstract: Complex, long-horizon planning and its combinatorial nature pose steep challenges for learning-based agents. Difficulties in such settings are exacerbated in low data regimes where over-fitting stifles generalization and compounding errors hurt accuracy. In this work, we explore the use of an often unused source of auxiliary supervision: language. Inspired by recent advances in transformer-based models, we train agents with an instruction prediction loss that encourages learning temporally extended representations that operate at a high level of abstraction. Concretely, we demonstrate that instruction modeling significantly improves performance in planning environments when training with a limited number of demonstrations on the BabyAI and Crafter benchmarks. In further analysis we find that instruction modeling is most important for tasks that require complex reasoning, while understandably offering smaller gains in environments that require simple plans. Our benchmarks and code will be publicly released.

Under review as a conference paper at ICLR 2022
IMPROVING LONG- IMITATION THROUGH LANGUAGE PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
Complex, long-horizon planning and its combinatorial nature pose steep chal- lenges for learning-based agents. Difficulties in such settings are exacerbated low data regimes where over-fitting stifles generalization and compounding errors hurt accuracy. this work, we explore the use of an often unused source of aux- iliary supervision: language. Inspired by recent advances in transformer baaoomeer-baeddd models, we train agents with an instruction prediction loss that encourages learn- ing temporally extended representations that operate at high level of abstrac- tion. Concretely, we demonstrate that instruction modeling significantly improves performance in planning environments when training with a limited number of demonstrations on the BabyAl and Crafter benchmarks. In further analysis we find that instruction modeling most important for tasks that require complex reasoning, while understandably offering smaller gains in environments that re- quire simple plans. Our benchmarks and code will publicly released !
INTRODUCTION
Intelligent agents ought to be able to complete complex, long horizon tasks and generalize to new scenarios. Unfortunately, policies learned by modern deep-learning techniques often struggle to quire either of these abilities. This is particularly true in planning regimes where multiple, complex, steps must be completed correctly in sequence complete task. Realistic constraints, such as partial observability, the underspecification of goals, or the sparse reward nature of many planning problems make learning even harder. Reinforcement learning approaches often struggle effec- tively learn policies and require billions of environment interactions to produce effective solutions (Wijmans et al., 2019; Parisotto al., 2020). Imitation learning an alternative approach based on learning from expert data, but can still require millions of demonstrations to learn effective planners (Chevalier-Boise) Boisvert et al., 2019). Such high data constraints make learning difficult and expensive.
Unfortunately the aforementioned issues with behavior learning are only exacerbated in the low data regime. First, with limited training data agents are less likely to act perfectly at each environment step, leading to small errors that compound overtime in the offline setting. Ultimately, this leads to sub-par performance over long horizons that can usually only be improved by carefully collecting additional expert data (Ross et al., 2011). Second, deep-learning based policies are more likely to overfit small training datasets, making them unable to generalize to new test-time scenarios. On the other hand, humans have the remarkable ability interpolate previous knowledge and solve unseen long-horizon tasks. After observing an environment, we might deduce plan or sequence of the steps follow to complete our objective. However, imitation learning agents are not required construct plans by default they are trained to only output the direct next action given seen observations. This begs the question: how can we make agents reason better in long-horizon tasks?
An attractive solution lies in language instructions, the same medium humans use for mental plan- ning (Gleitman & Papafragou, 2005). Several prior works directly provide agents with language instructions to follow (Anderson et al., 2018; Shridhar et al., 2020; Chen et al., 2019). Unfortu- nately, such approaches require the specification of exhaustive instructions at test time for systems to function. A truly intelligent agent ought to be able to devise its own plan and execute it, with only a handful of demonstrations. We propose improving policy learning in the low-data regime
"Code available <github will be made public after reviewing period>

---

Under review as a conference paper at ICLR 2022
Goal: Move the purple green box
Goal: Make stone pickaxe
"Chop down the get wood"
Instructions:
"Move the green box" "Open the blue "Pickup the purple box"
"Drop the box"
Figure 1: The left half of the figure shows key frames and their associated instructions for task in the BabyAl environment. The right half depicts single state from the Crafting environment and its associated goal and instruction.
by having agents predict high-level planning instructions in addition to their immediate next ac- tion. As do not input instructions the policy, time. Though prior works have used hierarchical structures that generate their own instructions condition on (Chen al., 2021c; Hu et al., 2019; Jiang al., 2019), we surprisingly find that just predicting language instructions is in itself a powerful objective to learn good representations for planning. Teaching agents to output language instructions for completing tasks has two con- crete benefits. First, it forces them to learn at higher level of abstraction where generalization is easier. Second, by outputting multi-step instructions agents explicitly consider the future. Practi- cally, we teach agents to output instructions by adding an auxiliary instruction prediction network to transformer-base policy networks, as in seq2seq translation (Vaswani et al., 2017). Our approach can be interpreted translating observations trajectories into instructions.
plan without their specification at test
We test our representation learning method limited data settings and combinatorially complex en- viornments. In the BabyAl benchmark, we attain near the same performance as Chevalier-Boiee et al. (2019) with only 5% of the data. To our knowledge, we achieve the highest success rate in the Crafting environment from (Chen et al., 2021c), while using fewer demonstrations and no reinforcement learning. We find that in many settings higher performance can be attained by rela- beling existing demonstrations with language instructions instead of collecting a new, scalable type of data collection for practitioners. Furthermore, our method is conceptually simple and easy to implement. This work the first to show that direct representation learning with language can accelerate imitation learning.
ones, creating
To summarize, our contributions are as follows. First, introduce method for training trans- former based planning networks on paired demonstration and instruction data via an auxiliary lan- guage prediction loss. Second, we test our objective in long-horizon planning based environments with limited data and find that it substantially outperforms contemporary approaches. Finally, we analyze the scenarios in which predicting instructions provides fruitful training signal, concluding that instruction modeling is valuable objective when tasks are sufficiently complex.
RELATED WORK
Language in the context of policy learning has been heavily studied (Luketina et al., 2019), usually communicate a task objective. Uniquely, we use natural language instructions aid in learning via an auxiliary objective. Here we survey the most relevant works to our approach.
Language Goals. Language offers natural medium communicate goals intelligent agents. As such, several prior work have focused on learning language goal conditioned policies, particularly for roboties (Nair al., 2021; Stepputtis al., 2020; Kanu et al., 2020: Hill al., 2020; Akakzia et al., 2021; Goyal et al., 2021: Shridhar et al., 202 or for games
Sidevalier-Boiveed et al., 2019; Chaplot et al., 2018; Hermann et al., 2017). Others in the area of inverse reinforcement learning use language to specify reward functions (Fu et al., 2019; Bahdanau et al., 2018; Williams et al., 2018) or shape them (Mirchandani et al., 2021; Goyal et al., 2019). Unlike these works, we use language instructions that dictate how agent should complete task instead of language goals that specify what the task itself is. Other works, particularly in the visual navigation space, provide agents with instructions similar to those et al. (2018); Fried et al. (2018); Chen et al. (2019); Krantz et al. (2020); Chen et al. (2021a: 2019) instructions for visual navigation or drones (Blukis al., 2019), while Shridhar al. (2020);
use, sometimes in addition to language goals. Anderson
2

---

Under review as a conference paper at ICLR 2022
Pashevich et al. (2021); Shridhar et al. (2021b) use instructions for household tasks. Critically unlike our method, these approaches use both language goals and instructions as input, and consequently require the creation of instructions at test-time for every desired task. Other proposed environments (Zhong et al., 2019; Wang & Narasimhan, 2021) assess understanding by prompting agents with necessary information about task dynamics, precluding the removal of text-prompting at test time.
Language and Hierarchical Learning. Instead of directly using instructions as policy inputs, other works use language instructions an intermediary representation for hierarchical policies. Usually, a high-level planner outputs language instructions for a low-level executor to follow. Andreas et al. (2017) and Oh et al. (2017) provide agents with fixed, hand sand-designed high-level language instruc- tions or policy "sketches". Such approaches require new instruction labels at test-time for every new task unlike our method. Jiang et al. (2019) and Shu et al. (2017) provide interactive language labels to agents to train hierarchical policies with reinforcement learning. In the imitation learn- ing setting, Hu et al. (2019) learn hierarchical policy using behavior cloning for a strategy game. Unlike the planning problems we consider, their environment has no oracle solution and does not consider generalization to unseen tasks. Most related to our work, Chen et al. (2021c) use latent representations from learned high-level instruction predictor aid low-level policy. However, unlike Chen et al. (2021c), we learn latent representations that can predict instructions, but do not explicitly condition on them at test-time. While hierarchical approaches have shown great promise, the quality of learned policies is inherently limited by the amount of language data available for training. Even with a perfect low-level policy, inaccurate high-level languages commands will yield poor overall performance. This is not an issue for our loss-based approach, as our instruction pre- diction network can be completely detached from the policy. Additionally, this allows our method to work on a mix of instruction annotated and unannotated data, letting it more easily scale than hierarchical approaches particularly in data-limited scenarios.
Auxiliary Objectives. The learning community has extensively studied the use of auxiliary objec- tives in policy learning. Though to our knowledge no prior works use instructions as an auxiliary objective, auxiliary objectives in general have been found to aid policy learning (Jaderberg et al., 2017). Laskin et al. (2020) and Stooke et al. (2021) demonstrated the success of contrastive auxil- iary objectives in robotic reinforcement learning domains. Schwarzer et al. (2020) and Anand et al. (2019) did the same in the Atari game-playing environments. We were inspired by their effective- ness. Additionally, works like Andreas et al. (2018) have previously used language question and answering for representation leaning in visual domains.
Transformers. Our approach is based several innovations involving transformer networks. Vaswani et al. (2017) previously showed state of the art results in machine translation using trans- formers. While the application of transformers has extended to behavior learning (Zambaldi al., 2018; Parisotto et al., 2020; Chen et al., 2021b), prior works in the area have not leveraged the transformer decoder. Closest to our domain, Lin et al. (2021) generate captions from video. The architecture of our policy networks take inspiration from recent works adapting transformers to mediums beyond text, namely in vision (Kolesnikov al., 2021) and offline reinforcement learning (Chen et al., 2021b).
3 METHOD
In this section we formally describe the problem of imitation learning with instruction prediction, then describe our implementation for both Markovian and non-Markovian environments.
3.1 PROBLEM SETUP
The standard learning from demonstrations setup assumes access to dataset of expert trajectory sequences containing paired observations and actions Ø1, 01, 02, Q2,..., Or, @r. The goal of imitation learning is to learn a policy π (a,l-) that predicts the correct actions an agent should take. In our work we consider both fully observed (MDP) and partially observed (POMDP) settings. In the partially observed case, policies are given access to previous observations in order to infer state (Kaelbling et al., 1998), and we denote the policy as (aal ..)...)). In the fully-observed setting this is unnecessary, and the policy is simply π (a, (..)or). It is common for policies to be goal conditioned, or even conditioned language goals is the case in our experiments. This means they take
3

---

Under review as a conference paper at ICLR 2022
action
Instr
[1
2
C22
(1)
X
X
X
db
Transformer Decoder gy
latent
21
22
Z
X2
X1
T
T Instr for time
Ti+1
Encoder fo
Transformer Encoder fo
Language Instructions
alss, goal
Instr
g
01
02
O₂
+1
OT
Mask future observations for instructions beyond time
Goal and Observation Sequence
Figure 2: The left diagram depicts the general model architecture used for our approach. Notice how the policy and encoder can be completed separated from the instruction component for mixed-data training or inference. The diagram on the right depicts its implementation for the partial observed environments using GPT-like transformer encoder. The diagram shows our masking scheme episode step t: latent vectors from beyond time are masked from the language decoder.
an encoding of the desired task goal g input. As approach works with without goal conditioning we omit it from the rest of this section for brevity. A standard imitation learning technique is behavior cloning, which in discrete domains maximizes the likelihood of the actions in the dataset using negative log likelihood objective, Laction - Σ. log ((all)).
In this work, we assume access oracle language instructions that tell an agent how it should complete a task to provide useful training signal. As mentioned in Section 2, for the purposes of our method we distinguish goals from instructions. Language goals tell an agent what do, whereas language instructions communicate how an agent should complete task. Each trajectory may have several language instructions z²(1,, 2x(2) ,..,,,!")) (n)) corresponding to different steps in the task. For ...., example, a language instruction like "open the door" only applies to the part of the demonstration before the agent opens the door and after it completes the last instruction. The i-th instruction (e()) thus corresponds to an interval [T], Ti+1) where T; marks the time the instruction was given and T:+1 denotes the start of the next instruction. depiction of an example instruction sequence can be found in Figure 1. While language instructions are an additional data requirement, they can be cheap to obtain, particularly in scenarios where demonstrations are expensive to collect. If one has to collect demonstrations in the real world, providing instructions while they are collected likely only constitutes a marginal increase in effort. Moreover, humans can easily re-label existing demonstrations with instructions. Video data could easily be captioned with voice-over. Similar statements can be made for simulators if one can code an oracle policy, instructions are likely easy to generate along the way. These modifications stacks, as we do with BabyAI. Moreover, we focus on the data limited regime, where the cost of setting up environment and collecting more demonstrations is likely be higher than annotating an existing small set of demonstrations. Next, we describe how we train agents to predict language instructions aid imitation learning.
easily be done on simulators with planning
3.2 INSTRUCTION PREDICTION FOR IMITATION LEARNING
The central hypothesis of this work is that predicting high-level language instructions will force agents to learn representations beneficial for horizoorizon planning. In our learning framework, we first construct an observation encoder fo(o1...,, ..., 01) that produces latent representations 2, of the observation(s). Ultimately, it will be trained using both behavior cloning and language modeling losses. As in standard behavior cloning we predict actions using policy network n T (z)) which in our case is placed on top of the encoder's 's latent representation. The same latent representation is also used to predict the current high-level language instruction 20(1), where E [T., T++1). This is accomplished via a language decoder ge (z())(2) Our general setup is shown in the left half of Figure 2. As is common in natural language processing, we treat each language instruction (4) as (6) sequence of multiple text tokens 2 ..2 ..... 2i. where is the length of the i-th instruction. The decoder is trained using the standard language modeling loss. We construct our total imitation learning objective for given trajectory is follows
4

---

Under review as a conference paper at ICLR 2022
T -
Ti41 1--1-100
4
C= ---2 log To (o [e,..., - A
lo go (1) l0 -..,, " 21,...
(1)
t=
where latent representations: z all produced by the shared encoder So. The first term of the loss is the standard classification loss used for behavior cloning in discrete domains. The second term of the loss corresponds to the negative log- likellooooodd of the language instructions. We index the language loss by instructions via the first sum. The second summations ensures that we compute the likelihood of instruction i using only observations during or before its execution. The final sum over token log likelihoods is from the standard auto-regressive language modeling framework, where the likelihood of an instruction is the product of the conditional probabiliities p(x(1) 1 (xx" ..... ....([1)) Finally, À is a weighting coefficient that trades off the importance of instruction prediction and ac- tion modeling. During training, we propagate gradients from both behavior cloning and language prediction to the encoder weights 0. In some of our experiments we test additional learning objec- tives which are also trained on top of the same latent representations 2 as is standard in the literature (Jaderberg et al., 2017).
Though our method is general to any network architecture, we train transformer based policies since they have been shown to be extremely effective at natural language processing tasks (Vaswani al., 2017) and carry good inductive bias for combinatorial planning problems (Zambaldi et al., 2018). For details on the transformer architectures we use, we defer to Kolesnikov et al. (2021); Chen et al. (2021b) and Radford et al. (2018). In the following sections describe our Transformer-based models for both partially observed and fully observed settings.
Partiilly-Observed Setting. For environments that are Partially Observed Markov Decision Pro- cesses (POMDPs) or fully observed we use a transformer based sequence model as our policy network, similar those employed in Chen et al. (2021b). Thus, the model can attempt to in- fer the true state from all observations. Every observation is preprocessed then fed into a trans- former encoder fe to produce latent representations. We operate in the entire sequence at once: 21:-..2y fo., o.). Causal masking similar to that in Radford et al. (2018) ensures that at time the representation 2, only depends on current and previous observations 01, ... The same policy network To (a @el (2)) is applied to each latent to produce actions for each timestep. The language decoder By operates on the same set of latents. Our overall architecture is depicted in Figure 2. Cru- cially, the decoder employs both causal attention masks to the language inputs and cross attention masks to the latents. Causal-self attention masks on the language inputs enforce the auto-regressive modeling of the instruction tokens. Cross attention masks to the latent representations ensure that predictions for the ith instruction cannot attend to latents from timesteps after its execution as is de- picted by the red "x"s in Figure 2. This forces language prediction during training mirror test-time as the agent cannot use the future information to predict what instruction it should execute.
Fully Observed Setting, For environments that are Markov Decision Processes (MDPs) fully ob- served, we do not need to model the entire sequence of observations and instead use only the most recent observation ot- This corresponds removing all conditioning prior latents 21, 2t-1 in Equation 1. As such, we employ networks based on the Vision Transformer architecture from Kolesnikov al. (2021) that predict actions only for a single timestep. Observations are prepro- cessed into tokens and prepended with a special CLS token: O - CLS, or,1, or,2, 0t,3, .... As we do not input future observations, the transformer encoder uses full unmasked self attention. At the end of the network we take the latent representation corresponding to the CLS token and use it to predict the action (a (,,,))). We use all latent tokens to predict the current language instruction with yø- A depiction of this architecture would be largely similar to that shown in Figure 2, except the encoder would have only a single observation o, as input and the decoder would only operate on the current instruction (9)) used for t E [T,, T,41). An architecture figure for this model can be found in Appendix C.
4 EXPERIMENTS
In this section detail our experimental setup and empirical results. In particular, we investigate the benefits of instruction modeling for planning in limited data regimes. We seek to answer the following questions: How effective is instruction modeling loss? How does instruction modeling
5

---

Under review as a conference paper at ICLR 2022
Env
Observable
Language
Train Tasks
Test Tasks
Vocab Size
Steps/Instr
BabyAl
Partial
Synthetic
=# Demos
00
45
11-12
Crafting
Full
Human
14
35
226
5-6
Table I: An outline of the differences between the two environments we study.
scale with both data and instruction annotations? What architecture choices are important? And finally, when is instruction modeling a fruitful objective?
4.1 ENVIRONMENTS
We test our method on two distinct environments, BabyAI (Chevalier-Boisse) et al., 2019) and the Crafting Environment from Chen al. (2021c), to evaluate the effectiveness of instruction prediction at enabling long-horizon planning and generalization to unseen tasks. Differences between the two environments are outlined in Table 1. Across both environments, we cover challenges in partial observability and modeling human generated text. Both environments provide temporally delineated coarse high-level instructions.
BabyAI: Here, an agent must navigate a partially observable grid-world to complete arbitrarily complex goals specified through procedurally generated language such as locating objects, moving objects, opening locked doors, and more. Goals can also be specified relative to an agents' initial position. Agents are evaluated on their ability to complete unseen missions in unseen environment configurations. We modify the BabyAI demonstration collection agent output language instruc- tions based on its high level planning logic. Each high level instruction corresponds to multiple low-level actions. The Baby environment suite comes with multiple environments of varying dif- ficulty. We focus our experiments on the hardest environment, BossLevel, and use only 1.25% 5% of the standard 1 million samples used for agent training with imitation learning. Because of the en- vironment's ' partial observability and reliance on memory, we employ a transformer sequence model as described in Section 3.2 with the same convolutional network extractor from Chevalier-Bi Boissvert et al. (2019). Goals from the environment are tokenized and fed as additional inputs to the policies. We train all AabyAl models for million steps and evaluate the model that achieves the highest validation action prediction accuracy on five hundred unseen tasks for two seeds.
Crafting:
This environment from Chen et al. (2021c) tests how well an agent can generalize to new tasks using instructions collected from humans. The original dataset contains around 5.5k trajectories with human instruction labels of which we use 20%-60%. Each task is specified by a specific item the agent should craft, encoded via language. The agent must complete number of independent steps, collecting and combining resources, to obtain the final item. The tasks vary in difficulty from one-step five-steps. As this environment is fully observed, we employ the Vision Transformer based model described in Section 3.2. Like in Chen et al. (2021c), we use GloVe embeddings (Pennington et al., 2014) to preprocess both the language goals and instructions, and tokenize the grid as input to our model. For more details on the dataset, we refer the reader to Chen al. (2021c). We train all Crafting Models for three-hundred thousand steps for four seeds.
For both environments and models the encoder is comprised of four transformer blocks of dimension 128 with two attention heads and the decoder has one transformer block of the same dimension. As both environments use language based goals, we tokenize the goal text, apply embeddings. and append to the beginning of the observation tokens. Attention then acts jointly over both modalities. More experiments are detailed in Appendix A.
4.2 BASELINES
We compare the effectiveness of our instruction modeling auxiliary loss to number of baselines. The text in parenthesis indicates how we refer to the method in Tables 4, 5, 6, and 7.
1. Original Architecture (Orig): The original state of the art model architectures proposed for each environment in their respective papers. The crafting environment uses language- instruction hierarchy. In BabyAl, we use convolutions and FiLM layers as in Chevalier- Boisvert et al. (2019).

---

Under review as a conference paper at ICLR 2022
Goal:: "apen purple put purple grey
Oracle
purple
"go the
"apen the grey doos
"go the grey door
"pickup purple box"
"drap purple
Ours
"opan purple door"
door
"open purple
"icckup purple grey door"
"drop the purple box"
Figure 3: Snapshots of a rollouts from an oracle agent and our trained agents on the same unseen task in BabyAI. Our agent is able to predict instructions, given below each image, with high fi- delity. Additionally, we see that our learned agent employs a different exploration strategy, but still completes the task exhibiting strong generalization.
2. Transformer (Xformer): Our transformer based models without any auxiliary objectives to determine the effectiveness our architectures.
3. Transformer Hierarchy (Hierarchy): A high-level transformer model is trained to out- put the current instruction that the agent should execute. A low-level executor transformer model is trained to output the actions conditioned the high-level's language instruction. Prior work has not used hierarchy on observation sequences from partially observed en- vironments like those in BabyAL, and thus we devise our own method. This baseline is designed to compare our auxiliary method to methods in Chen et al. (2021c) and Hu et al. (2019).
4. Transformer with ATC (ATC): Our transformer model with the active temporal contrast (ATC) supervisd objective proposed in Stooke al. (2021), which found to per- form better than Laskin et al. (2020) in our environments. This compares representation learning with language to vision based representation learning.
5. Transformer with Lang (Lang): Our transformer based models with the instruction mod- elling auxiliary objective only.
6.
Transformer with ATC and Lang (Lang + ATC): Our transformer based models with both instruction modeling and constrastive auxiliary losses.
4.3 How EFFECTIVE IS INSTRUCTION PREDICTION?
Our main experimental results can be found in Table where compare the performance of all methods on both environments with three differing dataset sizes. We find that for environments and dataset sizes instruction modeling objective improves has effect in the worst In Ba we achieve a 70% success rate on the hardest level with fifty thousand demonstrations and instructions. For comparison, it is worth noting that the original BabyAl implementation (Chevalier- Boisvert et al., 2019) achieved a success rate of 77% with one million demonstrations on a single seed. In the crafting environment, using instruction modeling boosts the success by about 5% or more in the 1.1k and 2.2k demonstration setting. To our knowledge our results are state of art in this environment, exceeding the reported 69% success rate on unseen tasks in Chen al. (2021c).
Visual representation learning was not as fruitful as language based representation learning over- all. The combination of ATC and instruction modeling was unfortunately not constructive in all scenarios: it performed better in some instances and worse than just language loss in others. This consistent with results found in Chen et al. (2021d) that show that observation based auxiliary objectives often yield mixed results in the imitation learning setting, We find that our hierarchical implementations do not perform very well in comparison to plain transformer models. This is likely because with only few demonstrations high level language policies are likely output incorrect instructions for unseen tasks leading low-level instruction conditioned policies to output sub-optimal actions. More analysis of the hierarchical baselines is in Appendix B.

---

Under review as a conference paper at ICLR 2022
Env
Demos
Orig
Xformer
Hierarchy
ATC
Lang
ATC+Lang
50k
35.31:0.1
40.2
36.813.55
45.82.6
70.3:11.3
64.3:00.5
Baby Al
25k
32.3:12.4
39.9:10.5
37.2:3.0
37.1
55.4:17.0
56.0:13.0
BessLevel
12.5k
29.9
37.3+0.1
36.442.6
38.4+1.4
39,4+1.0
38.6=0.6
3.3k
9.3440.4
7..54:33
59.9411
75.7+:1.0
74.5412.8
76.0-12.8
Crafting
2.2k
4.9+1.0
69.4
56.51999
73.942.1
75.2:4.4
78.2
1.1k
1.740.8
70.1
39.413.8
70.1
74.8#2.6
71.4#2.9
Table 2: Success rates (in %) of all methods for varying amounts of demonstration data in both environments. The best msthod(s) is bolded.
Model
Base
Lang
ATC
BabyAI
Crafting
Boisvert et al.
1052297
70
No Lang
78
GPT-Like
661.385
253056
98816
Success Rater (%)
Lang
76
Chen al. ViT-like
1379980
218083
60
74
579177
259712
98816
50
Table 3: Model Parameter counts by component. The hierarchical trans- former baselines roughly double the rameter counts of their respective mod- els.
40
20000 30000 40000 50000
1500 2000 2500 3000 Demonstrations
Demonstrations
Figure 4: Data scaling with and without instrucutions.
4.4 HOW DOES INSTRUCTION PREDICTION SCALE WITH MORE DATA AND ANNOTATIONS?
Overall, we find that instruction modeling scales well in the low to medium data regime. With too little data, policies are unlikely to learn good high-level representations that generalize even with the language objective. With a significant amount of data instruction modeling may become unnec- essary. Figure depicts how model performance changes with dataset size. In BabyAl, instruction modeling does not appear significantly help with the smallest number of demonstrations, likely because both the policy and language decoder overfit quickly. However, after that we find that pol- icy performance with language scales almost linearly while policies without language are unable to perform substantially better even with more data. This is not just because training with instruc- tions helps overcome partial observability we show similar results on fully observed version of Ba in Appendix A. The Crafting environment has only fourteen training tasks in comparison BabyAl' Al's potentially infinite number causing to require fewer demonstrations to solve. Thus, we observe the opposite problem: instruction modeling helps when the policy is data constrained, and then neutral when more data introduced.
A benefit of our loss-based approach is that it can easily be applied to mixed datasets that have only some instruction labels. To additionally study the scaling properties our language prediction ob- jective, construct datasets in BabyAl where only half of the trajectories have paired instructions. Results can be found in Table 4. Surprisingly, we find that using half the number of instructions demonstrations yields significantly more than half of the performance gains from language in the 25k demonstration setting and nearly all the gains in the 50k demonstration setting. As seen in both Tables 4 and one better off collecting 12.5k language annotations than collecting an additional 25k demonstrations in the Baby Al environments. A similar statement can be made in the crafting environment for 1.lk demonstrations. This means that collecting language annotations is real feasible alternative to collecting more demonstrations.
% Instr Labels
0%
50%
100%
Model
Success 1.3
50k Demos
40.2+2.2
6..6
7..
Lang
25k Demos
39.9+0.5
1..
55.4+7.0
Lang No Mask
10.1
Table 4: We ablate the number of language instruction annota- tions used in training. 0% means instructions, 50% means half of the demos have instructions, and 100% means all have instructions. Values % success rates.
Table 5: Comparison of BabyAI Model on Boss Level with and without masking.
8

---

Steps Under review as a conference paper at ICLR 2022
50k Demos
25k Demos
12.5k Demos
Level
Xformer
Lang
Xformer
Lang
Xformer
Lang
GoTo (Easy)
1..
91.0=11.4
77.6+2.2
82.3:13.3
68.4=2.2 68.9
SynthLoe (Med)
72.5 40.2+222
8..2
60.15:0.5
57.5 1..4
1.. 1.0
BossLevel (Hard)
70.3+
39.94
55.4417.0
37.3++0.1
Table 6: Performance, in percent, of instruction prediction when varying the BabyAl level difficulty.
Demonstrations
Model
Steps
Steps
Xformer
98.1= 4
66.916.2%
22.1 %
3.35k
Lang 9..1%3
73.011..2%
19.3:14,4%
Lang+ATC
97.1+2.2%%
75.1+
13.2++1.7%
Xformer
89.318.6%
58.411..2%
2..5
2.2k
Lang
96.1=0.7%
13.5%
17.3+1.7%%
Lang+ATC
93.9=2.7%
78.9%
19.8+227%%
Xformer
90.9=4.0%%
58.248.55
15.0=1.11%
1.1k
Lang
94.5+1.4%
76.0188.4%
13.8+660%
Lang+ATC
89.5+t3.2%
10.2%%
11.6++2.%
Table 7: Difficulty comparison in the crafting environment. Steps indicate the number of steps required for the agent to craft the goal item.
4.5 WHAT MODELING DECISIONS ARE IMPORTANT?
In Table 5 we ablate the of our instruction decoder cross-attention masking. We find that the omission of the masking scheme leads to a 20% drop in performance. Without masking the language decoder has easier time predicting instruction finished, creating a disparity between train and test time and leading to lower quality representations. Overall, the transformer architecture appears to be critical to high performance, likely because of its good inductive bias for reasoning about objects and their interactions. This especially evident in the Crafting environment. As stated in Chen al. (2021c), the imitation learning approaches with the original model were unable to achieve meaningful success rate on any of the unseen tasks, whereas our baseline transformer achieves a success rate of around 70%. This is better than the reported 69% success rate in Chen et al. (2021c) using all 5.5k trajectories from the dataset and additionally applying reinforcement learning. Our architecture choice is also extremely parameter efficient. As seen in Table 3, our models use significantly fewer parameters than baseline models, even accounting for the additional language decoder.
instruction
attend to observations from after the
4.6 WHEN IS INSTRUCTION PREDICTION USEFUL?
We hypothesize that instruction prediction is particularly useful for combinatorially complex, long horizon tasks. Many simple tasks, like "open the door" or "grab a cup and put it the coffee maker" communicate all required steps and consequently stand to gain little from instruction modeling. Conversely, tasks in both environments we study do not communicate all required steps to agents. In the AabyAl environment, exploration is needed to locate objects and implicit tasks arise. For example, agents may need to collect keys to open locked doors. In the crafting environment, goals single items that can only be attained through sequence of steps. Thus, task horizon and difficulty increase one would expect instruction modeling to be more important. In BabyAI we consider two additional levels GoTo, which only requires object localization, and SynthLoc which uses subset of the BossLevel goals. Results in Table 6 indicate that instruction modeling is indeed most important for harder tasks and yields modest gains in the easiest GoTo level. The same trend is true in the Crafting environment as seen in Table 7. All policies are able to craft items requiring two steps tasks with a success rate near or above 90%. However, models with language objectives boast significant performance boost in the 3-step tasks, from around 58% to closer to 75% in most cases. No agents were able to consistently complete five-step tasks, which is understandable given only one is included in the set of training tasks. The takeaway from these observations is that instructions offer less training signal for combinatorially simple tasks, where reaching the goal requries only few, obvious logical steps. Thus, we expect instruction modeling not perform extremely well in benchmarks where instructions can easily be predicted from goals alone. We provide further analysis of this using the ALFRED benchmark (Shridhar al., 2020) in Appendix A.2. As

---

Under review as a conference paper at ICLR 2022
approach works for both procedurally generated and human annotated language, we believe most of the training signal in instructions comes from information about how to solve the task and not the natural structure of human langauge. As task difficulty scales in the future expect instructions to become a critical modeling component.
5 CONCLUSION
We introduce auxiliary objective that predicts language instructions for imitation learning and associated transformer based architectures. We demonstrate that our language modeling objective consistently improves generalization unseen tasks with few demonstrations. Moreover, our ap- proach scales efficiently, leading us to conclude that in many settings providing language instructions for demonstrations yields greater impact than collecting larger dataset. We further analyze the domains where our method is successful, and make recommendations for when to apply it.
REFERENCES
Ahmed Akakzia, Cédric Colas, Pierre- Yves Oudeyer, Mohamed Chetouani, and Olivier Sigaud. Grounding language to autonomously-acquined skills via goal generation. In ICLR 2021, 2021.
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R Devon Hjelm. Unsupervised state representation learning in atari. In Proceedings of the 33rd Interna- tional Conference on Neural Information Processing Systems, pp. 8769-8782, 2019. 3
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-languuge navigation: Interpreting visually-groundee navigation instructions in real environments. Proceedings of the IEEE Con- ference Computer Vision and Pattern Recognition (CVPR), 2018. 1.2
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In International Conference Machine Learning, pp. 166-175. PMLR, 2017.
Jacob Andreas, Dan Klein, and Sergey Levine. Learning with latent language. NAACL-HLT, 2018. 3
Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, and Edward Grefenstette. Learning understand goal specifications by modelling reward. In Inter- national Conference on Learning Representations, 2018. 2
Valts Blukis, Yannick Terme, Eyvind Niklasson, Ross Knepper, and Yoav Artzi. Learning to map natural language instructions physical quadcopter control using simulated flight. arXiv preprint arXiv:1910.09664, 2019. 2
Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Ra- jagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018. 2
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural
language navigation and spatial reasoning in visual street environments. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12538- 2019. 1,
Kevin Chen, Junshen K Chen, Jo Chuang, Marynel Vázquez, and Silvio Savarese. Topological planning with transformers for vision-and aadion-and-lanuuaae navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11276-1 1286, 2021a.
Lili Chen, Kevin Lu. Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint 2rXiv:2106.01344 2021b. 5, 15
10

---

Under review as a conference paper at ICLR 2022
Valerie Chen, Abhinav Gupta, and Kenneth Marino. Ask your humans: Using human instructions to improve generalization in reinforcement learning. In International Conference on Learning Representations, 2021c. URL 2, 3, 6, 7, 9, 16
Xin Chen, Sam Toyer, Cody Wild, Scott Emmons, Ian Fischer, Kuang-Huei Lee, Neel Alex, Steven H Wang, Ping Luo, Stuart Russell, al. An empirical investigation of representation learning for imitation. 2021d. 7
Maxime Chevalier-Boisver,, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. AlbyAl: First steps towards grounded language learning with a human in the loop. In International Conference on Learning Representations, 2019. URL
https: http///ppeeeeeeeeeeeeeeeeeeeeeee net forum? (errumiiddrJeeeoooeeo
2, 6,
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philiipe Morency. Taylor Berg- Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker-follower models for aasion-and-language navigation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 2318-33221 2018.
Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to goals: Inverse reinforcement learning for vision-base based instruction following. arXiv preprint 190.0::902.00742, 2019. 2
Lila Gleitman and Anna Papafragou. Language and thought. Cambridge University Press, 2005. 1
Prasoon Goyal, Scott Niekum, and Raymond J Mooney. Using natural language for reward shaping in reinforcement learning. arXiv preprint 1eXiv:/903.02022 2019. 2
Prasoon Goyal, Raymond J Mooney, and Scott Niekum. Zero-shot task adaptation using natural language. arXiv preprint arXiv:2106.02977 2021.
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojeiech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded lan- guage learning in simulated 3d world. arXiv preprint urXiv:/706.0655// 2017.
Felix Hill, Sona Mokra, Nathaniel Wong, and Tim Harley. Human instruction-following with deep reinforcement learning via transfer-learning from text. arXiv preprint arXiv:2005.09382. 2020.
Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuandong Tian, and Mike Lewis. Hierarchical de- cision making by generating and following natural language instructions. Advances in Neural Information Processing Systems, 32: 12:10002-10034 2019. 3,
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In- fernational Conference on Learning Representations, 2017. 3,5
YiDing Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language abstrac- tion for hierarchical deep reinforcement learning. Advances in Neural Information Processing 2019. 2.
Systems,
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in
partially observable stochastic domains. Artificial intelligence, 101(1-2):9--14, 1998. 3
John Kanu, Eadom Dessalene, Xiaomin Lin, Cornelia Fermuller, and Yiannis Aloimonos. Following instructions by imagining and reaching visual goals. arXiv preprint wxXiv2200..09375 2020.
Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Un- terthiner, and Xiaohua Zhai, An image worth 16x16 words: Transformers for image recognition at scale. 2021. 3,5
Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav- graph: Vision-and languuuge navigation in continuous environments. In European Conference on Computer Vision, pp. 104-120. Springer, 2020.

---

Under review as a conference paper at ICLR 2022
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. In International Conference on Machine Learning, pp. 5639- 5650. PMLR. 2020. 7,
Xudong Lin, Gedas Bertasius, Jue Wang. Shih-Fu Chang, Devi Parikh, and Lorenzo Torresani. Vx2text: End-to-end learning of video-based text generation from multimodal inputs. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7005-7015, 2021. 3
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefen- stette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. arXiv preprini arXiv: 03926, 2019.
Suvir Mirchandani, Siddharth Karamcheti, and Dorsa Sadigh. Ella: Exploration through learned language abstraction. arXiv preprint arXiv:2103.05825 2021.
Suraj Nair, Eric Mitchell, Kevin Chen, Brian Ichter, Silvio Savarese, and Chelsea Finn. Learning language-conditioned robot behavior from offline data and crowd-sourced annotation. Conference on Robot Learning (CoRL), 2021. 2
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with task deep reinforcement learning. In International Conference on Machine Learning, pp- 2661-2670. PMLR, 2017.
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, al. Stabilizing transformers for reinforcement learning. In International Conference on Machine Learning, pp. 7487-7498. PMLR, 2020. 3.3
Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic Transformer for Vision-and- Language Navigation, 2021. 3, 14, 15
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543, 2014. URL http: /wwww aclweb. org/ wwt.aooooooooyoooooooooo DD14- 1162.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018. 5
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self- self-predictive representations. In International Conference Learning Representations, 2020. 3
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A Benchmark for Interpreting Grounded Instruc- tions for Everyday Tasks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. URL https:/ //tx/vavvvv////////a//.//////avv org/ abs / 1 1912. 01 01434.1110 1, 2,9
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. 2021a. 2
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté. Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021b. URI. https:/ //wxiiv..vvvv///////aa/ org/ alss/ 2000 03768. 3
Tianmin Shu, Caiming Xiong, and Richard Socher. Hierarchical and interpretable skill acquisition in multi-task reinforcement learning. arXiv preprint arXiv: 2017. 3
12

---

Under review as a conference paper at ICLR 2022
Simon Stepputtis, Joseph Campbell, Mariano Phielipp. Stefan Lee, Chitta Baral, and Heni
Ben Amor. Language-conditione imitation learning for robot manipulation tasks. Advances in Neural Information Processing Systems, 33, 2020.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International Conference on Machine Learning, pp. 9870-9879. PMLR, 2021. 3,7
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In L Guyon, U. V. Luxburg, S. Bengio,
Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https::/prroceedngg.. htwp://eeeedinnn..eneeeeeeeeeeee http://w.e......pppaaaeeeeeaeeaa 3 3f5ee243547dee Fbd053c11c4a855ee Paper pdf. 3,5
cc/pappr/2217/E110
HJ Wang and Karthik Narasimhan. Grounding language to entities and dynamics for generalization in reinforcement learning. arXiv preprint arXiv:2101.07393 2021. 2021. 3
Ross Wightman. Pytorch image models.
https: http://gtttthhccccee https://gtthbb..comm/ww htt.thaab.cmm/wwwgtttaa......
pytorch- imale-imae---ae iww.iiiaaaeeeeeeeeeeeeeeeeeeeeee models, s, 2019. 17
Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. arXiv preprint arxiv: 1XX1::911.00057. 2019.
Edward C Williams, Nakul Gopalan, Mine Rhee, and Stefanic Tellex. Learning to parse natural language grounded reward functions with weak supervision. In 2018 IEEE International Con- ference on Robotics and Automation (ICRA), pp. 4430 4436. IEEE, 2018. 2
Vinicius 7ambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Deep reinforcement learning with relational inductive biases. In International Conference on Learning Representations, 2018. 3,5
Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. Rtfm: Generalising to novel environment dynamics via reading, arXiv preprint
2019.
13

---

Under review as a conference paper at ICLR 2022
A ADDITIONAL EXPERIMENTS
A.1 FULLY-OBSERVED BABYAI
In addition to the standard partially observed BabyAl environment, we created fully observed version where the agent can view the entire world grid. In this fully observed setting we employ the same model architecture as in the Crafting environment, but with the hyperparameters from BabyAL. Results for a single seed on the BossLevel the fully observed setting can be found in Table 8, and look largely similar to those of the partially observed BabyAl environment.
Demonstrations
XFormer
Lang
50k
41.4%
73.4%
25k
39.8%
56.2%
12.5k
38.4%
40.2%
Table 8: Performance, in percent of unseen tasks completed, of instruction prediction loss fully observed version of the BabyAAl-Bsss.caa for a single seed.
A.2 ANALYZING INSTRUCTION INFORMATION AND ALFRED
In section 4.6 we find that instruction prediction more useful as task difficulty increases. this section, we try measure how useful provided instructions and additionally analyze instruction prediction in the ALFRED visual environment. Instruction prediction is likely to only provide a strong learning signal when they provide information not already available to the agent. Logically this makes sense: all of the requisite information has been given to the agent in its task, instructions will add nothing new. For example, one goal from the ALFRED environment is "put watch on the table". If the provided instructions for this task were "pick up the watch", "go to the table", and "put down the watch", the instructions would provide very little useful signal all of their information was already conveyed by the goal.
By default, ALFRED provides language instructions and goals as inputs to the agent. We remove the instructions from the imput so they can be used for our auxiliary instruction prediction objective. Following the methodology and architecture choices of Pashevich et al. (2021), we generate an ad- ditional 42K demonstrations in ALFRED and label them with vocabulary from the planner. Overall, we found that instruction prediction had little impact on performance across three seeds as seen in Table 9, Based on these results, we assess why instruction prediction was not fruitful in ALFRED.
We identify two ways which instructions can provide new information that useful for training- First, instructions can operate at differing resolutions than the goal. In BabyAl given tasks may involve completing numerous subgoals spanning navigation or object movement, and we provide instructions at this level instead of at the global scale. In the Crafting environment, goals like "make an iron pickaxe" are broken down into instructions like "mine iron ore", "go to the crafting bench".. Both scenarios have instructions that give information the agent at a finer temporal resolution than the goal. If provided goals are too obvious, they cannot practically be broken down at a finer resolution above action-level. The second way instructions can provide useful information is by revealing hidden sub-tasks. For example, instructions in BabyAl may reveal that an agent has to collect key to unlock door on the way to its objective. On the other hand, goals from ALFRED tend to be logically simpler as the ALFRED benchmark primarily focuses on visual understanding instead of logical task difficulty. As mentioned in section 4.6, instruction prediction most effective with harder tasks. Other example tasks in ALFRED include "Put clean sponge on metal rack" and "put a cooked potato slice on the counter".
Success Measure
Xformer
Lang
Task Success Rate Subgoal Success Rate
28.31 1.0%%..0% 28.5:E0.7% 1.01
36.0+0.8%
Table 9: We train models in the ALFRED environment with and without language prediction and evaluate their success the "seen" validation set.
14

---

Under review as a conference paper at ICLR 2022
Model Inputs
BabyAI, 50K
Crafting, 2.2k
ALFRED, 42K
Text Goal
86.4%
43.8%
96.9%
Text Goal and Observation(s)
92.4%
49.9%
99.0%
Table 10: Instruction prediction accuracies for models trained with and without access to observa- tions. When instructions can be easily predicted without access to observations, they likely provide little additional signal to the agent. The number next to the environment indicates the number of demonstrations used for training. The BabyAl Al level used was Bossl the hardest level.
We can measure how much information instructions able provide agent by measuring how easy it is to predict them from just the goal. If instructions can easily be predicted from the goal alone, then they are unlikely to provide any additional learning signal the agent. If the agent can only accurately predict instructions by observing the behavior in the demonstrations and the goal, then instructions prediction is more likely encourage salient representation learning or logical reasoning about the task. For each benchmark, we take our transformer based architecture for instruction prediction and train only the language head predict instructions with and without access to observations from the demonstrations. We report the best attained prediction accuracies in Table 10. Our results indicate that the instructions in ALFRED can be nearly predicted perfectly from the text goals (96.9%), indicating that the tasks are too easy and do not require instructions. This result has also been verified by the community. As of writing, one the top models on the ALFRED leaderboard found here does not even use the language instructions inputs. Conversely, we see lower accuracy overall and much larger gaps in accuracy for both BabyAl BossLevel (86.4% to 92.4%) and the Crafting environments (43.8% to 49.9%). We hope this result will drive the com- munity to develop more logically challenging benchmarks with complex tasks where the scaling of instruction prediction can be further studied. Additionally, one can begin assess the effectiveness of instruction prediction before using it by following this methodology.
A.3 ADDITIONAL BASELINES
We ran a additional baselines in the BabyAl environment.
1. GPT Enc: In order to demonstrate that instruction prediction is not just aiding in the agent's understanding text goals, we construct a baseline that encodes the text goals in BabyAl AI using pretrained GPT-2 Model before giving them to the agent. As the text-embeddings have been pretrained, this simulates the case where we have a maximal understanding of the text goal before interacting with the environment.
2. XFormer AC:
Our original architecture does not use previous actions as input due to the substantial increase in input tokens it causes. This baseline inputs both observation and action sequences into the transformer based model.
3. Goal Prediction:
Our architecture with language prediction, except instead of predicting the unseen instructions predict the goal text that is used as input to the policy. This is type of reconstruction objective in the text regime.
We ran these additional baselines for two seeds. Results for these new baselines and XFormer and Lang can be found in Table 11. We find that encoding text goals with GPT does not lead to perfor- mance gas as large as language prediction. This indicates that our instruction prediction helps with learning good representations for planning, and not just language understanding. The transformer with action inputs (XFormer AC) does not perform better than the regular transformer and in fact performs slightly worse, indicating that action inputs are not an important modeling component in the imitation domain and may just make learning harder by adding additional modalities and dou- bling sequence length. This is different than results found in the Offline RL setting in Chen et al. (2021b), which makes sense as rewards often depend on both states and actions. Moreover, inverse models in the discrete action spaces in BabyAl are relatively easy to learn Previous works with transformers in imitation (Pashevich et al., 2021) have also found that conditioning on entire action sequences leads to degradation of performance as policies more easily overfit.
15

---

Under review as a conference paper at ICLR 2022
Demos
XFormer
XFormer AC
GPT Enc
Goal Pred
Lang
50K
40.2 = 2.2
37.4 + 0.4
47.6 + 0.4
43.5 = 1.5
70.3 + 1.3
25K
39.9 = 0.5
37.0 + 0.3
37.6 0.3
39.6 + 1.9
55.4 + 7.0
Table 11: Results of additional baselines on the BabyAl Boss Level. The table gives success rates in % on 500 unseen levels.
B HIERARCHICAL BASELINES
Here we provide further details on our hierarchical baselines. The two prior works relevant on hierarchical language most relevant our investigations are Chen et al. (2021c) and Hu et al. (2019). Both of these works learn Markovian, or nearly-Markovian models.
In the crafting environment (Chen al., 2021c), the authors train high-level RNN to output the current language instruction. They then condition their low-level policy on the latent representation fed to the RNN that predicts instructions. While they show latent condition to be effective, trans- formers purposefully avoid encoding entire streams of data into a single vector, and instead operate on a token level. Thus, we found impractical to attempt this approach with our significantly more effective transformer models.
The environment in Hu et al. (2019) is a partially observed multi-player strategy game. As alluded to in the related work, this environment has multiple viable strategies and is thus distinct from the oracle imitation learning we mostly consider. Though the environment is partially observed, the authors do not train a sequence model. Instead, they concatenate command data from previous time-steps to the model input. As information from the very beginning of the trajectory is necessary for some BabyAl tasks, we found it impractical to scale this concatenation based approach. In Hu et al. (2019), a discriminative high level policy is trained to select an instruction from a fixed set. A lew-level is trained with ground-truth human labeled instructions output actions. In their strat- egy game, hierarchical approaches perform very well, unlike in our experiments where hierarchical models do not perform the best. One explanation for this comes from the nature of the strategy game environment. The distribution of optimal actions from a given state may be multi-modal, as different strategies may dictate different actions from the same state. Conditioning on an instruction would remove this multi colliii-odllity.... Below we describe the hierarchical approaches we tried. For all approaches we used the same architectures as detailed in Section 3.
F Fully-Observed Setting, In the fully observed setting, we adopt similar strategy to Hu et al. (2019). A high-level policy takes as input the observation o and goal g and predicts the current instruction (1) We train a low-level policy that predicts actions from the current instruction, goal, and observation T T(a, (*) .)) At test time, the high-level rute-segrysyyvely generates instructions that are then given to the low level.
Partially-Observee Setting. Unfortunately, find that there is no clear way train hierar- chical model using only transformers where instructions and actions operate at different time scales. Here are the methods we tried:
1. Sequences for Each Instruction. We take each trajectory take slices of it up until the com- pletion of each instruction. Our high-level model predicts only the language instruction corresponding to that trajectory slice. This essentially means that the high-level predicts one instruction conditioned on all the history before the instruction. The same sequences are used to train the low-level, conditioned on the single instruction. The low-level policy can be written as TT ( 0t, (*)) Because training sequence models with only indi- vidual losses is very inefficient, we train the low-level model output the correct actions at all points in time corresponding to the instruction it is conditioned on.
2. All Instructions. Instead of conditioning a policy on a single instruction, we train the high level policy to output all of the instructions for the entire task. This is especially challenging at the beginning of an episode when there are few frames. The low-level policy is then conditioned on the entire sequence of instructions and can be written as I (a, (aa/oor...., (o....., ot. x())) .... (n)). As this performs relatively well, hypothesize that the model learns to ignore instructions far in the future when deciding which actions to take at the current timestep.
16

---

Under review as a conference paper at ICLR 2022
Demonstrations
Seq for Each Instr
All Instr
All Instr, Aggressive Mask
50k
27.1 25.5:13.7%
36.8%135% 37.2:13.0 %
33.3+2.8% 32.4:12.3
25k
Table 12: Results for hierarchical configurations we tried.
3. All Instructions, Aggressive Mask. This is the same as the above, except we use an ag- gressive masking scheme when training the high level that only allows plans to be predicted from observations strictly preceding the time frame of the current instruction.
In Table 12 we give results for each the sequence-style hicrarchical approaches we tried on two seeds. In Table we report the accuracy for the All Instructions method which we found to perform best. Other potential methods could include providing an encoding the current instruction after each timestep the transformer. However, such approaches would either employ RNN bag- words style model to generate the encoding and not be purely transformer based like the rest of models.
C ADDITIONAL FIGURES
!
(i) Li
X2
X
Unmaksed Cross Attention
Seq2Seq Transformer Decoder
at
2
i
i
( +
CLS
t,,
ZrN
X-2
X-
X
Language Instructions
ViT Encoder
CLS
g
Or,1
Ot,N
Tokenized Observation and Goal
Figure 5: Architecture figure for the fully observed case. Observations are tokenized and then input to Vision Triio-Tranformmrr like encoder.
D HYPERPARAMETERS
Here we include all hyper-parameters we used. We used pytorch for our experiments. Our im- plementation for the partially observed sequence models was based on MinGPT by Andrej Karpa- thy. Our implementation of the Vision Transformer for fully observed environments Wightman (2019). We determined parameters for ATC by testing different frame skip values in both environments. In the crafting environment found 0.25 to perform better. We also tested using weight decay and dropout Baby with 50k demonstrations and found it have no significant impact and thus did not it for the other ex- periments. When evaluating models we found those for the Crafting environment to perform better Pytorch "train" mode, meaning with dropout on, while those for BabyAI worked better in "eval" mode. We ran our experiments on NVIDIA GTX 1080 Ti GPUs. In BabyAI we train all models for two seeds and the crafting environment we train all models for four. This was done because BabyAl AI experiments could take upwards of 48 hours to train.
based on
tested using À coefficients of 0.5 and 0.25 and
17

---

Under review as a conference paper at ICLR 2022
Hyperparameter
BabyAI
Crafting
Encoder Blocks
Decoder Blocks
1
Embedding Dim
128
128
MLP Size
256
256
Dropout
0
0.1
policy T
Dense Layer
Dense Layer
Batch Size
32
64
Training Steps
million
300k
Optimizer
Adam
A
Optimizer Epsilon
x 10-8
x 10-8
Learning Rate
0.0001
0.0001
Weight Decay
0.0
0.05
Grad Norm Clip
N/A
Xaag
0.7
0.25
AATC
0.7
0.25
EMA
0.01
0.01
EMA update freq
ATC Frame Skip
3
Table 13: Hyperparameters
18


Reviews:
[{"review_id": "vre2KlYyNRv", "reviewer": "Reviewer_cgzc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the overall problem and conclusion that predicting detailed instructions can improve performance using much smaller data than without this auxiliary objective. I think the perspective that labeling subtasks with language instructions is more sample efficient than collecting more demonstrations is also useful. I have some questions about the approach.\n\n1- Could you explain if \"instruction as a text\" or just \"objects and actions in instructions\" are more important? Based on Figure-3, it seems that each instruction includes an object and an action. It would clarify if your auxiliary objective is helping just the object detection, subtask understanding, even using OOD information that wouldn't otherwise be in the goal (such as a grey door is never seen in a goal but it is a part of navigating the environment), or if actual text is also important.\n\n2- What is the main bottleneck that instruction generation is solving? Similar to my above question, it could be object detection or subtask understanding but might also be that understanding a textual goal is difficult. A pre-trained language model for goal encoding would help answer this.\n\n3- To understand if detailed instructions are really crucial for instruction generation, an auxiliary loss baseline where goal is predicted at each time step is needed. It would help understand if text generation or detailed instructions are the most important; even for POMDPs, partially generated goals might be helpful.\n\n4- You use only observations and goal as input to the transformer encoder but DecisionTransformer [1] showed that using action and reward was crucial for success. I think a similar baseline with and without language prediction is needed.\n\n5- In Figure-2 and Figure-6, decoder outputs should be shifted by 1 but right now the same input is outputted at each step.\n\n6- In Table-1, you put \"=# Demos\", was this intentional or a placeholder for an exact number?\n\n7- Some grammatical errors:\n* In Section 3.1, \"works with our without\" should be \"works with or without\".\n* The same section, \"langauge\" should be \"language\".\n* Section 3.2, \"depcited\" should be \"depicted\".\n* The same section, \"except it the encoder\" should be \"except the encoder\".\n* Section 4.1, \"to an agents initial\" should be \"to an agents' initial\".\n* Section 4.2, \"Prior work has not use\" should be \"Prior work has not used\".\n* Appendix A.2, \"AFLRED\" should be \"ALFRED\".\n* Appendix B, \"to select a instruction\" should be \"to select an instruction\".\n\n[1] Decision Transformer: Reinforcement Learning via Sequence Modeling. Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, Igor Mordatch.", "summary_of_the_paper": "This paper investigates the usage of language prediction for imitation learning. The authors distinguish between a goal (a short description of the task) and an instruction (a detailed description of subtasks that span a sequence of observations). They show that predicting instructions at each step as an auxiliary loss improves the performance using lesser number of demonstrations. They experiment with minigrid and crafting environments to show the efficacy of the proposed approach.", "summary_of_the_review": "I think instruction generation as an auxiliary loss is an interesting approach and presents a new perspective that labeling subtasks with instructions is better than just collecting more demonstrations. I also think that some more baselines and clarifications are needed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qwzQrML_32n", "reviewer": "Reviewer_EYpA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I believe this to be a well-written, strong paper, with an elegant and simple approach. The baselines go above and beyond, and though the evaluation is limited to BabyAI and Crafting simulated environments in 2D, I understand how these environments can be complex and multi-faceted, and serve as excellent testbeds. \n\nI think the argument regarding the data regime is nuanced, but very very interesting, and is worth sharing with the community as well.\nHowever, the one weakness I have is a pretty big one, and one that I hope the authors can discuss during the rebuttal; the availability of the intermediate language instructions (those used for auxiliary language prediction), and ensuring these intermediate language instructions are of the structure that’s necessary for the approach to work! It seems to be the case that these “auxiliary” instructions need to be pretty distinct from the goal, spelling out explicit subtasks, that cannot be directly inferred.\n\nI’m curious where the authors think this data could come from when scaled up; when I look at the suggestions in the paper (exploiting PDDL planners in simulation environments — this seems debunked by the ALFRED experiments, or video captions — unclear that these have any instructional signal at all), I don’t really see the broader applicability of this approach. This feels like a strong, perhaps breaking assumption, which is really coloring my initial review.\n\n\nTypos/Style/Questions:\n\nPage 4: Langauge —> Language ", "summary_of_the_paper": "This paper explores a simple, elegant idea: the use of natural language as supervision when performing instruction following tasks. Centered around an evaluation of various tasks in the BabyAI complex grid world suite, as well as the Crafting environment introduced by Chen et. al (2021), the proposed approach takes an easy-to-digest tact: eat a sequence of observations from the environment with a Transformer, and predict two things: first, given a goal, predict the action to take (a traditional behavioral cloning objective). Second (and a core contribution of the approach) treat language prediction as an auxiliary task, predicting a sequence of language instructions that specify how to perform the task (which I liken to “subtasks” or some compact language describing an immediate outcome to optimize for, though the authors should please correct me if I’m misinterpreting!).\n\nThese language assumptions are assumed to come from an oracle (concretely, in the implementation, generated synthetically exploiting the simulated nature of the environments), though the authors argue that this sort of “guidance instructions” appears naturally in the wild; videos have captions, humans could provide this feedback cheaply, etc.\n\nThe bulk of the evaluation evaluates sample efficiency and performance on the long-horizon tasks in BabyAI/Crafting, comparing the proposed approach to several meaningful ablations (no auxiliary objectives, a hierarchical variant, a self-supervised contrastive variant, and the proposed approach). These baselines are incredibly well thought through, and serve as valuable comparison points for the proposed approach. \n\nThe final, perhaps salient point of this work is the data regime; the main body of the paper and the appendix both argue that the true value of this approach (language as auxiliary tasks) is when (1) we are operating in a moderate data regime — too few data, and we can’t learn decent behavior, too much data, and we can just learn tasks without additional supervision, and (2) when the supplemental language information is providing something new, over the goal information itself. Experiments on the ALFRED suite show that because intermediate language instructions can be predicted from the goal itself, they don’t have inherent value. ", "summary_of_the_review": "In general, I believe the proposed approach to be simple and elegant; the arguments about the data regime are well argued, and the evaluation is precise. However, my biggest concern has to do with the viability of obtaining the requisite auxiliary language instructions — I’m not convinced that this data can be easily obtained in the wild, which puts the value of this approach into question.\n\nI sincerely hope that the authors can engage with me on this point!", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oBO1iKe2F1", "reviewer": "Reviewer_hiJp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the core method of this work is quite simple (which is not inherently bad and I would even argue is inherently good). The authors add a auto-encoding objective to a transformer policy network, where during each step, an additional decoder ingests 1) the instruction for that step and 2) the observation encoder output for all previous steps and recovers the instruction for that step. This reconstruction loss is weighted with the usual policy loss and jointly optimized. On the BabyAI and Crafter experiments, this method achieve strong gains using fewer demonstrations (for imitation learning) and converges faster and to higher performance. I think it is quite clear that this method is very helpful for the tasks studied in this work.\n\nWhat I find lacking is how this comparison compares to the general claim that instruction modelling is useful in the more general sense (although I hope it is and would like to believe it is). I think what would significantly improve this paper is to study instructions with more complex behaviour which truly suffers from the lack-of-annotation problem. There are a number of such environments such as Touchdown (navigation with streetview panoramas, https://arxiv.org/abs/1811.12354) and DRIF (quadcopter control, https://arxiv.org/pdf/1910.09664.pdf), where due to the complexity of the trajectories and observation space it is prohibitively expensive to annotate language instructions. This would evaluate this technique on complex language instructions. Another way to go is to do more complex grid worlds with high stochasticity and more complex dynamics that actually require detailed understanding of the language because test language-entity associations are unseen and must be constructed from language. Examples here include RTFM (7+ hop reference resolution, https://arxiv.org/abs/1910.08210) and Messenger (NL instructions, small training distribution, https://arxiv.org/abs/2101.07393). These are also good related reference for the paper, which this draft doesn't include - some of the references this work does include would also be good test beds - because they also contain more complex grounding challenges due to visual observation complexity.\n\nAll of these previous works include dynamics and language instructions more complex that the ones studied in this paper. In particular, due to construction, once instructions are broken down into atomic steps, they don't even require sequential understanding. For example the instruction \"go to the red door\" is perfectly recovered with the binary features (goto, red, door). As a result, the experiments leave many questions unanswered, such as \n\n1) does this result generalize to complex environments where language-entity and language-dynamics associations are more difficult due to complex entity behaviour or complex observation?\n2) does this result generalize to complex language instructions where the instruction is more difficult to model (and therefore to understand)?\n\nUpdate because I looked at appendix:\nThe negative results from ALFRED further raises my suspicions that these results may not generalize. For the results in the Appendix: are the authors claiming that one can do ALFRED without visual observations? My understanding is that the ALFRED work shows exactly that you cannot do the task without visual observations (it is one of the ablations in their paper). Perhaps I am misunderstanding this result. Can the authors please comment on this?\n\nI would imagine that the result shown here is because the model is incapable of modelling the instructions, and therefore the reconstruction loss is not helpful. In particular, I would think that there is a high correlation between being able to do the task and being able to reconstruct instructions from the bottleneck representation. If the instructions are very complex to understand, then both are hard and the reconstruction loss is not helpful. One way to test this hypothesis is to plot the instruction recovery accuracy instead of loss (the authors show the latter but not the former).", "summary_of_the_paper": "This work proposes using instruction modelling as an auxiliary objective to improve long-term planning. On BabyAI and Crafter, this method demonstrates significant improvement on longer hop levels (e.g. 3 hop BabyAI).", "summary_of_the_review": "Strong results that show instruction modelling help policy learning on two simpler tasks. Evaluation leaves much to be desired in terms of substantiating main claims.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "BdWsypj46Ik", "reviewer": "Reviewer_z2VY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The novelty of this paper is limited. The authors use language as an auxiliary task to improve planning accuracy. Such a method has no technical contribution. I would like to see the summarization of the technical contributions from the authors in their feedback. \n\nThe experimental results are not very interesting. It is not surprising that using extra supervision will improve the results.\n\nCould the authors explain the difference between the claimed \"non-Markovian\" and the commonly used \"partially observable Markov decision process (POMDP)\"? Why do the authors use \"non-Markovian\" instead of \"partially observable Markov decision process (POMDP)\" in the paper?\n\nIn the Crafting environment, the results of \"Lang\" are comparable with the results of \"Xformer\" in Table 2. Is the model of \"Lang\" also larger than \"Xformer\" as it has the extra language encoder and decoder? Could the authors give any explanation of why using extra supervision generates comparable results? \n", "summary_of_the_paper": "In this work, the authors explore the use of language as auxiliary supervision for long-horizon imitation learning tasks. The authors show that using instruction modeling is able to improve the performance in planning environments when training with a limited number of\ndemonstrations on the BabyAI and Crafter environments. They further show that instruction modeling is most important for tasks that require complex reasoning in the experiments. ", "summary_of_the_review": "After reading the whole paper, I didn't find any interesting points that make me feel excited about this paper. I think the authors did good experiments to verify their thoughts, but this paper has no technical contribution and the results are not surprisingly good enough. \nThis paper is easy to read and has no significant fault. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all consider the paper to be below the acceptance bar. While the revision addressed some concerns, several critical ones remain open. This includes empirical concerns with regard to the extremely simple grid-world environments used, and with regard to the vague distinction between instructions and goal specifications. To improve the submission, the authors should seek stronger empirical foundations, and either refine or remove vague distinctions with regard to the phenomena they aim to study.

Special thanks to the reviewers for an extremely productive discussion.
Decision: Reject