Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning
Authors: Valerie Chen
Abhinav Gupta
Kenneth Marino
Venue: ICLR 2021 Poster (2021)

Abstract: Complex, multi-task problems have proven to be difficult to solve efficiently in a sparse-reward reinforcement learning setting. In order to be sample efficient, multi-task learning requires reuse and sharing of low-level policies. To facilitate the automatic decomposition of hierarchical tasks, we propose the use of step-by-step human demonstrations in the form of natural language instructions and action trajectories. We introduce a dataset of such demonstrations in a crafting-based grid world. Our model consists of a high-level language generator and low-level policy, conditioned on language. We find that human demonstrations help solve the most complex tasks. We also find that incorporating natural language allows the model to generalize to unseen tasks in a zero-shot setting and to learn quickly from a few demonstrations. Generalization is not only reflected in the actions of the agent, but also in the generated natural language instructions in unseen tasks. Our approach also gives our trained agent interpretable behaviors because it is able to generate a sequence of high-level descriptions of its actions.

Published as a conference paper at ICLR 2021
ASK YOUR HUMANS: USING HUMAN INSTRUCTIONS TO IMPROVE GENERALIZATION IN REINFORCEMENT LEARNING
Valerie Chen, Abhinav Gupta, & Kenneth Marino Carnegie Mellon University {vchen2, abbinavg, scdmarino]ed cmu. edu
ABSTRACT
Complex, multi-task problems have proven to be difficult to solve efficiently in a sparse-reward reinforcement learning setting. In order to be sample efficient, multi-task learning requires reuse and sharing of low-level policies. To facilitate the automatic decomposition of hierarchical tasks, we propose the use of step-by- step human demonstrations in the form of natural language instructions and action trajectories. We introduce a dataset of such demonstrations in crafting- basee grid world. Our model consists of high-level language generator and low-level policy, conditioned on language. We find that human demonstrations help solve the most complex tasks. We also find that incorporating natural language allows the model to generalize to unseen tasks in zero-shot setting and to learn quickly from a few demonstrations. Generalization is not only reflected in the actions of the agent, but also in the generated natural language instructions in unseen tasks. Our approach also gives our trained agent interpretable behaviors because is able to generate a sequence of high-level descriptions of its actions.
INTRODUCTION
One of the most remarkable aspects of human intelligence is the ability to quickly adapt to new tasks and environments. From young age, children are able acquire new skills and solve new tasks through imitation and instruction (Council et al. 2000; Meltzoff 1988 Hunt, 1965). The key is our ability to language learn abstract concepts and then reapply them in new settings. Inspired by this, one of the long term goals in AI is to build agents that can learn to accomplish new tasks and goals in an open-world setting using just a few examples or few instructions from humans. For example, if we had a health- care assistant robot, we might want to teach it how to bring our favorite drink or make us a meal in just the way we like it, perhaps by showing how to do this a few times and explaining the steps involved. However, the ability to adapt to new environments and tasks remains distant dream.
Previous work have considered using language high-level representation for RL (Andreas et al., 2017] Jiang et al., 2019). However, these approaches typically use language generated from tem- plates that are hard-coded into the simulators the agents are tested in, allowing the agents to receive virtually unlimited training data to learn language abstractions. But both ideally and practically, instructions are limited resource. If we want to build agents that can quickly adapt in open-world world settings, they need to be able to learn from limited, real instruction data (Luketina et al.) 2019). And unlike the clean ontologies generated in these previous approaches, human language noisy and diverse; there are many ways to say the same thing. Approaches that aim to learn new tasks from humans must be able use human-generated instructions.
In this work, we take a step towards agents that can learn from limited human instruction and demon- stration by collecting a new dataset with natural language annotated tasks and corresponding game- play. The environment and dataset is designed to directly test multi-task and sub-task learning, as consists of nearly 50 diverse crafting tasks] Crafts are designed to share similar features and
OOr dataset, environment, and code be found http://giiiieemmmmmaaccccaaaaaar

---

Published as a conference paper at ICLR 2021
Time 1
Time (+1
Language Generator
Inv: (EMPTY)
Inv: Key"" 1)
Palicy
Goal: "Iron Ore"
Goal: "Iron Ore"
Action: "grab"
Figure 1:
From state observation time step E, the agent generates natural language instruction "go to key and press grab," which guides the agent to grab the key. After the instruction is fulfilled and the agent grabs the key, the agent generates a new instruction &
sub-steps so would be able to test whether the method is able learn these shared features and reuse existing knowledge to solve new, but related tasks more efficiently. Our dataset is collected in crafting-based environment and contains over 6,000 game traces on 14 unique crafting tasks which serve as the training set. The other 35 crafting tasks will act as zero-shot tasks. The goal is for agent be able to learn one policy that is able solve both tasks it was trained as well as a variety of unseen tasks which contain similar sub-tasks as the training tasks.
To do this, we train neural network system to generate natural language instructions a high- level representation of the sub-task, and then policy to achieve the goal condition given these instructions. Figure shows how our agent takes in the given state of the environment and a goal (Iron Ore), generates a language representation of the next instruction, and then uses the policy select an action conditioned on the language representation in this case to grab the key. We incorporate both imitation learning (IL) using both the language and human demonstrations and Reinforcement Learning (RL) rewards to train our agent to solve complicated multi-step tasks.
Our approach which learns from human demonstrations and language outperforms or matches base- line methods in the standard RL setting. We demonstrate that language can be used to better gen- eralize to new tasks without reward signals and outperforms baselines crafting tasks. Our method uses language as high-level task to help decompose a larger, complex task into sub-tasks and identify correct sub-tasks to utilize in a zero-shot task setting. We also show that the agent can learn few-shot tasks with only a few additional demos and instructions. Finally, training with Human-gennratedd instructions gives an interpretable explanation of the agent's be- havior in cases of success and failure. Generalization is further demonstrated in the agent's ability explain how the task is decomposed both train and evaluation settings, way that reflects the actual recipes that describe the crafting task. With our dataset collection procedure and language- conditioned method, we demonstrate that using natural human language can be practically applied to solving difficult RL problems and begin solving the generalization problem in RL. We hope that this will inspire future work that incorporates human annotation, specifically language annotation, to solve more difficult and diverse tasks.
average over 35 zero-shot
RELATED WORK
Previous works on language descriptions of tasks and sub-tasks have generally relied on what An- dreas et al. (2017) calls "sketches." A sketch specifies the necessary sub-tasks for a final task and is manually constructed for every task. The agent then relies on reward signals from the sketches in order to learn these predefined sub-tasks. However, our setup, we want to infer such "sketches" from a limited number of instructions given by human demonstrations. This setting is not only more difficult but also more realistic for practical applications of RL where we might not have prede- fined ontology and simulator, just a limited number of numann-enneran test time, their true zero-shot task requires the sketch, whereas our method able to generate the "sketches" in the form of high-level language with no additional training and supervision.
instructions. In addition, at
2

---

Published as a conference paper at ICLR 2021
Similarly, other works have used synthetically generated sub-goals and descriptions to train their methods and suffer from similar problems of impracticality, [Shu et al. 2018)) introduces a Stochas- tic Temporal Grammar to enable interpretable multi-task RL in the Minecraft environment. Sim- ilarly, the BabyAl AI platform [Chvalier-Boisved et al. (2019a) presents synthetic language which models commands inside grid-based environment. They utilize curriculum training approach learning complex skills and demonstrate through experimentation in their environment that existing approaches of pure IL or pure RL are extremely sample inefficient. Cideron et al. (2019) extend Hindsight Experience Replay (HER) to language goals in the BabyAl platform to solve a single instruction generated from hand-crafted language. The BabyAI environment extended by Cao et al.] (2020) to include descriptive texts of the environment to improve the generalization of RL agents. Jiang et al. (2019) also uses procedural generated language using the MuJoCo physics en- gine and the CLEVR engine to learn hierarchical representation for multi-task RI.|Ohh et al. (2017) also tackles zero-shot generalizations, but like the others considers only procedurally-generee structions, learning to analogies to learn correspondences between similar sub-tasks.
in-
The main work that also investigates using a limited number of numann-generaun instructions in RL environments is Hu et al. (2019). This paper also uses natural language instructions in hierarchical decision making to play a real-time strategy game involving moving troop units across long time scales. This work uses only behavioral cloning with natural language instructions, whereas use a mixture of RL and imitation learning. They also do not investigate the benefits of language in zero-shot or few-shot settings and do not demonstrate cross-task generalization as we do.
Hierarchical approaches as way of learning abstractions is well studied in the Hicrarchical Re- inforcement Learning (HRL) literature (Dayan & Hinton 1993: Parr & Russell 1998 Stolle & Precup 2002). This is typically done by predefining the low-level policies by hand, by using some proxy reward to learn a diverse set of useful low-level policies (Heess et al., 2016 Florensa et al. 2017 Eysenbach et al.) 2018 Hausman et al., [0118] Marino et (0119), or more generally learning options (Sutton et al. 1999). Our approach differs in that unlike in options and other frameworks, we generate language as a high-level state which conditions the agent's policy rather than handing
control over to low-level policies directly.
Other works have shown the effectiveness of using combination of reinforcement learning and im- itation learning. Le et al. (0018) presents hybrid hierarchical reinforcement learning and imitation learning algorithm for the game Montezuma's 's revenge by leveraging IL for the high-level controller and RL for the low-level controller demonstrating the potential for combining IL and RL to achieve the benefits of both algorithms. By learning meta-actions, the agent is able to learn to solve the complex game. However, their meta-actions were also hand specified.
Others have utilized natural language for other tasks, including Williams et al. (0018) Co-Reyes
et al. 2018), and Andreas et al. (2018) but not focused on the multi-task learning setting. Matthews
al. 2019) demonstrates the use of word embeddings inform robotic motor control as evidence
of particular promise for exploiting the relationship between language and control. Narasimhan
al. (2018) uses language deseriptions of the environment to aid domain transfer. The sub-field
of language and vision navigation specifically has investigated how to train agents to navigate to
a particular location in an environment given templated or natural language (Chaplot et al.
2018
Anderson et al., 2018 Tellex et al., [011] 1 Mei et al. 2016) Chen & Mooney 2011 Yu et al. 2018)
or to navigate to a particular location to answer a question Das et al. (2018). Similarly to this work, Nguyen et al. (2019) uses human-generate generated language to find objects in a simulated environment, Zhong et al. (220)) and Branavan et al. (2012) reads a document (i.e. a players manual) to play a variety of games, and Lynch & Sermanet (2020) trains agents to follow both image and language- based goals. All of these works require the agent to read some text at both train and test time and follow those instructions to achieve some goal. In contrast, at test time, our agent only receives a high-level goal, which is what item to craft. Our agent must take the high-level goal as input and generates its own instructions to solve the task. In other words, our task both instruction following and instruction generation. Related to instruction generation, some work have explored more generally intrinsic motivation for goal generation (Florensa al., al,) 2018 Forestier al. 2017). In our work, however, we learn the goals via the human language instructions.
3

---

Published as a conference paper at ICLR 2021
3 HUMAN ANNOTATION COLLECTION
The first step of our approach requires human demonstrations and instructions. To that requirement, we built an interface to collect human-annotated data to guide the learning model.
Example Turker Instructions:
toggle switch and open door gruh axe and mine free grab pickixe cobblestonc
88 000
eoicauT kLOW BuT TICeE
20
8
wood plank and press craft to stick and press craft go to stone pickaxe and craft
Cnars
Figure 2: (Left) Example view of game interface that the worker would see on AMT. On the left the worker is given the goal and recipes; the board is in the middle; the worker provides annotations on the right. (Right) Example sequence of instructions provided by the Turker for the given task of Stone Pickaxe.
Crafting Environment: As shown in Figure our environment
Minecraft-inspired 5-by-5
gridworld. The Crafting Agent navigates the grid by moving up. down, left, and right. The agent can grab certain objects, like tools, if it is next to them and use the tools to mine resources. The agent must also use a key switch to open doors blocking its path. Finally, the agent can also go to a crafting table to build final items. The agent can choose from 8 actions to execute: up, down, left, right, toggle, grab, mine, and craft. The environment is fully observable. Our crafting environment extends the crafting environment of Andreas et al. (2017) to include obstacles and crafts that are specified by material, introducing compositionally complex tasks (i.e. instead of 'Make Axe", we have 'Make Iron Axe' etc). In total, we consider about 50 crafting tasks, 14 of which we collect annotations for and 35 of which are used for test time. At the start each game, all object/resoorce locations are fully randomized in the environment.
The goal of the agent our world is to complete crafts. By design, grafting-based world allows for complexity and hierarchy in how the agent interacts with items in the gridworld. To craft an item, the agent must generally first pick up tool, go to a resource, mine the resource, and then go to a table to craft the item. To make an iron ore, the agent must use the pickaxe at the Iron Ore Vein to mine Iron Ore to complete the task. The Iron Ore recipe is an example of a 1-step task because it creates one item. A 5-step task, like Diamond Pickaxe, involves the mining and/or crafting of 5 items. We capped the tasks at a maximum length of recipe steps to limit the amount of time worker would have to spend the task. Note that each recipe step requires multiple time-steps to complete. Crafts are designed to share similar features and sub-steps to test whether the agent is able to learn these shared features and reuse existing knowledge to solve new, but related tasks more efficiently (these relations between tasks are detailed in Table 3 and in Figure I0). While the task may seem simple human annotators to solve, such compositional tasks still pose difficulties for sparse-rewand reward RL. We further increase the difficulty of this task by restricting the agent to limited number of steps (100) to complete the task, leaving little room to make unrecoverable mistakes such as spending time collecting or using unnecessary resources.
Crafting Task:
Data Collection Process: Figure 2 shows our interface, Given the goal craft, relevant recipes, and the initial board configuration, the worker provides step-by-step instructions accompanied by exe- cution on the actual game board each instruction. The workflow would be to type one instruction, execute the instruction, then type the next instruction, and execute until the goal was completed. The data collection interface and corresponding example set of natural language instructions provided by Turker is illustrated the rightmost side of Figure 2 This is but one way that Turker might choose to break down the 5-step crafting task. The appendix has more details on the collection We will release the environment and dataset.
process in Section A.1
4

---

Published as a conference paper at ICLR 2021
Dataset Analysis: Between the 14 crafts, collected 6,322 games AMT. In total, this dataset contains 195,405 state-action pairs and 35,901 total instructions. In the supplementary material we present relevant summary statistics about the data, including the number of instructions provided for each n-step task. The number of instructions, and consequently, actions required increases with the number steps shown in Table
4 METHODS
Our proposed approach to solving these multi-step crafting tasks is to learn from human -emaanggeeertedd natural language instructions and demonstrations. The model is first pre-trained using imitation learning (II.) and then fine-tuned using sparse-reward in reinforcement learning (RL). The goal of the agent is to learn one policy that is able solve variety of tasks (around 50) in the environment including ones it has not seen when only trained on a subset of the total tasks.
Staee Encoding
Ensoding
the
Cabeet
Sibeet
Sacer
<xtartt
Figure 3: (Left) High-level language generator. (Right) Low-level policy conditioned on language.
Architecture: As outlined in Figure we factor the agent into a hierarchical set-up with a language generator at the high-level and policy conditioned the language at the low-level. At each time step, the state encoder produces a vector representation that is then used as input to both the language generator and language-conditionel policy. Relevant information about the state, including the grid, inventory, and goal are encoded. Items which are relevant for crafting are embedded using a 300- dimension GloVe embedding, summing the embeddings for multiple word items (i.e. Iron Ore Vein). Non-crafting items, such as door, wall, or key, are represented using one-hot vector. Further details are provided in Section B
Imitation Learning Pre-training: We warm-start the model using the human demonstrations. Lan- guage is generated at the high-level with an encoder-decoder framework. The encoding from the state encoder is decoded by an LSTM which generates natural language instruction. The target language instruction is the AMT worker's provided instruction. our dataset, the vocabulary size was 212, after filtering for words that appeared at least times. At test time, do not have access to the ground truth instructions, so instead the LSTM decoder feeds back the previously generated word as the next input and terminates when the stop token is generated. From the language generator module, we extract the last hidden state of the generated instruction. The hidden state is concate- nated with the encoded state and passed through a series of fully connected layers. The final layer outputs the action. In the supervised training phase, the full model is trained by backpropagating through a language and action loss (cross entropy loss).
Reinforcement Learning Fine-tuning: We use the proximal policy optimization (PPO) algorithm Schulman al. (2017) in RL with the reward defined below to learn an optimal policy map from state encoding to output action. The maximum number of steps in an episode is set to 100. We utilize a training set-up which samples from all tasks (1-step through 5-step tasks). In preliminary experiments, we observe that sampling from 3-step tasks alone, for example, poses too complex of an exploration problem for the model to receive any reward. We define sparse reward, where the agent only receives a reward when it has completed the full craft. In RL fine-tuning, we freeze the language generator component because there is no more language supervision provided in the simu- lated environment. We also find that empirically backpropagating the loss through language distorts the output language as there is no constraint for it to continue to be similar to human language. All training hyperparameters and details are provided in supplementary materials.
5

---

Published as a conference paper at ICLR 2021
5 EXPERIMENTS
We compare our method against five baselines (1-5) which are reduced forms of our method to evaluate the necessity of each component. We also consider two baselines (6-7), which swap out the language generator for alternative high-level tasks, to evaluate the usefulness of language as selected high-level task. These baselines have the additional training that our method received, as well as the implicit compositionality, but without language. In both baselines (6-7), we perform the same training steps as with our method. Implementation details are presented in Section B
1. II: The II. baseline uses the same low-level architecture our method, without high-level hidden state. The model learns to map state encoding to an output action.
2. IL w/ Generative Language: IL w/ Generative Language is the supervised baseline of our method, which does not include RL reward. This baseline allows us to observe and compare the benefit of having a reward to train in simulation when the model has access to both actions and language instructions.
3. IL w/ Discriminative Language: We compare method to closely adapted version of the method proposed in Hu et al. (019) which similarly uses language in the high-level. Rather than generate language, their high-level language is selected from a set of instructions from the collected user annotation. We discuss this adaptation in the Appendix. They consider instruction sets of sizes N (50, 250, 500} and find the best performance the largest instruction set N 500 which is the size we use in our implementation.
4. RL: Another baseline consider is the reinforcement learning (RL. setting where the agent is provided no demonstrations but has access to sparse-reward in RL. The architecture we use here is the same as the IL architecture. This baseline demonstrates the capacity learn the crafting tasks without any human demonstrations and allows us to see whether human demonstrations are useful.
5. IL + RL: We also consider a baseline that does not incorporate language which is IL+RL. In IL+RL, we pretrain the same IL architecture using the human demonstrations as a warm-start to RL. It is important to note that this baseline does not include the natural language instructions as a part of training. We extract all of the state-action pairs at train a supervised model on the data as in the II model and then we utilize the RI. sparse-reward to fine-tune.
6. State Reconstruction (SR): We train an autoencoder to perform state reconstruction. The au- toencoder reconstructs the state encoding and the vector at the bottleneck of the autoencoder used as the hidden layer for the policy. SR as a baseline allows us to consider latent representations in the state encoding as a signal for the policy.
7. State Prediction (SP): We train a recurrent network, with the same architecture our language generator, to perform state prediction. The model stores the past 3 states from time T to predict the and T are used to predict state T + 1. From
T + state. So time T, the states T - 2, T the LSTM, the hidden state is extracted in the same manner as our IL + RL w/ Lang model. SP as a baseline allows to compare against another recurrent high-level method with the additional computation power.
5.1 RESULTS
Standard setting: We evaluate the various methods on crafts which we have collected human demonstrations for to benchmark comparative performance in our environment. An initial analy- sis is to first consider how much the IL model is able to learn from human demonstrations alone, so we consider IL, IL with Generative Language, and IL with Discriminative Language (results are in Section C. None of these approaches are able to solve the most difficult 5-step tasks or the simpler tasks consistently, with an average of about 18-19% success rate for 1-step tasks. We believe the and 5-step tasks are difficult enough such that annotations alone were not able to capture the diver- sity of board configurations for the varicty of crafts given that the board is randomly initialized each time. However, based an analysis of the language selected (see Tables [1 vs. Table 12), the gener- ated language is more interpretable and made more sense in a zero shot setting. Given the language is fixed after this point, all remaining experiments moving forward use generative language.
As shown in Figure 4 our method performs well against baselines. We find that human demonstra- tions necessary to guide learning because the learned behavior for RL is essential to arbitrarily

---

Published as a conference paper at ICLR 2021
Step Tasks
Step Tasks
Step Tasks
Step Tasks
1.0
1.0
1.0
0.81
0.8-
0.81
20c
ACCURICC
ACCLEWI
0.4
0.4
02
20000 60030
60000
20000 40030 09003 35030
20000 00000 60000 EAGADES
20000 40000 00000 B3003
Eptoodes
EpisOds
Figure 4: Comparing baselines with our method on accuracy. Human demonstrations are necessary complete tasks with 3 or more steps. Averaged over 3 runs.
walk around the grid and interact with items. For simple 1 and step tasks, this a feasible strategy for the allotted steps for an episode. However, there is little room for error in the most difficult 5-step tasks, as even human demonstrations take on average 40 steps to solve. We also find that for the standard setting, incorporating high-level network allows the model to achieve good results when comparing our method to SP and SR.
In Figure | we show the result of method when ablate the number of demonstrations use. This lets us see how many demonstrations we would feasibly need for the model to learn how to solve the crafting tasks. As we decrease the amount of data provided, find that there is greater variance in the policy's 's ability to complete the task, but the performance only significantly degrades when we start using only 25% of the data on the hardest tasks.
1 Stop Tasks
2 Stop Tasks
3 Stop Tasks
5 Steep Tasks
1.0
1.0
1.0
25%
0.6
75%
0.0
0.4
25%
0.4
004
25%
50%
50%
50%
75%
75%
75%
0.0
100%
0.0
100%
0.0
100%
0.0
20000
40000
00000
80300
20000
40090
69000
80300
20000
40000 Episodes
63000
80200
20003
40050 50000
90000
Ep-sodes
Eprsodes
Episodes
Figure 5: Ablation our method with varying amounts of human annotations (25%, 50%. 75% and 100%). For each fraction, we sample that number of demonstrations from the dataset for each type of task. Averaged over runs.
Zero Shot: Our method is able to use natural language instructions to improve performance on difficult tasks in the standard setting. But how well is our method able to do on completely new tasks not seen during training? We investigate our performance receives no human demonstrations or instructions, and no rewards on these tasks. The agent has to try to complete these tasks that it has never seen before and cannot train on at all. These unseen tasks do share sub-task structure with tasks which were seen in the training process, so the desired behavior is for the model to reuse subpolicies seen in other contexts for this new context. For example, in training the agent might have seen demonstrations or received rewards for task like "Cobblestone Stairs" and "Iron Ingot." At test time, we evaluate the agent "Cobblestone Ingot", which has never been seen by the agent. The agent should be able to infer the sub-task breakdown given prior knowledge of similar tasks.
zero-shot tasks, where the agent
item like
We present 35 examples of unseen tasks in Table I We find that overall our method outperforms all other baselines. While SR and SP were able match our method's performance in standard setting, they are not able to generalize. SR and SP are viable solutions to learn complex tasks in the standard RL setting, but the representations these models learned do not aid in generalizing to unseen tasks. Here, we believe, using language is key because it creates a representation that better abstracts to new tasks. the supplementary material, we show that in the cases of unseen tasks, the model indeed is able to generate language that properly corresponds to these new combinations of materials and items, particularly decomposing the complex item into sub-tasks that were previously seen in the training phase.
Demonstration Only and Few-Shot: In the demonstration only, we assume that we have access to only human demonstrations for some subset of tasks. From the entire pool of 14 tasks we collected demonstrations for, we withhold tasks (around 20% of total tasks) for testing. These 3 tasks consist of one, two, and three step task. We results on 3 permutations of withholding
7

---

Published as a conference paper at ICLR 2021
Table Accuracy evaluated on 100 games for 35 unseen crafts. Our method outperforms baselines. We do not list IL or IL w/ Language results which are 0% for all tasks.
Colllestone Chespylate
Diemo States
Coldestion: Boots
Stene Basis
Brisk Booto
Coldestone Legaine
Stane Logants
Briek Laggios
Diamoed Loggins
Cobble sane Davor
Stond Dovar
Brick Dast
Olamond Daose
Stowe Chesppatee
Brick Chesppaaee
Diabedd Cheaplate
Colbletone Helms
Stowe Hehmon
2-VG
2
95
90
81
72
28
RL.
92
98
85
94
91
95
97
21
96
37
18
97
82
97
95
94
91
SP
0
20
0
23
37
2
69
98)
0
98
89
1
74
33
74
70
100
Loather Stairs
Wood Boots
Cobblexuse Ingos
Stooe Engus
Goild Engost
Binck Engedt
Dianond loger
Wosd Inged
Wood Loggint
Loather Dawe
Woad Chesplate
Woad Helond
Brick Pickane
Loathrr Pickate
Windl Piakare
Frep Pick Pickanc
Goid Piebae
OvaaBlAVG
3AG
SANG
Stops
RL
0
0
0
0
0
23
11 IR
90
87
0
85
29
86
87
39
55
4
10
SR
17
38
Our
97 98
3
18
96
39
95
98
66
tasks. For each of the 3 withheld tasks, we include these demonstrations in the supervised training phase but do not provide reward in RL fine-tuning. We vary the amount of demonstrations that are provided: 5%, 10%, and 100%. The most generous case is to assume that the model has access to all demonstrations that were collected in the dataset. Per task, the total number demonstrations was about 300-500. Additionally we considered more strict few-shot case where we reduce the number of demonstrations to 20-40 which is about 5-10% of the original number of demonstrations. We do not include 5-step tasks because we only collected demonstrations for two 5-step tasks. From the results in Table 2 we can see that our method outperforms baselines in its ability to utilize the few demonstrations to improve performance.
Table 2: Evaluation of few-shot tasks for our method against baseline comparisons. We consider three settings for how many demonstrations are given to the model: 5% (20 demos), 10% (40 de- mos), 100%. Variance results are included in supplementary material. Results are averaged across seeds.
II. Lang
Steps
5%
1083
1000%
5%
108
100%6
5%
10%
100%6
5%
10%
100%
5%
10%
100%
5%
10%:
100%
I-step
16%
18%
18%
17%
19%
19%
96%
91%
9875
966
98%
97%
53%
84%
9496
97%
90%
95%
2-sepp
3%
6/1%
71%
40%
73%
3-step
1%
29
0%
1%
3%
4%
156
23%
228
10%
27%
46%
0%
31%
50%
5%
47%
74%
Interpretahility: One key benefit of incorporating natural language into the model is the ability for
humans to interpret how the model is making decisions. We observe that the generated instructions closely match those of the recipes that we provide to the annotators in the data collection phase in both train (Table (2) and test (Table 13 14 settings. However, the discriminative language didn't 't break down the task into steps that made sense (Table TTp. Figure presents example instructions generated by our model.
CONCLUSION
In this paper. we present a dataset of human demonstrations and natural language instructions to solve hicrarchical tasks in a trafting-based world. We also describe a hierarchical model to en- able efficient learning from this data through a combined supervised and reinforcement learning approach. In general, we find that leveraging human demonstrations allows the model to drastically outperform RL baselines. Additionally, our results demonstrate that natural language not only allows the model to explain its decisions but it also improves the model's performance on the most difficult crafting tasks and further allows generalization to unseen tasks. We also demonstrate the model's ability to expand its skillset through few additional human demonstrations, While we demonstrate our approach's 's success in a grid-based crafting environment, we believe that our method is able to be adapted towards generalizable, multi-task learning in a variety of other environments.
8

---

Published as a conference paper at ICLR 2021
"Go key and press grab"
"Go to pickaxe and press grab"
"Go to cobhlestomes and press mine"
"Go to cobblestones press craft"
X
Tav: (EMPTY)
Im: Mmn:"%ee ("Kyy"133
Ins: ("Pickas" I. "Key. 3)
lav: PPickane" 1,
Iev: P'Picktt''
"Coobisteee": 11
1]
Figure 6: Generated language test time for 2-step craft. We only display key frames of the trajectory which led to changes in the language. These key frames match changes in the inventory the object mentioned in the generated instruction. Qualitatively, the generated instructions consistent during what we would describe as sub-task. Quantitatively, the network will spend on average 4.8 steps in the environment for the same generated language output.
REFERENCES
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 3674-3683, 2018
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 166-175. JMLR. org, 2017.
Jacob Andreas, Dan Klein, and Sergey Levine. Learning with latent language. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies, Volume (Long Pupers), pp. 2166-2179, 2018.
SRK Branavan, David Silver, and Regina Barzilay. Learning to win by reading manuals in monte- carlo framework. Journal of Artificial Intelligence Research, 43:661-704, 2012.
Tianshi Cao, Jingkang Wang, Yining Zhang, and Sivabalan Manivasagam, Babyai++: Towards grounded-language learning beyond memorization. arXiv preprint arXiv:2004.077 072200004.700, 2020.
Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Ra- jagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
David L Chen and Raymond Mooney, Learning to interpret natural language navigation instruc- tions from observations. In Tiinty-Fifth AAAI Conference Artificial Intelligence, 2011.
Maxime -hevseter-Boisvert,, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Blby First steps towards grounded language learning with a human in the loop. International Conference on Learning Representations, 2019a. URL
ew net. wtt.eeew..eeeeteeeeeeeeeee
Maxime Snevalier-Boisvert Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. AabyAl: First steps towards grounded language learning with human in the loop. In International Conference Learning Representations, 2019b. URL htttps://openreviiee
net/
Geoffrey Cideron, Mathieu Seurin, Florian Strub, and Olivier Pietquin. Self-educated lan- guage agent with hindsight experience replay for instruction following. arXiv preprint 1rXiv:/910009455 2019.
John D Co-Reyes, Abhishek Gupta, Suvansh Sanjeev, Nick Altieri, Jacob Andreas. John DeNero, Pieter Abbeel, and Sergey Levine. Guiding policies with language via meta-learning In Interna- fional Conference Learning Representations, 2018.

---

Published as a conference paper at ICLR 2021
National Research Council et al. How people learn: Brain, mind, experience, and school: Expanded edition. National Academies Press, 2000.
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Em- bodied Question Answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In Advances neural infor- mation processing systems, pp. 271-278, 1993.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. In International Conference on Learning Representa- fions, 2018.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical rein- forcement learning. In International Conference on Learning Representations, 2017.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pp. 1515-1528, 2018.
Sébastien Forestier, Rémy Portelas, Yoan Mollard, and Pierre- Yves Oudeyer. Intrinsically motivated goal exploration processes with automatic curriculum learning. arXiv preprint 1roXiv:700..21111 2017.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. 2018.
Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, and David Silver. Learning and transfer of modulated locomotor controllers. arXiv preprint arXiv:1610005182, 2016.
Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuandong Tian, and Mike Lewis. Hierarchical deci- sion making by generating and following natural language instructions. In Advances neural information processing systems, pp. 10025-10034, 2019.
JMcVLevine Hunt. Intrinsic motivation and its role psychological development. Nebraska symposium on motivation, volume 13, pp. 189-282. University of Nebraska Press, 1965.
Yiding Jiang, Shixiang Shane Gu, Kevin Murphy, and Chelsea Finn. Language as an abstrac- tion for hierarchical deep reinforcement learning. In Advances in Neural Information Processing Systems, pp. 9414-9426, 2019.
Hoang Le, Nan Jiang, Alekh Agarwal, Miroslav Dudik, Yisong Yue, and Hal Daumé. Hierarchical imitation and reinforcement learning. In International Conference on Machine Learning, pp. 2917-2926, 2018
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefen- stette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. In Proceedings of the Twenty-Eighth International Join! Conference on Arti- ficial Intelligence, IJCAI-19, pp. 6309-633 6311. International Joint Conferences Artificial Intel- ligence Organization, 7 2019. doi: 10. 10.24963/jcca.220199888 24963/11ca .ttai..019//00 2019/88890
Corey Lynch and Pierre Sermanet. Grounding language in play. arXiv preprint arXlv:2005.07648, 2020.
Kenneth Marino, Abhinav Gupta, Rob Fergus, and Arthur Szlam. Hierarchical rl using an ensemble of proprioceptive periodic policies. ICLR, 2019.
David Matthews, Sam Kriegman, Collin Cappelle, and Josh Bongard. Word2vec to behavior: mor- phology facilitates the grounding of language in machines. 2019.
10

---

Published as a conference paper at ICLR 2021
Hongyuan Mei, Mohit Bansal, and Matthew R Walter. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In Thirtieth AAAI Conference on Artificial Intelli- gence, 2016.
Andrew N Meltzoff.
Imitation, objects, tools, and the rudiments of language in human ontogeny. Human evolution, 3(1-2):45-64 1988.
Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. Grounding language for transfer in deep reinforcement learning. Journal of Artificial Intelligence Research, 63:849-874 874, 2018.
Khanh Nguyen, Debadcepta Dey, Chris Brockett, and Bill Dolan. Vision-based navigation with language-based assistance via imitation learning with indirect intervention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12527-12537, 2019.
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Bearning-Volume Volume 70, pp. 2661-2670. JMLR. org, 2017.
Ronald Parr and Stwart Russell. Reinforcement learning with hierarchies of machines. In Advances in neural information processing systems, pp. 1043-1049, 1998.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Tianmin Shu, Caiming Xiong, and Richard Socher. Hierarchical and interpretable skill acquisition in multi-task reinforcement learning. In International Conference on Learning Representations, 2018.
Martin Stolle and Doina Precup. Learning options reinforcement learning. In International Symposium abstraction, reformulation, and approximation, pp. 212-223. Springer, 2002.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181- 211, 1999.
Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew Walter, Ashis Gopal Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In TwentyFiftth AAAI Conference on Artificial Intelligence, 2011.
Edward Williams, Nakul Gopalan, Mine Rhee, and Stefanie Tellex. Learning to parse natural language grounded reward functions with weak supervision. In 2018 IEEE International Con- ference on Robotics and Automation (ICRA), pp. 1-7. JEEE, 2018.
Haonan Yu, Haichao Zhang, and Wei Xu. Interactive grounded language acquisition and general- ization in 2d world. In International Conference on Learning Representations, 2018. URL
net/ ptttps://pperrevee wttp://wwwwweetoouueeeeeeeeee CaaeuummddH110mmm
Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. Rtfm: Generalising to new environment dynamics via reading. In International Conference on Learning Representations, 2020. URL
hnt.eiiwwweeeeeeeee....e.m forum? diaimmmddddsseoeseeeeee

---

Published as a conference paper at ICLR 2021
A DATASET
A.1 COLLECTION PROCESS
In this section, we provide additional details for our data collection process on Amazon Mechanical Turk (AMT). Firstly, we filter workers by the following criteria: a) HIT Approval % of greater than 95, b) Location is in an English speaking country, c) soft block has not been granted. These criteria help to ensure the quality of collected data, particularly in terms of natural language.
clisage
Accepts
Complets
HT
ertrance
Webslle
HIT
worker
Figure 7: Workflow to collect human demonstrations for our dataset.
We collected the dataset over the course of a few weeks. For each HIT, paid the Turker $0.65. On average the task took about 3-4 minutes. For each HIT, we generate a unique entrance code that provided to the Turker on the AMT website. The Turker is also provided with unique exit code once the HIT is complete. Given the entrance and exit code we are able to pay workers accordingly for their demonstrations.
Workers were provided with an entrance code at the beginning of the task to enter the website and an exit code when they completed the task be able submit the HIT. This enforces that we do not have workers doing extra HIT's that we are unable to pay for and to ensure that worker who submit HITs have indeed completed our task. Then we also wrote parsing script to be able quickly verify all submitted HITs to the task before payment. Specifically, we used this script to manually review the list of instructions generated for each task to ensure that the instructions provided were indeed pertinent the task at hand.
We had many returning workers who completed our task. A few even wrote emails to our requesting email to let us know that they really enjoyed the HIT and thought was quite interesting work on. This suggests that collecting demonstrations this kind is relatively interesting for humans to provide.
A.2 HIT INSTRUCTIONS
Prior to starting the task, each worker was provided with this instructions page which gives an analogy for cooking stir-fry as an analogy for the types of instructions that we believe they would provide. The demo page is shown in Figure g
Invoking users provide partieular level of specificity was difficult convey without explicitly providing example instructions. We deliberately chose not to provide examples as to not prime the worker particular format to follow. New workers were given two short games complete to familiarize themselves with the environment. Returning workers were given one longer game to complete as they already had experience with the task. Workers who completed the task as previously described were fully compensated.
Originally, some workers provided not enough instructions, meaning that they wanted to finish the task as quickly possible, and other workers provided instructions that were too granular, meaning that they did not abstract the task into sub-tasks and rather wrote "press left" or "go up 1" as their instruction. Prior approving the HIT, checked prior the worker submitting the HIT and built in precautions so that they had to redo a level they did not comply with such instructions clearly delineated in the demo.
12

---

Published as a conference paper at ICLR 2021
Demo
Instructions
instructions the
These of ceme Wiehow the annotation for; setting.
gives you acttletee dooccietion level
speelfic about
1. Select vegatables to
Slice the vegetabres pleces.
Creats NITER AaR Payer--rr---aa Vegetaace....
Seniuurfy our taek, you All be glen goal for agent accompilsh.
2014
And instruction schlove
high just Wilkoow
meaninghu
DOLA INSTRCTIION
this DOWN,
goal
By and this should MULTIPLE uctions, each fullowed button/eey
should meaninghu Instructions which "upp, timee""
fellos instructions sill baid
Figure Demo instructions for AMT workers.
13

---

Published as a conference paper at ICLR 2021
A.3 ADDITIONAL ENVIRONMENT AND CRAFTING TASK DETAILS
FigureDgives an example of the type of task (Make Iron Ore) that would be presented to the worker, which includes the goal, recipes, and the current board. Table 3 shows how the tasks related in terms of sub-tasks. This might be because of a similar material (i.e. Iron) or a similar craft (i.e. Stairs).
GOAL: Make Iron Ore (Iron Ore=1)
RECIPES:
MINE
OO
Ore
Pickaxe
Frn
Figure 9: Example board and goal configuration where the goal make an iron ore. The worker uses the recipes provided to give appropriate instructions and execute accordingly.
Table 3: List of recipes for which have collected annotations, labeled by the number of steps needed to complete it and other recipes which may share sub-tasks underlying structure.
ID Recipe Name
Steps
Related Crafts by ID
Gold Ore
2
Iron Ore
1,8
Diamond Boots
12, 14
4
Brick Stairs
2
5,7
Cobblestone Stairs
4,7,
6
Wooden Door
Wood Stairs
4,5,6
Iron Ingot
9
Leather Leggins
10, 11, 12
10
Leather Chestplate
9,11, 12
11
Leather Helmet
3
9, 10,
12
Leather Boots
3, 10, 11
13
Stone Pickaxe
5
5, 14
Diamond Pickaxe
3.
14

---

Published as a conference paper at ICLR 2021
Wood stairs
Diamond Pickasre
Stone Pickaxe
Slick
Diamond Vein
Slick
Cooblesione Stash
Wood Plank
Wood Plank
Wood Plank
Tree
Tree
Tree
Figure 10: A more in-depth example of out of the 14 training tasks to show how the subtasks are related (red boxes final craft, blue boxes raw material).
Table 4: Summary statistics for tasks of varying difficulty.
Steps
Average of Instructions
Average # of Actions
1-step
3.7
15.4
2-step
4.9
21.5
3-step
6.1
27.6
5-step
8.8
40.1
A.4 EXAMPLE DATA COLLECTED
Table gives examples of instructions randomly sampled from our dataset. Even with limited number of crafts, we were able to collect language instructions with diversity in sentence construc- tion. There are inconsistencies in capitalization and spelling which we handled in the preprocessing of the data. Table shows the most frrquently used instructions. Figure II gives summary statistics of the instruction side the dataset.
500
15000
6000
Prequcniy
Feqqeenyy
300
10000
4000
200
2000
0.0
length
Figure 11: (Left) Instruction frequency (Middle) Word frequency (Right) Histogram of instruction lengths.
15

---

Published as a conference paper at ICLR 2021
Table 5: Examples of randomly sampled instructions.
Grab the pickaxe.
Craft diamond axe with its bench, stick, diamond.
Make stone pickaxe from stick and cobblestone.
Pick up key using "GRAB". Grab the key; this may be useful. Collect Tool and Pass Through Door.
Go to leather boots bench and Craft Leather Boots.
Move to Tree.
Craft at the leather helmet bench
Go to brick stairs bench and craft.
Make a leather chestplate from the leather.
Move up to the axe and pick up.
Unlock the door.
Use Mine on Iron Ore Vein to collect Iron.
Move up two squares and grab the key.
Use pickaxe to mine diamond ore vein.
Go swtich and open door with Toggle Switch.
Move to Stick bench and Craft Stick.
Chop down the tree to get its wood.
Craft Wood Plank.
Go the stone pickaxe crafting bench.
Harvest wood from using "MINE".
Mine diamond ore.
2. "Open Door", then move to Pickax and "grab".
Go stock and click Mine harvest Cobblestone.
Craft brick stairs with its bench and brick.
Toggle the switch to open the door.
Go "Mine" both the iron ore vein and the coal vein.
Grab the pickaxe.
Mine the wood.
Go back through door and mine gold ore vein.
Mine Diamond Ore Vein.
Walk to brick factory and acquire bricks.
Chop down the tree to get its wood.
Go to wood bench and Craft Wood Plank.
Move to Rabbit.
Craft Diamond Pickaxe.
Go to tools and click Grab to take each one.
Move to ingot.
Grab key.
Next step, use axe on tree to get wood.
Go to stick workbench and press craft.
Go iron vein and mine for iron ore.
Go TOOL (Pickaxe).
Unlock the door with the key and enter the room.
Go to the stone pickaxe table.
Pick up axe and put into inventory.
Move to Wood Plank bench and Craft.
Craft at the stick bench.
Mine cobblestone stash, then go to tree. Craft Wooden Door.
Open the door to your right.
Mine the cobblestone stash.
Craft diamond boots.
Get eht pickaxe.
Use pickaxe to mine iron ore vein.
Go to tools and click Grab to take each one.
Flip the switch.
B METHODS DETAILS
Given the dataset of human demonstrations, convert traces of each game into state-action pairs for training. the subsequent sections are additional details of training parameters for both the supervised IL training and RL fine-tuning. The computing infrastructure for each experiment was on GeForce GTX 1080 Ti GPU. Each experiment took between a few hours to a few days to run.
B.1 DATA PREPROCESSING
From the dataset, which had 6,322 game traces, we extracted 195,405 state-action pairs and 35,90 total instructions. This is done by matching action to the corresponding state within trace as well as the high-level natural language instruction. Each instruction was edited using a spell checker package to reduce the size of the final vocabulary. In the link above, provide the cleaned version of the dataset.
B.2 IL PARAMETERS
Both language generation and language-condittonee policy networks use Cross Entropy Loss and Adam optimizer (learning rate 0.001). In addition, the language loss also includes the addition of doubly stochastic regularization, based on the attention mechanism, and clipping of the gradient norm to max norm of 3. We train for 15-20 epochs. The batch size used during supervised training was 64. By evaluating after each epoch on 100 randomly spawned games, we find that performance plateaus after that number of epochs. As in the RL reward, only consider game to be complete the final craft is completed within the given number of steps. We use the entire dataset for training, since validation/testing were performed randomly generated new games.
16

---

Published as a conference paper at ICLR 2021
Table 6: Instruction sorted by usage frequency.
Grab pickaxe.
Mine cobblestones stash.
Grab the pickaxe.
Pick up the pickaxe.
Open the door.
Go to switch.
Open door.
Go to pickaxe and click grab to take it.
Grab axe.
Go to wood plank and press craft.
Craft wood plank.
Go to axe.
Toggle the switch.
Get the key.
Grab the key.
Mine diamond ore vein.
Mine tree.
Mine cobblestones.
Grab key.
Go to stick and press craft.
Craft stick.
Go pickaxe and select "grab".
Toggle switch to open door.
Make planks. Use key to open door
Grab the axe.
Toggle switch.
Craft diamond pickaxe.
Go to pickaxe.
Go to pickaxe and press grab.
Go to wood bench and craft wood plank.
Craft stone pickaxe.
Grab key to open door.
Unlock the door.
Get the pickaxe.
Make sticks. Go to stone bench and craft stone pickaxe.
Go to tools and click grab to take each one.
Craft leather.
Mine the diamond ore vein.
Go to key grab it go to door and open door.
Go to key.
Go switch and open door with toggle switch.
Go door.
Go to tree.
Mine cobblestone stash. Grab axe to mine tree.
Grab the sword.
Go to stick bench and craft stick.
Pick axe using grab.
Mine the tree.
Get the axe.
Grab sword.
Go to stocks click mine to harvest wood/stone.
Pick up pickaxe using grab.
Craft wood plank with bench and wood. Use the switch to open the door.
Mine rabbit.
Table 7: We compare to related datasets/environmentts (Chevalier-Boisse et al., 2019b Jiang ctt al. 2019 Hu et al. 2019 Anderson et al.) 2018). We don't report dataset size for an environment that generates synthetic language. Note that means limited evaluation, they demonstrate unseen evaluation on one setting only). Most notably, our work focuses on developing a method that per- forms well on unseen tasks. We want clarify that unseen means tasks/environments which the agent has never received supervised reward for. This is not the same as generating new configura- tion of task that the agent received reward for in training.
Dataset
I Language
Dataset Size
Task
Unseen Tasks
BabyAI
Synthetic
Navigation/object placement
No
HAL/CLEVR
Synthetic
Object porting/arrangementt
R2R
Natural
10,800 Views
Vision+language navigation
Yes
MiniRTS
Natural
5.392 Games
Real-time strategy game
No
Ours
Natural
6,322 Games
Compositional crafting tasks
Yes
B.3 RL PARAMETERS
To fine-tune with RL, we first created a gym environment for the Mazebase game, which at reset time will spawn a new Mazebase game in the backend. the parameters the environment, we define the maximum episode steps to be 100. The action space of the environment is Discrete space of size 8 (up, down, left, right, toggle switch, grab, craft, mine) and the observation space is flat vector that concatenates all state observations. For the PPO algorithm, 2.5e4, clip parameter of 0. value loss coefficient of 0.5, 8 processes, 128 steps, size 4 mini- mini-batch, linear learning rate decay, 0.01 entropy coefficient, and 100000000 environment steps.
learning rate of
17

---

Published as a conference paper at ICLR 2021
B.4 ARCHITECTURE DETAILS
B.4.1 STATE ENCODING
As shown in Figure 12 the relevant information about the state that is encoded includes the 5x5 grid, inventory, and goal. We have two representations of the 5x5 grid: one with items relevant for crafting and another with a one-hot representation of coon-crafiiggrelattee items, such as a door, wall, key. All crafting-related items the board, inventory, and goal are embedded using 300-dimension GloVe embedding, summing the embeddings for multiple word items (i.e. Iron Ore Vein). The intuition for this distinction is that for generalization, crafting items should be associated in terms of compositionality, whereas non-crafting items are standalone.
To compute the state encoding, we first passed the grids, inventory, and goal through separate fully connected layers to reduce to the same dimension and concatenated along the vectors. The final size of the state encoding tensor (27, 128), where 25 are for the grid, for inventory, and for goal.
"tan't
Inventory
Oup
One-hot
Boord
Config
Embed
Figure 12: At each time step encode state relevant observations including the goal, inventory, and grid. This encoding is utilized by both the language generator and the language-conditione policy. The boxes in green, denote the observations that were encoded using the GloVe embedding.
B.4.2 IL w/ DISCRIMINATIVE LANGUAGE
Since Hu et al. (2019) is a closely related work to ours, we wanted to compare our method against theirs. Their method only uses behavioral cloning, which is our w/ Generative Language but instead with discriminative language. We modify our high-level language generator to discriminate amongst the most frequent N 500 instructions by adapting code released from Hu al, (2019) of their LSTM-baaed based language model hhere). We plugged in our own state encoding instead of theirs, which anyways tailored to their environment. In summary, the high-level language module is largely the same predicts over the set of possible instructions. We similarly extract the hidden state to condition the low-level policy on. The low-level policy is kept the same as IL w/ Generative Language model for fair comparison. The training parameters are the same as other baselines.
model, both LSTMs, except for the modification of the output layer, which
B.4.3 OUR METHOD
The state encoder, which is used in the high- and low-level model covered in the section above. The size of the hidden layer in the language generator LSTM takes input size 128 and has a hidden size of 32. The size of layers the policy network 48 and 8, and uses ReLU activations.
B.4.4 STATE RECONSTRUCTION
The state reconstruction architecture was instantiated using an autoencoder with 4 hidden layers, taking input the state encoding tensor. The low-dimensional representation after 2 hidden layers used as the hidden state for the policy. The autoencoder trained with MSE loss on the state encoding. This model trained for total of 25 epochs in the II. phase.
18

---

Published as a conference paper at ICLR 2021
B.4.5 STATE PREDICTION
The state prediction architecture largely resembled the language generator. However, removed the GloVe weight embedding layer. In IL training, the dataset was modified to include the past T states. In RL training, the environment was modified record the previous T states. If there were < T states in the current trajectory, the same state is used as input and subsequently replaced. The recurrent network trained with MSE loss on the state encoding. This model trained for total of 20 epochs in the II. phase.
B.4.6 UNI-MODAL INPUT
baseline we considered, but did not included in the main text, to evaluate the necessity of the state encoding in multi-modal datasets. In other works, language instructions are sufficient to solve the task without a state encoding or some representation of the current state. This ablation helps verify that the generated instructions agent with all the information necessary to complete the task in addition to the simulator itself. We considered baseline where the agent only sees language instructions without the state encoding, so the state encoding is used to generate language, but is not provided as additional input to the policy network. This performs poorly and is not able to solve even the simplest 1-step task. We believe this is because representation of the current state is critical to completing the task completion, and is not captured by the high-level instructions.
sufficiently high level that they do provide the
C SUPPLEMENTARY RESULTS
C.I IL RESULTS IN STANDARD SETTING
As shown Table having access natural language instructions only marginally beneficial. While the two environments capture different tasks, we found empirically that the model proposed by Hu et al. [14], which is most similar to our LL+Lang baseline, was able to solve their miniRTS environment using similarly sized dataset to ours, whereas IL+Lang is not sufficient to complete the most difficult tasks our environment.
Table 8: Accuracy of (with and without language) evaluated over 100 games with different seeds.
Steps
IL no language
IL Gen. Language
IL Disc. Language
1-step
18.00% 3.55%
19.33% +\t 1.89%
20.00% + 1.35%
2-step
0.00% 0.00%
9.33% + 2.05%
8.33% 0.98%
3-step
0.00% 0.00%
4.33% + 0.47%
0.00% 0.00%
5-step
0.00% 0.00%
0.00% 0.00%
0.00%: 0.00%
C.2 DEMONSTRATION ONLY AND FEW-SHOT
As shown in Table 9 we find low deviation our multiple variance runs. However, we do observe in some cases such as I[.RI 10% and 100% higher variance because in some trials the model was able to solve any of the 3-step tasks and in others was. In the cases of low variance, either the model was able to consistently or not solve the tasks.
Table Variance results from Table 3 in the main paper, which presents accuracy.
Lung
IL*RL
Steps
5%
10%
100%
5%
10%
100%
5%
10%
100%
5%
10%
100%
5%
10%
100%
5%
10%
100%
1-stap
3%
5%
3%
2%
5%
8%
1%
2%
1%
2%
2-4tep
1%
1%
0%
1%
1%
9%
10%
17%
19%
9%
13%
40%
15%
1%
3%
3-stop
1%
2%
0%
0%
0%
0%
1%
33%
31%
14%
24%
18%
0%
7%
15%
4%
27%
10%
C.3 REWARD ONLY
Finally, for completeness, we consider the scenario where the agent receives a reward but no demon- strations. The tasks which select for this setting sampled from the unseen tasks list. We
19

---

Published as a conference paper at ICLR 2021
choose 3 2-5 step crafts. We evaluate this scenario train using a reward signal. In Table [0] we evaluate on tasks for which we do not have demon- strations and fine-tune trained model with the reward signal for these tasks. This setting is not very interesting from a generalization perspective, since rewards are a far more expensive resource compared to demonstrations and instructions. We don't t include 1-step tasks since that is able to be solved easily by RL alone (see 1-step results in Figure 4). IL and IL w/ Language is not included because this reduces to the zero shot setting.
method against other baselines which
Table 10: Comparison of 2-5 step tasks where only reward is provided to the agent. We believe IL+RL is not able to adapt to these new tasks, given reward only, since it has overfit to the original training tasks. We find that our method outperforms baselines in this setting.
Steps
RL
IL+RL
Ours
2-step
0..00% %
0%
95.93%:10.94%%
3-step
71.67%:10.47%
0%
88.00% 41%
5-step
0.00%=10.000%
0%
65.00%:+5.67%
C.4 INTERPRETABILITY
Table II: Step-by-step discriminated high-level instructions for seen crafts.
Goal: Iron Ore grab the pickaxe mine iron ore mine the iron ore vein unknown
Goal: Gold Ore
unknown
go to the key unknown
go to gold ore vein and mine
Goal: Cobblestone Stairs
take the pickaxe
Goal: Brick Stairs grab key and open door mine bricks unknown
go to cobblestone stash and mine
use pickaxe to mine cobblestone stash
go to cobblestone stash and mine
got to stock and click mine harvest cobblestones
unknown
craft cobblestone stairs
Goal: Iron Ore
Goal: Diamond Boots unknown
toggle switch to open door take the pickaxe
go to pickaxe unknown
toggle switch to open door unknown
We present more examples of generated language for both seen and unseen tasks (Table 12 and Table The tables show a complete set of instructions for tasks which were successfully completed. We observe that if the task was not complete then the language generator would be stuck on particular instruction. The language generated for tasks where the model received supervised data is, as expected, more consistent using the correct language when generating instructions. However, the language generated for tasks which are new to the model also generated instructions which indicated the use of the correct items. We can observe some mistakes such as generating "leather boots" instead of "leather stairs" or generating Pickaxe examples, the model able to generate fairly consistent language for challenging, new 5-step task. Note that in the Gold Pickaxe example, the model was not able to predict the use of item gold so it uses the word stocks. In the Brick Pickaxe example the model correctly inferred the use of brick. These inconsistencies perhaps can be explained by the amount of training examples where brick and gold appeared, with the former being much more frequent than the latter.
unknown token. Particularly for the Gold/Brick
Natural language allows understand why model fails to complete task from simply looking at the generated instructions, which can facilitate future model development and refinement. We identify two failure cases: (1) failure of language generation and (2) failure of policy. In (1),
20

---

Published as a conference paper at ICLR 2021
Table 12: Step-by-step generated high-level instructions for crafts.
Goal: Gold Ore
Goal: Brick Stairs
go key and press grab.
go pickaxe and press grab.
go to pickaxe and grab.
go to the brick factor and mine brick.
go to gold ore vein and mine.
go brick stairs and press craft.
Goal: Diamond Pickaxe
Goal: Wooden Door
go to axe and press grab.
go the axe and grab
go key grab it go door and open door. go to tools and click grab to take each one.
go the switch and open door.
go to the axe and grab
go tree and press mine.
go the tree.
go stocks click mine to harvest.
go to the tree and press mine.
go tree and mine.
go wood plank and press craft.
go wood plank and press craft.
go to wood plank bench and craft wooden door.
go to stick bench and craft stick.
Goal: Leather Helmet
Goal: Diamond Boots go key and press grab. go pickaxe and press grab. go to diamond ore vein and mine. go diamond boots and press craft. go to diamond bench and craft diamond boots.
go to sword and click grab to take it. go key and press grab.
go to sword and click grab to take it.
go rabbit and press mine.
go to leather and press craft. go leather boots bench and craft leather.
Goal: Cobblestone Stairs go to key and press grab. go pickaxe and press grab.
Goal: Iron Ore
go key and press grab. go to pickaxe and press grab. go iron ore vein and press mine.
go to cobblestone stash and press mine. go cobblestone stairs and press craft. Goal: Leather Chestplate
Goal: Wood Stairs
go to axe and press grab.
go sword and press grab.
go tree and mine.
go rabbit and mine.
go to wood plank and press craft. go wood stairs and press craft.
go leather and craft.
go leather chestplate and craft.
Goal: Leather Leggins
Goal: Tron Ingot
go sword and click grab take it.
go key and press grab.
go to rabbit and press mine.
go to pickaxe and press grab, go iron ore vein and mine.
go leather and press craft.
go to leather bench and craft leather
go iron ingot and craft.
observe occasional instructions generated for unseen tasks which have incorrect crafting materials. For example, "Go to iron ingot and press craft" was generated as instruction for Gold Ingot. In (2), while the generated language is sensical for the task, the policy failed to execute correctly.
21

---

Published as a conference paper at ICLR 2021
Table 13: Step-by-step generated high-level instructions for unseen crafts.
Goal: Cobblestone Boots
Goal: Diamond Leggins
go to key and press grab.
go pickaxe and press grab. go to diamond ore vein and mine. go diamond boots and press craft.
go to pickaxe and press grab.
go to cobblestone stash and mine.
go workbench and press craft.
Goal: Leather Stairs
Goal: Stone Helmet
go to sword and press grab.
go to pickaxe and press grab.
go to rabbit and mine the rabbit.
go to the cobblestones stash and mine.
go to leather and press craft.
go the workbench and craft.
Goal: Diamond Ingot go to pickaxe and press grab. go diamond ore vein.
Goal: Brick Door
go to pickaxe and press grab.
go the brick factory and mine the brick. go to the brick stairs and craft.
go to the workbench and craft.
Goal: Brick Pickaxe
Goal: Gold Pickaxe
go to the pickaxe and grab it
go to the pickaxe and press grab. go the axe and grab it.
go to the axe and press grab.
go to the tree.
go to the tree.
go to the tree and mine.
go stocks and click mine harvest <unk>>
go to the brick factory and mine.
go to the tree and mine the tree.
go to the wood plank and craft.
go to wood plank and press craft.
go the stick bench and craft stick. go to stick and craft.
go to stick and press craft.
Goal: Wood Chestplate go to key and grab it. go to axe and grab it.
Goal: Diamond Stairs go to key and press grab. go to pickaxe and press grab. go to the diamond ore vein and mine. go the bench and craft.
go to the tree.
go to tree and mine.
go wood plank and craft.
Example of instruction and inventory side-by-side for unseen tasks. As Figure from the main paper, the inventory changes when subtask, given by the instruction, is completed. Goal: Leather Door
Table 14:
Instruction
Inventory
go to the sword and grab it go to the rabbit and mine go to the leather and press craft
'wordd: 1}
"Sword": 1, 'Rabbit Hide": 1}
'Sword': 1, 'Rabbit Hide': 0, 'Leather': 1}
go to the leather boots bench and craft leather
'woodd:: 1, 'Rabbit Hide': 0, 'Leather': 0,
Leather Door': 1}
Goal: Stone Boots
Instruction
Inventory
go to key and press grab go pickaxe and press grab go to the cobblestone stash and mine the <unk>> go to the bench and craft
("ey"; 1)
("key': 1, 'Pickaxe': 1}
('key': 1, 'Pickaxe': 1, 'Cobblestone': 1}
[key': 1, 'Pickaxe': "Cobblestone': 0,
Stone Boots':
Goal: Diamond Stairs
Instruction go to key and press grab go pickaxe and press grab go the diamond ore vein go to diamond ore vein and mine
Inventory
['ky': 1}
('key": 1, 'Pickaxe:: 13
(key': I, "Pickaxe": 1]
"key": I, "Pickaxe": 1, Diamondd: 1]
(key': I, 'Pickaxe': 'Diamond':
go the bench and craft
Diamond Stairs': 1}
22


Reviews:
[{"review_id": "JNRsWw90Q8x", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of generating natural language instructions to guide a policy to generalize to new environments where reward function is not given but a small set of demonstrations can be provided. A natural language generation (NLG) LSTM network and an initial policy network is trained from a labeled data collected from human workers. Environment state is encoded into a state encoding and LSTM is initialized with this vector to decode the correct sequence of instruction tokens. Policy network consumes both the state encoding and LSTM hidden state to output the correct action. This model is fine-tuned via reinforcement learning (RL) where the NLG component (assuming only the LSTM) is fixed and only policy network is updated (assuming state encoder is also updated from policy network). During testing, the instruction is not given but sampled from the NLG module. Experimental results show that language improves the performance, especially in zero-shot learning setting, compared to baselines where language is not incorporated or used naively in a discriminator network.\n\n1-) My main concern is that the proposed model is very incremental compared to the previous work by (Hu et. al. 19). Similar to the previous work, the authors claim that the main contribution of their work is to show that language is crucial to improve the generalizability of RL models where there are multiple subtasks and long-horizon problem setting. The only difference is using RL to fine-tune the model and using LSTM hidden state as input to the policy rather then explicitly encoding the generated instructions. Please detail the main contributions compared to the previous work.\n\n2-) There are several confusing sentences as to whether human instructions are given during testing or not. This sentence \"If we want to build agents that can quickly adapt in open-world settings, they need to be able to learn from limited, real instruction data.\" reads as to adapt effectively during testing, it is critical to learn from limited but real instruction data. Similarly, this sentence \"We also show that the agent can learn few-shot tasks with only a few additional demos and instructions.\" also reads as it is important to use few additional demos as well as instructions to learn in few-shot learning during testing. Please clarify the paper if and where you use human instructions during testing.\n\n3-) The authors claim that template based NLG has drawbacks as summarized in 2nd paragraph of the introduction. Given that instructions are not given but generated during testing, it is not clear why having realistic natural language would be preferable to a template based approach. Instructions are only used as latent variables that are fed to the policy to improve the overall performance and to have some interpretability. Please discuss why a template based approach lacks these properties and why also they can not outperform realistic human-based instructions given that they can be sampled indefinitely. Also, I can understand that in general natural language is much more complex than a fixed grammar can generate but examples from Table-11 and Table-12 suggest that a fixed grammar can generate these instructions. \n\n4-) The proposed model has some similarities to multi-task learning where the NLG is trained along with the policy network. It is not clear if this is the major source of the performance improvement or the explicit conditioning of the LSTM hidden state on the policy network. A baseline where the same model is used but hidden state of the LSTM is not fed into the policy would help.\n\n\nAdditional questions and comments.\n- Could you compare the results of the proposed model from Table-9 and Figure-5? There is no significant difference that I can see between the two for the \"Ours\" model which suggests that demonstrations are not useful for improved performance.\n- Why is GloVe vectors are removed in state prediction baseline?\n- Could you give results with longer training episodes for \"5 step tasks\" in Figure-5? \n- Could you clarify how much the language diverges from human language when gradient is backpropagated via LSTM? \n- Please add sublabels to Figure-2.\n- In page 5, \"an vector representation\" --> \"a vector representation\"\n- In page 6, \"eesults\" --> \"results\"\n- In page 6, \"essentially\" --> \"essential\"\n- In appendix A.3., \"Table ??\" is missing the ref.\n- In appendix C1, reference [14] should also show the corresponding authors and years.", "extra_scores": {}, "extra_text": {}}, {"review_id": "IwRT9XJZmb0", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use natural language to aid reinforcement learning by generating instructions for sub-goals that allow the agent to complete tasks with delayed rewards. The authors first design a multi-task crafting environment and collect step-by-step human demonstrations along with sub-goal instructions using crowdsourced workers. The proposed method then involves a model architecture that includes a recurrent neural network for generating instructions, and a policy network that produces actions conditioned on the state and the instruction. The model is trained using a combination of imitation learning and reinforcement learning. Experiments reveal that the proposed method outperforms several baselines and also generalizes well to zero-shot unseen tasks (where the goal is to craft new combinations of seen components). \n\nStrengths: Overall, the approach is quite novel and interesting, and the experiments (with several baselines) clearly demonstrate the advantage of incorporating language. Paper is clearly written. \n\nWeakness: One aspect that can improve the paper is some more analysis to tease out the effects of using ‘natural’ language vs variations (see points 7, 8 below).\n\nQuestions for the authors (M = major questions):\n1. (M) From what I understand, the model only uses the current state (grid) as input to generate an instruction at every step. A question I had in my mind while reading was: if so, how does the model know when to switch to the next instruction? I later figured the inventory change is probably the main factor (e.g. after the agent mines some iron ore, it gets added to its inventory), but it would be good to discuss this point a bit in the text since knowing when to switch sub-goals is an important aspect in hierarchical RL.\n2. Related to the above point, have you tried providing a few past frames to the input as well?\n3. Do you provide the recipes (seen in Figure 2) to the agent during the learning phase? Or is this provided only to the human workers who provide the demonstrations?\n4. Thanks for providing details on the data collection in the Appendix. Do you perform any post-processing on the data at all (e.g. have checks to make sure instructions are valid and relevant, correct spelling errors, etc.?)\n5. “Items which are relevant for crafting are embedded using a 300-dimension Glove embedding …. Non-crafting items are represented using a one-hot vector.” Why this design choice? Why not represent everything as a GloVe embedding/one-hot vector? I couldn’t find intuition/explanation for this even in the Appendix.\n6. “... after filtering for words that appeared at least 5 times.” How do you represent the words that are filtered out in the instruction? Do you use a single out-of-vocabulary (OOV) token for all of them?\n7. (M) From the architecture description, the last hidden state of the LSTM is used as input to the policy network, so the generated instruction is not explicitly re-processed by the action generator. While I think this is a neat trick to avoid many issues, the paper leaves open the question of how much of the performance boost is obtained by using ‘natural language’ provided by humans vs just producing a random sequence of tokens. In other words, how much of the performance gain is due to the fact that human language is useful, perhaps because it is (somewhat) compositional? It would be nice to tease out this fact by just generating random instruction sequences for all the train tasks and training the model on them (as a sort of ablation). Note that this is different from the state prediction (SP) baseline which is predicting future states. \n8. (M) Related to above, how does an ‘oracle’ baseline which gets access to the human instructions perform on the test tasks? This can provide an (approximate) upper bound for the proposed model. \n\nMissing references:\n- Branavan et al., Learning to Win by Reading Manuals in a Monte-Carlo Framework\n- Andreas et al., Learning with Latent Language\n- Narasimhan et al., Deep Transfer in Reinforcement Learning by Language Grounding\n- Schwartz et al., Language is Power: Representing States Using Natural Language in Reinforcement Learning\n- Cao et al., BABYAI++: TOWARDS GROUNDED-LANGUAGE LEARNING BEYOND MEMORIZATION\n- Luketina et al., A Survey of Reinforcement Learning Informed by Natural Language", "extra_scores": {}, "extra_text": {}}, {"review_id": "57Ide7u1cGH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies whether a model can generate its language instruction to solve long and complex sequential tasks. Importantly, ground-truth (natural) language instruction is only provided during a pretraining phase. The contributions of the authors are the following:\n - The extension of the grid-world Minecraft-like environment with a natural language dataset of 6000 instructions over 35 tasks\n - A proof of concept that agent may generate their own language instructions, and such process is beneficial when generalizing to new tasks in the zero-shot / few shot learnings\n\nI provide below several remarks:\n\n### Introduction: \nThe introduction is quite standard and correctly define the paper goal and organization. On a personal note, I think that the introduction is even a bit too consensual, and it may lack citations to frame it into the literature. For instance, language is still marginal in the RL literature [1], or the concept goal self-generation with language had a very small literature. \n I would reduce the description of what we want, and detail, what we have achieved so far, and the recurrent pitfalls. It is a bit done in the related work section, but it could also be done in the introduction.\n\n[1] Luketina, Jelena, et al. \"A survey of reinforcement learning informed by natural language.\"IJCAI (2019).\n\n### Related work\n - Although the core related works are mostly cited, I may recommend the authors to draw link with the intrinsic motivation literature with self-goal generation (not intrinsic reward) [2-3]. \n - I also recommend adding this missing reference [4]. Indeed, the intuition and training procedure are very similar. The current paper uses instruction sequence, while [4] only generates a single instruction and tries not to use human instruction.\n - the difference with Hu et al 2019, a close work, are well-justified\n - why not drawing links with \"sketch\" and HRL?\n \n[2] Florensa, Carlos, et al. \"Automatic goal generation for reinforcement learning agents.\" ICML. (2018).\n\n[3] Forestier, Sébastien, et al. \"Intrinsically motivated goal exploration processes with automatic curriculum learning.\" arXiv preprint arXiv:1708.02190 (2017).\n\n[4] Cideron, Geoffrey, et al. \"Self-educated language agent with hindsight experience replay for instruction following.\" ADPRL (2020).\n\n### Human annotation collection\n\n - The data-collection is well justified. I also like the authors' feedback in the Appendix, as it may be useful to other people. Yet, I am missing a small table to compare with other environments or datasets, e.g., MiniRTS, BabyAI, and R2R. (synthetic vs. real, size, can generate new RL scenario, have human dataset, etc.)\n - I am a bit sad that the authors did not create an oracle to generate synthetic instruction on the fly. I acknowledge that it the authors wanted to promote natural language, but it would have able other research analysis (e.g., upper bond by feeding IL+RL with oracle instruction). Besides, it would allow trying different levels of granularity in the instruction. Note that it is not a negative point in the paper, but I think that the paper could be a lot more impactful this way.\n - the analysis of the dataset in the Appendix is neat and appreciated, with multiple examples, and basic statistics\n - Table 3 is not very clear. Would it be possible to turn it into an (evolution) graph?  \n\n\n### Method:\n - The method is pretty simple and standard (but it is well-justified). I found the concept of high-level representation a bit misleading; why not simply refer as language conditioning rather than low/high-level representation. Please also cite the UFVA framework [5]\n - [Key question]: It is unclear when/how the model generates new language instruction? Can you elaborate? \n - I would recommend mentioning that the code is available in this section. (not lost in the Appendix)\n - RL-finetuning: Did you observe some differences in performance/generalization while backpropagating the loss on the language generator? It is a bit frustrating that you mention such experiments without having a small paragraph in the experiments.\n - How important is the use of GloVE in your experiments? This design choice is somehow surprising as the word embedding could have been learned from scratch.\n\n[5] Schaul, Tom, et al. \"Universal value function approximators.\" ICML. (2015).\n\n### Experiments:\n - The baseline 1-5 are sound. Baseline 3 is never correctly described (neither in the main text nor Appendix). Please update the paper accordingly to reproduce it\n - What is the motivation behind baseline 6-7? 6 is an auxiliary loss, and it is quite orthogonal to the approach. I would have replace 7 by a position, instead of the past to have high-level conditioning. Can you explain me why do you think that the past is a relevant high-level signal? Another valid baseline could be simple intrinsic signals (e.g. RIDE [6]), which can be trivially added in light of the current state representation. (Yet, it is also a bit orthogonal)\n - Figure 4.d is quite interesting, and I am curious whether an intrinsic motivation approach would be successful too. \n - Figure 5: What makes the policy worse: lower initial policy or lower language quality? Could you quantify it in some way?\n - Can you analyze further Table1. Saying that your perform better is not fully satisfactory. Why RL collapse in some cases? Why is SP so erratic? What are the successful features that make such a method works when the other one failed? Can you verify it? How does your model handle the cases? Table 1 is very rich, and I would appreciate if the authors analyze it further.\n - I am not sure to have fully understood the few shot learning. Especially, this point \"Additionally we considered a more strict few-shot case where we reduce the\nnumber of demonstrations to 20-40, which is about 5-10% of the original number of demonstrations.\" I thought we already have 5,10,100% of some instructions for pretraining.\n - Interpretability is interesting. I would recommend the authors to have some quantitative results. What is the overlap with human language (Basic BLEU score on held-out data)? What about performing a human evaluation where the human that follow the instruction from the machine and then report the achievement score (100 games would be enough)\n\n[6] Raileanu, Roberta, and Tim Rocktäschel. \"RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments.\" arXiv preprint arXiv:2002.12292 (2020).\n\n### Conclusion:\nOverall, the paper has numerous interests and complement the existing literature nicely. \nThe paper has still some weaknesses such as, \n - some analysis could have been explored further, e.g., better stating hypothesis beyond the experiments, better complement accuracies with other quantitative \n - Information is dispatched in multiple places, and it takes some time to have a global understanding of the experimental protocol and models. I would recommend to better organize/cluster the information, and add better references from the main text to the Appendix and between sections\n - the environment could be enhanced with a language oracle to extend the spectrum of experiments \n\nOn the other side, I believe the paper also has many take-away such as an interesting new dataset, extensive experiments, a valid-research claim with no overstatement, and an open-source code. Therefore, the paper is still marginally above the acceptance threshold.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1vmJ959aC2J", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors present a system that exploits both natural-language instructions, and demonstrations, to learn how to perform multi-subtask tasks in a Minecraft-like environment.  The system has two objectives: First, given a state and objective, learn to generate a natural language description of the high-level subtasks to perform (based on training from human instructions and demonstrations); and second, learn a policy to actually perform the steps.  The policy network accepts both the state description and the final state of the natural language generation network as input at every step.  The system's zero-shot performance is evaluated on previously-unseen tasks with neither demonstrations nor natural-language descriptions provided.\n\nThe performance of the network is compared to a number of baselines, some of which are ablated versions of the proposed system, and some of which resemble systems from the literature with similar goals.\n\nThe basic idea --- predicting the high-level instructions for a goal, and then using the predicted instructions as an input to the low-level policy --- is a very creative solution to the problems of hierarchical planning, and seems to be effective.  The additional interpretability benefits are very valuable in their own right.  Overall, I think this technique is likely to have a significant impact on future work.\n\nI have some reservations about the clarity of the writing, and the breadth of the empirical comparisons:\n\nThe writing is at times difficult to follow, in ways that I think could be straightforwardly addressed, e.g.:\n- p.5/6: I found it hard to understand what the difference was between baseline 2 (IL + Generative Language) and baseline 3 (IL + Discriminative Language)\n- p.6: \"The model stores the past T states as input to predict the T+1 state, where (T=3).\" The letter T is probably being incorrectly reused here; maybe it's something like \"at time T, the states T-2, T-1, and T are used to predict state T+1\", but I can't be sure.\n\nI'm also a little concerned about whether the right baselines were used in the empirical evaluation.  The \"ablated\" versions of the proposed system are definitely valuable for demonstrating that all of the proposed components are necessary, but I am left wondering how the system's performance compares to the state of the art.  There appears to be only one baseline (#3) that directly corresponds to related work in the literature.  I would appreciate some more direct comparisons to what can already be accomplished in this domain.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although some reviewers still had concerns about the novelty of the proposed method, most of the other concerns have been addressed in a satisfying manner according to reviewers. They globally have a positive opinion about the paper after revision. 
Decision: Accept (Poster)