ASK YOUR HUMANS: USING HUMAN INSTRUCTIONS TO IMPROVE GENERALIZATION IN REINFORCEMENT LEARNING
Year: 2021

Abstract:
Complex, multi-task problems have proven to be difficult to solve efficiently in a sparse-reward reinforcement learning setting. In order to be sample efficient, multi-task learning requires reuse and sharing of low-level policies. To facilitate the automatic decomposition of hierarchical tasks, we propose the use of step-bystep human demonstrations in the form of natural language instructions and action trajectories. We introduce a dataset of such demonstrations in a crafting-based grid world. Our model consists of a high-level language generator and low-level policy, conditioned on language. We find that human demonstrations help solve the most complex tasks. We also find that incorporating natural language allows the model to generalize to unseen tasks in a zero-shot setting and to learn quickly from a few demonstrations. Generalization is not only reflected in the actions of the agent, but also in the generated natural language instructions in unseen tasks. Our approach also gives our trained agent interpretable behaviors because it is able to generate a sequence of high-level descriptions of its actions.

heading: 1 INTRODUCTION
text: One of the most remarkable aspects of human intelligence is the ability to quickly adapt to new tasks and environments. From a young age, children are able to acquire new skills and solve new tasks through imitation and instruction (Council et al., 2000; Meltzoff, 1988; Hunt, 1965). The key is our ability to use language to learn abstract concepts and then reapply them in new settings. Inspired by this, one of the long term goals in AI is to build agents that can learn to accomplish new tasks and goals in an open-world setting using just a few examples or few instructions from humans. For example, if we had a health-care assistant robot, we might want to teach it how to bring us our favorite drink or make us a meal in just the way we like it, perhaps by showing it how to do this a few times and explaining the steps involved. However, the ability to adapt to new environments and tasks remains a distant dream.
Previous work have considered using language as a high-level representation for RL (Andreas et al., 2017; Jiang et al., 2019). However, these approaches typically use language generated from templates that are hard-coded into the simulators the agents are tested in, allowing the agents to receive virtually unlimited training data to learn language abstractions. But both ideally and practically, instructions are a limited resource. If we want to build agents that can quickly adapt in open-world settings, they need to be able to learn from limited, real instruction data (Luketina et al., 2019). And unlike the clean ontologies generated in these previous approaches, human language is noisy and diverse; there are many ways to say the same thing. Approaches that aim to learn new tasks from humans must be able to use human-generated instructions.
In this work, we take a step towards agents that can learn from limited human instruction and demonstration by collecting a new dataset with natural language annotated tasks and corresponding gameplay. The environment and dataset is designed to directly test multi-task and sub-task learning, as it consists of nearly 50 diverse crafting tasks.1 Crafts are designed to share similar features and
1Our dataset, environment, and code can be found at: https://github.com/valeriechen/ask-your-humans.
sub-steps so we would be able to test whether the method is able to learn these shared features and reuse existing knowledge to solve new, but related tasks more efficiently. Our dataset is collected in a crafting-based environment and contains over 6,000 game traces on 14 unique crafting tasks which serve as the training set. The other 35 crafting tasks will act as zero-shot tasks. The goal is for an agent to be able to learn one policy that is able to solve both tasks it was trained on as well as a variety of unseen tasks which contain similar sub-tasks as the training tasks.
To do this, we train a neural network system to generate natural language instructions as a highlevel representation of the sub-task, and then a policy to achieve the goal condition given these instructions. Figure 1 shows how our agent takes in the given state of the environment and a goal (Iron Ore), generates a language representation of the next instruction, and then uses the policy to select an action conditioned on the language representation - in this case to grab the key. We incorporate both imitation learning (IL) using both the language and human demonstrations and Reinforcement Learning (RL) rewards to train our agent to solve complicated multi-step tasks.
Our approach which learns from human demonstrations and language outperforms or matches baseline methods in the standard RL setting. We demonstrate that language can be used to better generalize to new tasks without reward signals and outperforms baselines on average over 35 zero-shot crafting tasks. Our method uses language as a high-level task to help decompose a larger, complex task into sub-tasks and identify correct sub-tasks to utilize in a zero-shot task setting. We also show that the agent can learn few-shot tasks with only a few additional demos and instructions. Finally, training with human-generated instructions gives us an interpretable explanation of the agent’s behavior in cases of success and failure. Generalization is further demonstrated in the agent’s ability to explain how the task is decomposed in both train and evaluation settings, in a way that reflects the actual recipes that describe the crafting task. With our dataset collection procedure and languageconditioned method, we demonstrate that using natural human language can be practically applied to solving difficult RL problems and begin solving the generalization problem in RL. We hope that this will inspire future work that incorporates human annotation, specifically language annotation, to solve more difficult and diverse tasks.

heading: 2 RELATED WORK
text: Previous works on language descriptions of tasks and sub-tasks have generally relied on what Andreas et al. (2017) calls “sketches.” A sketch specifies the necessary sub-tasks for a final task and is manually constructed for every task. The agent then relies on reward signals from the sketches in order to learn these predefined sub-tasks. However, in our setup, we want to infer such “sketches” from a limited number of instructions given by human demonstrations. This setting is not only more difficult but also more realistic for practical applications of RL where we might not have a predefined ontology and simulator, just a limited number of human-generated instructions. In addition, at test time, their true zero-shot task requires the sketch, whereas our method is able to generate the “sketches” in the form of high-level language with no additional training and supervision.
Similarly, other works have used synthetically generated sub-goals and descriptions to train their methods and suffer from similar problems of impracticality. Shu et al. (2018) introduces a Stochastic Temporal Grammar to enable interpretable multi-task RL in the Minecraft environment. Similarly, the BabyAI platform Chevalier-Boisvert et al. (2019a) presents a synthetic language which models commands inside a grid-based environment. They utilize curriculum training to approach learning complex skills and demonstrate through experimentation in their environment that existing approaches of pure IL or pure RL are extremely sample inefficient. Cideron et al. (2019) extend Hindsight Experience Replay (HER) to language goals in the BabyAI platform to solve a single instruction generated from a hand-crafted language. The BabyAI environment is extended by Cao et al. (2020) to include descriptive texts of the environment to improve the generalization of RL agents. Jiang et al. (2019) also uses procedural generated language using the MuJoCo physics engine and the CLEVR engine to learn a hierarchical representation for multi-task RL. Oh et al. (2017) also tackles zero-shot generalizations, but like the others considers only procedurally-generated instructions, learning to use analogies to learn correspondences between similar sub-tasks.
The main work that also investigates using a limited number of human-generated instructions in RL environments is Hu et al. (2019). This paper also uses natural language instructions in hierarchical decision making to play a real-time strategy game involving moving troop units across long time scales. This work uses only behavioral cloning with natural language instructions, whereas we use a mixture of RL and imitation learning. They also do not investigate the benefits of language in zero-shot or few-shot settings and do not demonstrate cross-task generalization as we do.
Hierarchical approaches as a way of learning abstractions is well studied in the Hierarchical Reinforcement Learning (HRL) literature (Dayan & Hinton, 1993; Parr & Russell, 1998; Stolle & Precup, 2002). This is typically done by predefining the low-level policies by hand, by using some proxy reward to learn a diverse set of useful low-level policies (Heess et al., 2016; Florensa et al., 2017; Eysenbach et al., 2018; Hausman et al., 2018; Marino et al., 2019), or more generally learning options (Sutton et al., 1999). Our approach differs in that unlike in options and other frameworks, we generate language as a high-level state which conditions the agent’s policy rather than handing control over to low-level policies directly.
Other works have shown the effectiveness of using a combination of reinforcement learning and imitation learning. Le et al. (2018) presents a hybrid hierarchical reinforcement learning and imitation learning algorithm for the game Montezuma’s revenge by leveraging IL for the high-level controller and RL for the low-level controller demonstrating the potential for combining IL and RL to achieve the benefits of both algorithms. By learning meta-actions, the agent is able to learn to solve the complex game. However, their meta-actions were also hand specified.
Others have utilized natural language for other tasks, including Williams et al. (2018), Co-Reyes et al. (2018), and Andreas et al. (2018) but not focused on the multi-task learning setting. Matthews et al. (2019) demonstrates the use of word embeddings to inform robotic motor control as evidence of particular promise for exploiting the relationship between language and control. Narasimhan et al. (2018) uses language descriptions of the environment to aid domain transfer. The sub-field of language and vision navigation specifically has investigated how to train agents to navigate to a particular location in an environment given templated or natural language (Chaplot et al., 2018; Anderson et al., 2018; Tellex et al., 2011; Mei et al., 2016; Chen & Mooney, 2011; Yu et al., 2018) or to navigate to a particular location to answer a question Das et al. (2018). Similarly to this work, Nguyen et al. (2019) uses human-generated language to find objects in a simulated environment, Zhong et al. (2020) and Branavan et al. (2012) reads a document (i.e. a players manual) to play a variety of games, and Lynch & Sermanet (2020) trains agents to follow both image and languagebased goals. All of these works require the agent to read some text at both train and test time and follow those instructions to achieve some goal. In contrast, at test time, our agent only receives a high-level goal, which is what item to craft. Our agent must take the high-level goal as input and generates its own instructions to solve the task. In other words, our task is both instruction following and instruction generation. Related to instruction generation, some work have explored more generally intrinsic motivation for goal generation (Florensa et al., 2018; Forestier et al., 2017). In our work, however, we learn the goals via the human language instructions.

heading: 3 HUMAN ANNOTATION COLLECTION
text: The first step of our approach requires human demonstrations and instructions. To that requirement, we built an interface to collect human-annotated data to guide the learning model.
Crafting Task: The goal of the agent in our world is to complete crafts. By design, a crafting-based world allows for complexity and hierarchy in how the agent interacts with items in the gridworld. To craft an item, the agent must generally first pick up a tool, go to a resource, mine the resource, and then go to a table to craft the item. To make an iron ore, the agent must use the pickaxe at the Iron Ore Vein to mine Iron Ore to complete the task. The Iron Ore recipe is an example of a 1-step task because it creates one item. A 5-step task, like Diamond Pickaxe, involves the mining and/or crafting of 5 items. We capped the tasks at a maximum length of 5 recipe steps to limit the amount of time a worker would have to spend on the task. Note that each recipe step requires multiple time-steps to complete. Crafts are designed to share similar features and sub-steps to test whether the agent is able to learn these shared features and reuse existing knowledge to solve new, but related tasks more efficiently (these relations between tasks are detailed in Table 3 and in Figure 10). While the task may seem simple to human annotators to solve, such compositional tasks still pose difficulties for sparse-reward RL. We further increase the difficulty of this task by restricting the agent to a limited number of steps (100) to complete the task, leaving little room to make unrecoverable mistakes such as spending time collecting or using unnecessary resources.
Data Collection Process: Figure 2 shows our interface. Given the goal craft, relevant recipes, and the initial board configuration, the worker provides step-by-step instructions accompanied by execution on the actual game board of each instruction. The workflow would be to type one instruction, execute the instruction, then type the next instruction, and execute until the goal was completed. The data collection interface and a corresponding example set of natural language instructions provided by a Turker is illustrated on the rightmost side of Figure 2. This is but one way that a Turker might choose to break down the 5-step crafting task. The appendix has more details on the collection process in Section A.1. We will release the environment and dataset.
Dataset Analysis: Between the 14 crafts, we collected 6,322 games on AMT. In total, this dataset contains 195,405 state-action pairs and 35,901 total instructions. In the supplementary material we present relevant summary statistics about the data, including the number of instructions provided for each n-step task. The number of instructions, and consequently, actions required increases with the number steps as shown in Table 4.

heading: 4 METHODS
text: Our proposed approach to solving these multi-step crafting tasks is to learn from human-generated natural language instructions and demonstrations. The model is first pre-trained using imitation learning (IL) and then fine-tuned using sparse-reward in reinforcement learning (RL). The goal of the agent is to learn one policy that is able to solve a variety of tasks (around 50) in the environment including ones it has not seen when only trained on a subset of the total tasks.
Architecture: As outlined in Figure 3, we factor the agent into a hierarchical set-up with a language generator at the high-level and policy conditioned on the language at the low-level. At each time step, the state encoder produces a vector representation that is then used as input to both the language generator and language-conditioned policy. Relevant information about the state, including the grid, inventory, and goal are encoded. Items which are relevant for crafting are embedded using a 300- dimension GloVe embedding, summing the embeddings for multiple word items (i.e. Iron Ore Vein). Non-crafting items, such as door, wall, or key, are represented using a one-hot vector. Further details are provided in Section B.
Imitation Learning Pre-training: We warm-start the model using the human demonstrations. Language is generated at the high-level with an encoder-decoder framework. The encoding from the state encoder is decoded by an LSTM which generates a natural language instruction. The target language instruction is the AMT worker’s provided instruction. In our dataset, the vocabulary size was 212, after filtering for words that appeared at least 5 times. At test time, we do not have access to the ground truth instructions, so instead the LSTM decoder feeds back the previously generated word as the next input and terminates when the stop token is generated. From the language generator module, we extract the last hidden state of the generated instruction. The hidden state is concatenated with the encoded state and passed through a series of fully connected layers. The final layer outputs the action. In the supervised training phase, the full model is trained by backpropagating through a language and action loss (cross entropy loss).
Reinforcement Learning Fine-tuning: We use the proximal policy optimization (PPO) algorithm Schulman et al. (2017) in RL with the reward defined below to learn an optimal policy to map from state encoding to output action. The maximum number of steps in an episode is set to 100. We utilize a training set-up which samples from all tasks (1-step through 5-step tasks). In preliminary experiments, we observe that sampling from 3-step tasks alone, for example, poses too complex of an exploration problem for the model to receive any reward. We define a sparse reward, where the agent only receives a reward when it has completed the full craft. In RL fine-tuning, we freeze the language generator component because there is no more language supervision provided in the simulated environment. We also find that empirically backpropagating the loss through language distorts the output language as there is no constraint for it to continue to be similar to human language. All training hyperparameters and details are provided in supplementary materials.

heading: 5 EXPERIMENTS
text: We compare our method against five baselines (1-5) which are reduced forms of our method to evaluate the necessity of each component. We also consider two baselines (6-7), which swap out the language generator for alternative high-level tasks, to evaluate the usefulness of language as a selected high-level task. These baselines have the additional training that our method received, as well as the implicit compositionality, but without language. In both baselines (6-7), we perform the same training steps as with our method. Implementation details are presented in Section B.
1. IL: The IL baseline uses the same low-level architecture as our method, without a high-level hidden state. The model learns to map state encoding to an output action.
2. IL w/ Generative Language: IL w/ Generative Language is the supervised baseline of our method, which does not include RL reward. This baseline allows us to observe and compare the benefit of having a reward to train in simulation when the model has access to both actions and language instructions.
3. IL w/ Discriminative Language: We compare our method to a closely adapted version of the method proposed in Hu et al. (2019) which similarly uses language in the high-level. Rather than generate language, their high-level language is selected from a set of instructions from the collected user annotation. We discuss this adaptation in the Appendix. They consider instruction sets of sizes N = {50, 250, 500} and find the best performance on the largest instruction set N = 500 which is the size we use in our implementation.
4. RL: Another baseline we consider is the reinforcement learning (RL) setting where the agent is provided no demonstrations but has access to sparse-reward in RL. The architecture we use here is the same as the IL architecture. This baseline demonstrates the capacity to learn the crafting tasks without any human demonstrations and allows us to see whether human demonstrations are useful.
5. IL + RL: We also consider a baseline that does not incorporate language which is IL+RL. In IL+RL, we pretrain the same IL architecture using the human demonstrations as a warm-start to RL. It is important to note that this baseline does not include the natural language instructions as a part of training. We extract all of the state-action pairs at train a supervised model on the data as in the IL model and then we utilize the RL sparse-reward to fine-tune.
6. State Reconstruction (SR): We train an autoencoder to perform state reconstruction. The autoencoder reconstructs the state encoding and the vector at the bottleneck of the autoencoder is used as the hidden layer for the policy. SR as a baseline allows us to consider latent representations in the state encoding as a signal for the policy.
7. State Prediction (SP): We train a recurrent network, with the same architecture as our language generator, to perform state prediction. The model stores the past 3 states from time T to predict the T + 1 state. So at time T , the states T − 2, T − 1, and T are used to predict state T + 1. From the LSTM, the hidden state is extracted in the same manner as our IL + RL w/ Lang model. SP as a baseline allows us to compare against another recurrent high-level method with the additional computation power.

heading: 5.1 RESULTS
text: Standard setting: We evaluate the various methods on crafts which we have collected human demonstrations for to benchmark comparative performance in our environment. An initial analysis is to first consider how much the IL model is able to learn from human demonstrations alone, so we consider IL, IL with Generative Language, and IL with Discriminative Language (results are in Section C). None of these approaches are able to solve the most difficult 5-step tasks or the simpler tasks consistently, with an average of about 18-19% success rate for 1-step tasks. We believe the 3 and 5-step tasks are difficult enough such that annotations alone were not able to capture the diversity of board configurations for the variety of crafts given that the board is randomly initialized each time. However, based on an analysis of the language selected (see Tables 11 vs. Table 12), the generated language is more interpretable and made more sense in a zero shot setting. Given the language is fixed after this point, all remaining experiments moving forward use generative language.
As shown in Figure 4, our method performs well against baselines. We find that human demonstrations are necessary to guide learning because the learned behavior for RL is essential to arbitrarily
walk around the grid and interact with items. For simple 1 and 2 step tasks, this is a feasible strategy for the allotted steps for an episode. However, there is little room for error in the most difficult 5-step tasks, as even human demonstrations take on average 40 steps to solve. We also find that for the standard setting, incorporating a high-level network allows the model to achieve good results when comparing our method to SP and SR.
In Figure 5 we show the result of our method when we ablate the number of demonstrations we use. This lets us see how many demonstrations we would feasibly need for the model to learn how to solve the crafting tasks. As we decrease the amount of data provided, we find that there is greater variance in the policy’s ability to complete the task, but the performance only significantly degrades when we start using only 25% of the data on the hardest tasks.
Zero Shot: Our method is able to use natural language instructions to improve performance on difficult tasks in the standard setting. But how well is our method able to do on completely new tasks not seen during training? We investigate our performance on zero-shot tasks, where the agent receives no human demonstrations or instructions, and no rewards on these tasks. The agent has to try to complete these tasks that it has never seen before and cannot train on at all. These unseen tasks do share sub-task structure with tasks which were seen in the training process, so the desired behavior is for the model to reuse subpolicies seen in other contexts for this new context. For example, in training the agent might have seen demonstrations or received rewards for a task like “Cobblestone Stairs” and “Iron Ingot.” At test time, we can evaluate the agent on an item like “Cobblestone Ingot”, which has never been seen by the agent. The agent should be able to infer the sub-task breakdown given prior knowledge of similar tasks.
We present 35 examples of unseen tasks in Table 1. We find that overall our method outperforms all other baselines. While SR and SP were able to match our method’s performance in standard setting, they are not able to generalize. SR and SP are viable solutions to learn complex tasks in the standard RL setting, but the representations these models learned do not aid in generalizing to unseen tasks. Here, we believe, using language is key because it creates a representation that better abstracts to new tasks. In the supplementary material, we show that in the cases of unseen tasks, the model indeed is able to generate language that properly corresponds to these new combinations of materials and items, particularly decomposing the complex item into sub-tasks that were previously seen in the training phase.
Demonstration Only and Few-Shot: In the demonstration only, we assume that we have access to only human demonstrations for some subset of tasks. From the entire pool of 14 tasks we collected demonstrations for, we withhold 3 tasks (around 20% of total tasks) for testing. These 3 tasks consist of a one, two, and three step task. We run results on 3 permutations of withholding 3
tasks. For each of the 3 withheld tasks, we include these demonstrations in the supervised training phase but do not provide reward in RL fine-tuning. We vary the amount of demonstrations that are provided: 5%, 10%, and 100%. The most generous case is to assume that the model has access to all demonstrations that were collected in the dataset. Per task, the total number of demonstrations was about 300-500. Additionally we considered a more strict few-shot case where we reduce the number of demonstrations to 20-40 which is about 5-10% of the original number of demonstrations. We do not include 5-step tasks because we only collected demonstrations for two 5-step tasks. From the results in Table 2, we can see that our method outperforms baselines in its ability to utilize the few demonstrations to improve performance.
Interpretability: One key benefit of incorporating natural language into the model is the ability for humans to interpret how the model is making decisions. We observe that the generated instructions closely match those of the recipes that we provide to the annotators in the data collection phase in both train (Table 12) and test (Table 13, 14) settings. However, the discriminative language didn’t break down the task into steps that made sense (Table 11). Figure 6 presents example instructions generated by our model.

heading: 6 CONCLUSION
text: In this paper, we present a dataset of human demonstrations and natural language instructions to solve hierarchical tasks in a crafting-based world. We also describe a hierarchical model to enable efficient learning from this data through a combined supervised and reinforcement learning approach. In general, we find that leveraging human demonstrations allows the model to drastically outperform RL baselines. Additionally, our results demonstrate that natural language not only allows the model to explain its decisions but it also improves the model’s performance on the most difficult crafting tasks and further allows generalization to unseen tasks. We also demonstrate the model’s ability to expand its skillset through few additional human demonstrations. While we demonstrate our approach’s success in a grid-based crafting environment, we believe that our method is able to be adapted towards generalizable, multi-task learning in a variety of other environments.

heading: A DATASET

heading: A.1 COLLECTION PROCESS
text: In this section, we provide additional details for our data collection process on Amazon Mechanical Turk (AMT). Firstly, we filter workers by the following criteria: a) HIT Approval % of greater than 95, b) Location is in an English speaking country, c) soft block has not been granted. These criteria help to ensure the quality of collected data, particularly in terms of natural language.
We collected the dataset over the course of a few weeks. For each HIT, we paid the Turker $0.65. On average the task took about 3-4 minutes. For each HIT, we generate a unique entrance code that is provided to the Turker on the AMT website. The Turker is also provided with a unique exit code once the HIT is complete. Given the entrance and exit code we are able to pay workers accordingly for their demonstrations.
Workers were provided with an entrance code at the beginning of the task to enter the website and an exit code when they completed the task to be able to submit the HIT. This enforces that we do not have workers doing extra HITs that we are unable to pay for and to ensure that worker who submit HITs have indeed completed our task. Then we also wrote a parsing script to be able to quickly verify all submitted HITs to the task before payment. Specifically, we used this script to manually review the list of instructions generated for each task to ensure that the instructions provided were indeed pertinent to the task at hand.
We had many returning workers who completed our task. A few even wrote emails to our requesting email to let us know that they really enjoyed the HIT and thought it was quite interesting to work on. This suggests that collecting demonstrations of this kind is relatively interesting for humans to provide.

heading: A.2 HIT INSTRUCTIONS
text: Prior to starting the task, each worker was provided with this instructions page which gives an analogy for cooking stir-fry as an analogy for the types of instructions that we believe they would provide. The demo page is shown in Figure 8.
Invoking users to provide a particular level of specificity was difficult to convey without explicitly providing example instructions. We deliberately chose not to provide examples as to not prime the worker to a particular format to follow. New workers were given two short games to complete to familiarize themselves with the environment. Returning workers were given one longer game to complete as they already had experience with the task. Workers who completed the task as previously described were fully compensated.
Originally, some workers provided not enough instructions, meaning that they wanted to finish the task as quickly as possible, and other workers provided instructions that were too granular, meaning that they did not abstract the task into sub-tasks and rather wrote “press left” or “go up 1” as their instruction. Prior to approving the HIT, checked prior to the worker submitting the HIT and built in precautions so that they had to redo a level if they did not comply with such instructions clearly delineated in the demo.

heading: A.3 ADDITIONAL ENVIRONMENT AND CRAFTING TASK DETAILS
text: Figure 9 gives an example of the type of task (Make Iron Ore) that would be presented to the worker, which includes the goal, recipes, and the current board. Table 3 shows how the tasks are related in terms of sub-tasks. This might be because of a similar material (i.e. Iron) or a similar craft (i.e. Stairs).

heading: A.4 EXAMPLE DATA COLLECTED
text: Table 5 gives examples of instructions randomly sampled from our dataset. Even with a limited number of crafts, we were able to collect language instructions with diversity in sentence construction. There are inconsistencies in capitalization and spelling which we handled in the preprocessing of the data. Table 6 shows the most frequently used instructions. Figure 11 gives summary statistics of the instruction side of the dataset.

heading: B METHODS DETAILS
text: Given the dataset of human demonstrations, we convert traces of each game into state-action pairs for training. In the subsequent sections are additional details of training parameters for both the supervised IL training and RL fine-tuning. The computing infrastructure for each experiment was on 1 GeForce GTX 1080 Ti GPU. Each experiment took between a few hours to a few days to run.

heading: B.1 DATA PREPROCESSING
text: From the dataset, which had 6,322 game traces, we extracted 195,405 state-action pairs and 35,901 total instructions. This is done by matching an action to the corresponding state within a trace as well as the high-level natural language instruction. Each instruction was edited using a spell checker package to reduce the size of the final vocabulary. In the link above, we provide the cleaned version of the dataset.
B.2 IL PARAMETERS
Both language generation and language-conditioned policy networks use Cross Entropy Loss and Adam optimizer (learning rate 0.001). In addition, the language loss also includes the addition of doubly stochastic regularization, based on the attention mechanism, and clipping of the gradient norm to a max norm of 3. We train for 15-20 epochs. The batch size used during supervised training was 64. By evaluating after each epoch on 100 randomly spawned games, we find that performance plateaus after that number of epochs. As in the RL reward, we only consider a game to be complete if the final craft is completed within the given number of steps. We use the entire dataset for training, since validation/testing were performed on randomly generated new games.

heading: B.3 RL PARAMETERS
text: To fine-tune with RL, we first created a gym environment for the Mazebase game, which at reset time will spawn a new Mazebase game in the backend. In the parameters of the environment, we define the maximum episode steps to be 100. The action space of the environment is Discrete space of size 8 (up, down, left, right, toggle switch, grab, craft, mine) and the observation space is a flat vector that concatenates all state observations. For the PPO algorithm, we use a learning rate of 2.5e4, clip parameter of 0.1, value loss coefficient of 0.5, 8 processes, 128 steps, size 4 mini-batch, linear learning rate decay, 0.01 entropy coefficient, and 100000000 environment steps.

heading: B.4 ARCHITECTURE DETAILS

heading: B.4.1 STATE ENCODING
text: As shown in Figure 12, the relevant information about the state that is encoded includes the 5x5 grid, inventory, and goal. We have two representations of the 5x5 grid: one with items relevant for crafting and another with a one-hot representation of non-crafting-related items, such as a door, wall, or key. All crafting-related items on the board, inventory, and goal are embedded using a 300-dimension GloVe embedding, summing the embeddings for multiple word items (i.e. Iron Ore Vein). The intuition for this distinction is that for generalization, crafting items should be associated in terms of compositionality, whereas non-crafting items are standalone.
To compute the state encoding, we first passed the two grids, inventory, and goal through separate fully connected layers to reduce to the same dimension and concatenated along the vectors. The final size of the state encoding tensor is (27, 128), where 25 are for the grid, 1 for inventory, and 1 for goal.
B.4.2 IL W/ DISCRIMINATIVE LANGUAGE
Since Hu et al. (2019) is a closely related work to ours, we wanted to compare our method against theirs. Their method only uses behavioral cloning, which is our IL w/ Generative Language but instead with discriminative language. We modify our high-level language generator to discriminate amongst the most frequent N = 500 instructions by adapting code released from Hu et al. (2019) of their LSTM-based language model here). We plugged in our own state encoding instead of theirs, which is anyways tailored to their environment. In summary, the high-level language module is largely the same as our model, both LSTMs, except for the modification of the output layer, which predicts over the set of possible instructions. We similarly extract the hidden state to condition the low-level policy on. The low-level policy is kept the same as our IL w/ Generative Language model for fair comparison. The training parameters are the same as other baselines.

heading: B.4.3 OUR METHOD
text: The state encoder, which is used in the high- and low-level model is covered in the section above. The size of the hidden layer in the language generator LSTM takes input size 128 and has a hidden size of 32. The size of layers in the policy network is 48 and 8, and uses ReLU activations.

heading: B.4.4 STATE RECONSTRUCTION
text: The state reconstruction architecture was instantiated using an autoencoder with 4 hidden layers, taking as input the state encoding tensor. The low-dimensional representation after 2 hidden layers is used as the hidden state for the policy. The autoencoder is trained with MSE loss on the state encoding. This model trained for a total of 25 epochs in the IL phase.

heading: B.4.5 STATE PREDICTION
text: The state prediction architecture largely resembled the language generator. However, we removed the GloVe weight embedding layer. In IL training, the dataset was modified to include the past T states. In RL training, the environment was modified to record the previous T states. If there were < T states in the current trajectory, the same state is used as input and subsequently replaced. The recurrent network is trained with MSE loss on the state encoding. This model trained for a total of 20 epochs in the IL phase.

heading: B.4.6 UNI-MODAL INPUT
text: A baseline we considered, but did not included in the main text, is to evaluate the necessity of the state encoding in multi-modal datasets. In other works, language instructions are sufficient to solve the task without a state encoding or some representation of the current state. This ablation helps verify that the generated instructions are sufficiently high level that they do not provide the agent with all the information necessary to complete the task in addition to the simulator itself. We considered a baseline where the agent only sees language instructions without the state encoding, so the state encoding is used to generate language, but is not provided as additional input to the policy network. This performs poorly and is not able to solve even the simplest 1-step task. We believe this is because a representation of the current state is critical to completing the task completion, and is not captured by the high-level instructions.

heading: C SUPPLEMENTARY RESULTS
text: C.1 IL RESULTS IN STANDARD SETTING
As shown in Table 8, having access to natural language instructions is only marginally beneficial. While the two environments capture different tasks, we found empirically that the model proposed by Hu et al. [14], which is most similar to our IL+Lang baseline, was able to solve their miniRTS environment using a similarly sized dataset to ours, whereas IL+Lang is not sufficient to complete the most difficult tasks in our environment.

heading: C.2 DEMONSTRATION ONLY AND FEW-SHOT
text: As shown in Table 9, we find low deviation in our multiple variance runs. However, we do observe in some cases such as IL+RL 10% and 100% higher variance because in some trials the model was not able to solve any of the 3-step tasks and in others it was. In the cases of low variance, either the model was able to consistently or not solve the tasks.

heading: C.3 REWARD ONLY
text: Finally, for completeness, we consider the scenario where the agent receives a reward but no demonstrations. The tasks which we select for this setting are sampled from the unseen tasks list. We
choose 3 2-5 step crafts. We evaluate this scenario on our method against other baselines which train using a reward signal. In Table 10, we evaluate on tasks for which we do not have demonstrations and fine-tune a trained model with the reward signal for these tasks. This setting is not very interesting from a generalization perspective, since rewards are a far more expensive resource compared to demonstrations and instructions. We don’t include 1-step tasks since that is able to be solved easily by RL alone (see 1-step results in Figure 4). IL and IL w/ Language is not included because this reduces to the zero shot setting.
We present more examples of generated language for both seen and unseen tasks (Table 12 and Table 13). The tables show a complete set of instructions for tasks which were successfully completed. We observe that if the task was not complete then the language generator would be stuck on a particular instruction. The language generated for tasks where the model received supervised data is, as expected, more consistent in using the correct language when generating instructions. However, the language generated for tasks which are new to the model also generated instructions which indicated the use of the correct items. We can observe some mistakes such as generating “leather boots” instead of “leather stairs” or generating an unknown token. Particularly for the Gold/Brick Pickaxe examples, the model is able to generate fairly consistent language for a challenging, new 5-step task. Note that in the Gold Pickaxe example, the model was not able to predict the use of item gold so it uses the word stocks. In the Brick Pickaxe example the model correctly inferred the use of brick. These inconsistencies perhaps can be explained by the amount of training examples where brick and gold appeared, with the former being much more frequent than the latter.
Natural language allows us to understand why a model fails to complete a task from simply looking at the generated instructions, which can facilitate future model development and refinement. We identify two failure cases: (1) failure of language generation and (2) failure of policy. In (1), we
observe occasional instructions generated for unseen tasks which have incorrect crafting materials. For example, “Go to iron ingot and press craft” was generated as an instruction for Gold Ingot. In (2), while the generated language is sensical for the task, the policy failed to execute correctly.

Review summaries:
In this paper, the authors present a system that exploits both natural-language instructions, and demonstrations, to learn how to perform multi-subtask tasks in a Minecraft-like environment.  The system has two objectives: First, given a state and objective, learn to generate a natural language description of the high-level subtasks to perform (based on training from human instructions and demonstrations); and second, learn a policy to actually perform the steps.  The policy network accepts both the state description and the final state of the natural language generation network as input at every step.  The system's zero-shot performance is evaluated on previously-unseen tasks with neither demonstrations nor natural-language descriptions provided.

This paper proposes to use natural language to aid reinforcement learning by generating instructions for sub-goals that allow the agent to complete tasks with delayed rewards. The authors first design a multi-task crafting environment and collect step-by-step human demonstrations along with sub-goal instructions using crowdsourced workers. The proposed method then involves a model architecture that includes a recurrent neural network for generating instructions, and a policy network that produces actions conditioned on the state and the instruction. The model is trained using a combination of imitation learning and reinforcement learning. Experiments reveal that the proposed method outperforms several baselines and also generalizes well to zero-shot unseen tasks (where the goal is to craft new combinations of seen components). 

This paper studies the problem of generating natural language instructions to guide a policy to generalize to new environments where reward function is not given but a small set of demonstrations can be provided. A natural language generation (NLG) LSTM network and an initial policy network is trained from a labeled data collected from human workers. Environment state is encoded into a state encoding and LSTM is initialized with this vector to decode the correct sequence of instruction tokens. Policy network consumes both the state encoding and LSTM hidden state to output the correct action. This model is fine-tuned via reinforcement learning (RL) where the NLG component (assuming only the LSTM) is fixed and only policy network is updated (assuming state encoder is also updated from policy network). During testing, the instruction is not given but sampled from the NLG module. Experimental results show that language improves the performance, especially in zero-shot learning setting, compared to baselines where language is not incorporated or used naively in a discriminator network.