Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning

1 INTRODUCTION . One of the most remarkable aspects of human intelligence is the ability to quickly adapt to new tasks and environments . From a young age , children are able to acquire new skills and solve new tasks through imitation and instruction ( Council et al. , 2000 ; Meltzoff , 1988 ; Hunt , 1965 ) . The key is our ability to use language to learn abstract concepts and then reapply them in new settings . Inspired by this , one of the long term goals in AI is to build agents that can learn to accomplish new tasks and goals in an open-world setting using just a few examples or few instructions from humans . For example , if we had a health-care assistant robot , we might want to teach it how to bring us our favorite drink or make us a meal in just the way we like it , perhaps by showing it how to do this a few times and explaining the steps involved . However , the ability to adapt to new environments and tasks remains a distant dream . Previous work have considered using language as a high-level representation for RL ( Andreas et al. , 2017 ; Jiang et al. , 2019 ) . However , these approaches typically use language generated from templates that are hard-coded into the simulators the agents are tested in , allowing the agents to receive virtually unlimited training data to learn language abstractions . But both ideally and practically , instructions are a limited resource . If we want to build agents that can quickly adapt in open-world settings , they need to be able to learn from limited , real instruction data ( Luketina et al. , 2019 ) . And unlike the clean ontologies generated in these previous approaches , human language is noisy and diverse ; there are many ways to say the same thing . Approaches that aim to learn new tasks from humans must be able to use human-generated instructions . In this work , we take a step towards agents that can learn from limited human instruction and demonstration by collecting a new dataset with natural language annotated tasks and corresponding gameplay . The environment and dataset is designed to directly test multi-task and sub-task learning , as it consists of nearly 50 diverse crafting tasks.1 Crafts are designed to share similar features and 1Our dataset , environment , and code can be found at : https : //github.com/valeriechen/ask-your-humans . sub-steps so we would be able to test whether the method is able to learn these shared features and reuse existing knowledge to solve new , but related tasks more efficiently . Our dataset is collected in a crafting-based environment and contains over 6,000 game traces on 14 unique crafting tasks which serve as the training set . The other 35 crafting tasks will act as zero-shot tasks . The goal is for an agent to be able to learn one policy that is able to solve both tasks it was trained on as well as a variety of unseen tasks which contain similar sub-tasks as the training tasks . To do this , we train a neural network system to generate natural language instructions as a highlevel representation of the sub-task , and then a policy to achieve the goal condition given these instructions . Figure 1 shows how our agent takes in the given state of the environment and a goal ( Iron Ore ) , generates a language representation of the next instruction , and then uses the policy to select an action conditioned on the language representation - in this case to grab the key . We incorporate both imitation learning ( IL ) using both the language and human demonstrations and Reinforcement Learning ( RL ) rewards to train our agent to solve complicated multi-step tasks . Our approach which learns from human demonstrations and language outperforms or matches baseline methods in the standard RL setting . We demonstrate that language can be used to better generalize to new tasks without reward signals and outperforms baselines on average over 35 zero-shot crafting tasks . Our method uses language as a high-level task to help decompose a larger , complex task into sub-tasks and identify correct sub-tasks to utilize in a zero-shot task setting . We also show that the agent can learn few-shot tasks with only a few additional demos and instructions . Finally , training with human-generated instructions gives us an interpretable explanation of the agent ’ s behavior in cases of success and failure . Generalization is further demonstrated in the agent ’ s ability to explain how the task is decomposed in both train and evaluation settings , in a way that reflects the actual recipes that describe the crafting task . With our dataset collection procedure and languageconditioned method , we demonstrate that using natural human language can be practically applied to solving difficult RL problems and begin solving the generalization problem in RL . We hope that this will inspire future work that incorporates human annotation , specifically language annotation , to solve more difficult and diverse tasks . 2 RELATED WORK . Previous works on language descriptions of tasks and sub-tasks have generally relied on what Andreas et al . ( 2017 ) calls “ sketches. ” A sketch specifies the necessary sub-tasks for a final task and is manually constructed for every task . The agent then relies on reward signals from the sketches in order to learn these predefined sub-tasks . However , in our setup , we want to infer such “ sketches ” from a limited number of instructions given by human demonstrations . This setting is not only more difficult but also more realistic for practical applications of RL where we might not have a predefined ontology and simulator , just a limited number of human-generated instructions . In addition , at test time , their true zero-shot task requires the sketch , whereas our method is able to generate the “ sketches ” in the form of high-level language with no additional training and supervision . Similarly , other works have used synthetically generated sub-goals and descriptions to train their methods and suffer from similar problems of impracticality . Shu et al . ( 2018 ) introduces a Stochastic Temporal Grammar to enable interpretable multi-task RL in the Minecraft environment . Similarly , the BabyAI platform Chevalier-Boisvert et al . ( 2019a ) presents a synthetic language which models commands inside a grid-based environment . They utilize curriculum training to approach learning complex skills and demonstrate through experimentation in their environment that existing approaches of pure IL or pure RL are extremely sample inefficient . Cideron et al . ( 2019 ) extend Hindsight Experience Replay ( HER ) to language goals in the BabyAI platform to solve a single instruction generated from a hand-crafted language . The BabyAI environment is extended by Cao et al . ( 2020 ) to include descriptive texts of the environment to improve the generalization of RL agents . Jiang et al . ( 2019 ) also uses procedural generated language using the MuJoCo physics engine and the CLEVR engine to learn a hierarchical representation for multi-task RL . Oh et al . ( 2017 ) also tackles zero-shot generalizations , but like the others considers only procedurally-generated instructions , learning to use analogies to learn correspondences between similar sub-tasks . The main work that also investigates using a limited number of human-generated instructions in RL environments is Hu et al . ( 2019 ) . This paper also uses natural language instructions in hierarchical decision making to play a real-time strategy game involving moving troop units across long time scales . This work uses only behavioral cloning with natural language instructions , whereas we use a mixture of RL and imitation learning . They also do not investigate the benefits of language in zero-shot or few-shot settings and do not demonstrate cross-task generalization as we do . Hierarchical approaches as a way of learning abstractions is well studied in the Hierarchical Reinforcement Learning ( HRL ) literature ( Dayan & Hinton , 1993 ; Parr & Russell , 1998 ; Stolle & Precup , 2002 ) . This is typically done by predefining the low-level policies by hand , by using some proxy reward to learn a diverse set of useful low-level policies ( Heess et al. , 2016 ; Florensa et al. , 2017 ; Eysenbach et al. , 2018 ; Hausman et al. , 2018 ; Marino et al. , 2019 ) , or more generally learning options ( Sutton et al. , 1999 ) . Our approach differs in that unlike in options and other frameworks , we generate language as a high-level state which conditions the agent ’ s policy rather than handing control over to low-level policies directly . Other works have shown the effectiveness of using a combination of reinforcement learning and imitation learning . Le et al . ( 2018 ) presents a hybrid hierarchical reinforcement learning and imitation learning algorithm for the game Montezuma ’ s revenge by leveraging IL for the high-level controller and RL for the low-level controller demonstrating the potential for combining IL and RL to achieve the benefits of both algorithms . By learning meta-actions , the agent is able to learn to solve the complex game . However , their meta-actions were also hand specified . Others have utilized natural language for other tasks , including Williams et al . ( 2018 ) , Co-Reyes et al . ( 2018 ) , and Andreas et al . ( 2018 ) but not focused on the multi-task learning setting . Matthews et al . ( 2019 ) demonstrates the use of word embeddings to inform robotic motor control as evidence of particular promise for exploiting the relationship between language and control . Narasimhan et al . ( 2018 ) uses language descriptions of the environment to aid domain transfer . The sub-field of language and vision navigation specifically has investigated how to train agents to navigate to a particular location in an environment given templated or natural language ( Chaplot et al. , 2018 ; Anderson et al. , 2018 ; Tellex et al. , 2011 ; Mei et al. , 2016 ; Chen & Mooney , 2011 ; Yu et al. , 2018 ) or to navigate to a particular location to answer a question Das et al . ( 2018 ) . Similarly to this work , Nguyen et al . ( 2019 ) uses human-generated language to find objects in a simulated environment , Zhong et al . ( 2020 ) and Branavan et al . ( 2012 ) reads a document ( i.e . a players manual ) to play a variety of games , and Lynch & Sermanet ( 2020 ) trains agents to follow both image and languagebased goals . All of these works require the agent to read some text at both train and test time and follow those instructions to achieve some goal . In contrast , at test time , our agent only receives a high-level goal , which is what item to craft . Our agent must take the high-level goal as input and generates its own instructions to solve the task . In other words , our task is both instruction following and instruction generation . Related to instruction generation , some work have explored more generally intrinsic motivation for goal generation ( Florensa et al. , 2018 ; Forestier et al. , 2017 ) . In our work , however , we learn the goals via the human language instructions . 3 HUMAN ANNOTATION COLLECTION . The first step of our approach requires human demonstrations and instructions . To that requirement , we built an interface to collect human-annotated data to guide the learning model . Crafting Task : The goal of the agent in our world is to complete crafts . By design , a crafting-based world allows for complexity and hierarchy in how the agent interacts with items in the gridworld . To craft an item , the agent must generally first pick up a tool , go to a resource , mine the resource , and then go to a table to craft the item . To make an iron ore , the agent must use the pickaxe at the Iron Ore Vein to mine Iron Ore to complete the task . The Iron Ore recipe is an example of a 1-step task because it creates one item . A 5-step task , like Diamond Pickaxe , involves the mining and/or crafting of 5 items . We capped the tasks at a maximum length of 5 recipe steps to limit the amount of time a worker would have to spend on the task . Note that each recipe step requires multiple time-steps to complete . Crafts are designed to share similar features and sub-steps to test whether the agent is able to learn these shared features and reuse existing knowledge to solve new , but related tasks more efficiently ( these relations between tasks are detailed in Table 3 and in Figure 10 ) . While the task may seem simple to human annotators to solve , such compositional tasks still pose difficulties for sparse-reward RL . We further increase the difficulty of this task by restricting the agent to a limited number of steps ( 100 ) to complete the task , leaving little room to make unrecoverable mistakes such as spending time collecting or using unnecessary resources . Data Collection Process : Figure 2 shows our interface . Given the goal craft , relevant recipes , and the initial board configuration , the worker provides step-by-step instructions accompanied by execution on the actual game board of each instruction . The workflow would be to type one instruction , execute the instruction , then type the next instruction , and execute until the goal was completed . The data collection interface and a corresponding example set of natural language instructions provided by a Turker is illustrated on the rightmost side of Figure 2 . This is but one way that a Turker might choose to break down the 5-step crafting task . The appendix has more details on the collection process in Section A.1 . We will release the environment and dataset . Dataset Analysis : Between the 14 crafts , we collected 6,322 games on AMT . In total , this dataset contains 195,405 state-action pairs and 35,901 total instructions . In the supplementary material we present relevant summary statistics about the data , including the number of instructions provided for each n-step task . The number of instructions , and consequently , actions required increases with the number steps as shown in Table 4 .

Summary: This paper studies the problem of generating natural language instructions to guide a policy to generalize to new environments where reward function is not given but a small set of demonstrations can be provided. A natural language generation (NLG) LSTM network and an initial policy network is trained from a labeled data collected from human workers. Environment state is encoded into a state encoding and LSTM is initialized with this vector to decode the correct sequence of instruction tokens. Policy network consumes both the state encoding and LSTM hidden state to output the correct action. This model is fine-tuned via reinforcement learning (RL) where the NLG component (assuming only the LSTM) is fixed and only policy network is updated (assuming state encoder is also updated from policy network). During testing, the instruction is not given but sampled from the NLG module. Experimental results show that language improves the performance, especially in zero-shot learning setting, compared to baselines where language is not incorporated or used naively in a discriminator network.