Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning

1 INTRODUCTION . One of the most remarkable aspects of human intelligence is the ability to quickly adapt to new tasks and environments . From a young age , children are able to acquire new skills and solve new tasks through imitation and instruction ( Council et al. , 2000 ; Meltzoff , 1988 ; Hunt , 1965 ) . The key is our ability to use language to learn abstract concepts and then reapply them in new settings . Inspired by this , one of the long term goals in AI is to build agents that can learn to accomplish new tasks and goals in an open-world setting using just a few examples or few instructions from humans . For example , if we had a health-care assistant robot , we might want to teach it how to bring us our favorite drink or make us a meal in just the way we like it , perhaps by showing it how to do this a few times and explaining the steps involved . However , the ability to adapt to new environments and tasks remains a distant dream . Previous work have considered using language as a high-level representation for RL ( Andreas et al. , 2017 ; Jiang et al. , 2019 ) . However , these approaches typically use language generated from templates that are hard-coded into the simulators the agents are tested in , allowing the agents to receive virtually unlimited training data to learn language abstractions . But both ideally and practically , instructions are a limited resource . If we want to build agents that can quickly adapt in open-world settings , they need to be able to learn from limited , real instruction data ( Luketina et al. , 2019 ) . And unlike the clean ontologies generated in these previous approaches , human language is noisy and diverse ; there are many ways to say the same thing . Approaches that aim to learn new tasks from humans must be able to use human-generated instructions . In this work , we take a step towards agents that can learn from limited human instruction and demonstration by collecting a new dataset with natural language annotated tasks and corresponding gameplay . The environment and dataset is designed to directly test multi-task and sub-task learning , as it consists of nearly 50 diverse crafting tasks.1 Crafts are designed to share similar features and 1Our dataset , environment , and code can be found at : https : //github.com/valeriechen/ask-your-humans . sub-steps so we would be able to test whether the method is able to learn these shared features and reuse existing knowledge to solve new , but related tasks more efficiently . Our dataset is collected in a crafting-based environment and contains over 6,000 game traces on 14 unique crafting tasks which serve as the training set . The other 35 crafting tasks will act as zero-shot tasks . The goal is for an agent to be able to learn one policy that is able to solve both tasks it was trained on as well as a variety of unseen tasks which contain similar sub-tasks as the training tasks . To do this , we train a neural network system to generate natural language instructions as a highlevel representation of the sub-task , and then a policy to achieve the goal condition given these instructions . Figure 1 shows how our agent takes in the given state of the environment and a goal ( Iron Ore ) , generates a language representation of the next instruction , and then uses the policy to select an action conditioned on the language representation - in this case to grab the key . We incorporate both imitation learning ( IL ) using both the language and human demonstrations and Reinforcement Learning ( RL ) rewards to train our agent to solve complicated multi-step tasks . Our approach which learns from human demonstrations and language outperforms or matches baseline methods in the standard RL setting . We demonstrate that language can be used to better generalize to new tasks without reward signals and outperforms baselines on average over 35 zero-shot crafting tasks . Our method uses language as a high-level task to help decompose a larger , complex task into sub-tasks and identify correct sub-tasks to utilize in a zero-shot task setting . We also show that the agent can learn few-shot tasks with only a few additional demos and instructions . Finally , training with human-generated instructions gives us an interpretable explanation of the agent ’ s behavior in cases of success and failure . Generalization is further demonstrated in the agent ’ s ability to explain how the task is decomposed in both train and evaluation settings , in a way that reflects the actual recipes that describe the crafting task . With our dataset collection procedure and languageconditioned method , we demonstrate that using natural human language can be practically applied to solving difficult RL problems and begin solving the generalization problem in RL . We hope that this will inspire future work that incorporates human annotation , specifically language annotation , to solve more difficult and diverse tasks . 2 RELATED WORK . Previous works on language descriptions of tasks and sub-tasks have generally relied on what Andreas et al . ( 2017 ) calls “ sketches. ” A sketch specifies the necessary sub-tasks for a final task and is manually constructed for every task . The agent then relies on reward signals from the sketches in order to learn these predefined sub-tasks . However , in our setup , we want to infer such “ sketches ” from a limited number of instructions given by human demonstrations . This setting is not only more difficult but also more realistic for practical applications of RL where we might not have a predefined ontology and simulator , just a limited number of human-generated instructions . In addition , at test time , their true zero-shot task requires the sketch , whereas our method is able to generate the “ sketches ” in the form of high-level language with no additional training and supervision . Similarly , other works have used synthetically generated sub-goals and descriptions to train their methods and suffer from similar problems of impracticality . Shu et al . ( 2018 ) introduces a Stochastic Temporal Grammar to enable interpretable multi-task RL in the Minecraft environment . Similarly , the BabyAI platform Chevalier-Boisvert et al . ( 2019a ) presents a synthetic language which models commands inside a grid-based environment . They utilize curriculum training to approach learning complex skills and demonstrate through experimentation in their environment that existing approaches of pure IL or pure RL are extremely sample inefficient . Cideron et al . ( 2019 ) extend Hindsight Experience Replay ( HER ) to language goals in the BabyAI platform to solve a single instruction generated from a hand-crafted language . The BabyAI environment is extended by Cao et al . ( 2020 ) to include descriptive texts of the environment to improve the generalization of RL agents . Jiang et al . ( 2019 ) also uses procedural generated language using the MuJoCo physics engine and the CLEVR engine to learn a hierarchical representation for multi-task RL . Oh et al . ( 2017 ) also tackles zero-shot generalizations , but like the others considers only procedurally-generated instructions , learning to use analogies to learn correspondences between similar sub-tasks . The main work that also investigates using a limited number of human-generated instructions in RL environments is Hu et al . ( 2019 ) . This paper also uses natural language instructions in hierarchical decision making to play a real-time strategy game involving moving troop units across long time scales . This work uses only behavioral cloning with natural language instructions , whereas we use a mixture of RL and imitation learning . They also do not investigate the benefits of language in zero-shot or few-shot settings and do not demonstrate cross-task generalization as we do . Hierarchical approaches as a way of learning abstractions is well studied in the Hierarchical Reinforcement Learning ( HRL ) literature ( Dayan & Hinton , 1993 ; Parr & Russell , 1998 ; Stolle & Precup , 2002 ) . This is typically done by predefining the low-level policies by hand , by using some proxy reward to learn a diverse set of useful low-level policies ( Heess et al. , 2016 ; Florensa et al. , 2017 ; Eysenbach et al. , 2018 ; Hausman et al. , 2018 ; Marino et al. , 2019 ) , or more generally learning options ( Sutton et al. , 1999 ) . Our approach differs in that unlike in options and other frameworks , we generate language as a high-level state which conditions the agent ’ s policy rather than handing control over to low-level policies directly . Other works have shown the effectiveness of using a combination of reinforcement learning and imitation learning . Le et al . ( 2018 ) presents a hybrid hierarchical reinforcement learning and imitation learning algorithm for the game Montezuma ’ s revenge by leveraging IL for the high-level controller and RL for the low-level controller demonstrating the potential for combining IL and RL to achieve the benefits of both algorithms . By learning meta-actions , the agent is able to learn to solve the complex game . However , their meta-actions were also hand specified . Others have utilized natural language for other tasks , including Williams et al . ( 2018 ) , Co-Reyes et al . ( 2018 ) , and Andreas et al . ( 2018 ) but not focused on the multi-task learning setting . Matthews et al . ( 2019 ) demonstrates the use of word embeddings to inform robotic motor control as evidence of particular promise for exploiting the relationship between language and control . Narasimhan et al . ( 2018 ) uses language descriptions of the environment to aid domain transfer . The sub-field of language and vision navigation specifically has investigated how to train agents to navigate to a particular location in an environment given templated or natural language ( Chaplot et al. , 2018 ; Anderson et al. , 2018 ; Tellex et al. , 2011 ; Mei et al. , 2016 ; Chen & Mooney , 2011 ; Yu et al. , 2018 ) or to navigate to a particular location to answer a question Das et al . ( 2018 ) . Similarly to this work , Nguyen et al . ( 2019 ) uses human-generated language to find objects in a simulated environment , Zhong et al . ( 2020 ) and Branavan et al . ( 2012 ) reads a document ( i.e . a players manual ) to play a variety of games , and Lynch & Sermanet ( 2020 ) trains agents to follow both image and languagebased goals . All of these works require the agent to read some text at both train and test time and follow those instructions to achieve some goal . In contrast , at test time , our agent only receives a high-level goal , which is what item to craft . Our agent must take the high-level goal as input and generates its own instructions to solve the task . In other words , our task is both instruction following and instruction generation . Related to instruction generation , some work have explored more generally intrinsic motivation for goal generation ( Florensa et al. , 2018 ; Forestier et al. , 2017 ) . In our work , however , we learn the goals via the human language instructions . 3 HUMAN ANNOTATION COLLECTION . The first step of our approach requires human demonstrations and instructions . To that requirement , we built an interface to collect human-annotated data to guide the learning model . Crafting Task : The goal of the agent in our world is to complete crafts . By design , a crafting-based world allows for complexity and hierarchy in how the agent interacts with items in the gridworld . To craft an item , the agent must generally first pick up a tool , go to a resource , mine the resource , and then go to a table to craft the item . To make an iron ore , the agent must use the pickaxe at the Iron Ore Vein to mine Iron Ore to complete the task . The Iron Ore recipe is an example of a 1-step task because it creates one item . A 5-step task , like Diamond Pickaxe , involves the mining and/or crafting of 5 items . We capped the tasks at a maximum length of 5 recipe steps to limit the amount of time a worker would have to spend on the task . Note that each recipe step requires multiple time-steps to complete . Crafts are designed to share similar features and sub-steps to test whether the agent is able to learn these shared features and reuse existing knowledge to solve new , but related tasks more efficiently ( these relations between tasks are detailed in Table 3 and in Figure 10 ) . While the task may seem simple to human annotators to solve , such compositional tasks still pose difficulties for sparse-reward RL . We further increase the difficulty of this task by restricting the agent to a limited number of steps ( 100 ) to complete the task , leaving little room to make unrecoverable mistakes such as spending time collecting or using unnecessary resources . Data Collection Process : Figure 2 shows our interface . Given the goal craft , relevant recipes , and the initial board configuration , the worker provides step-by-step instructions accompanied by execution on the actual game board of each instruction . The workflow would be to type one instruction , execute the instruction , then type the next instruction , and execute until the goal was completed . The data collection interface and a corresponding example set of natural language instructions provided by a Turker is illustrated on the rightmost side of Figure 2 . This is but one way that a Turker might choose to break down the 5-step crafting task . The appendix has more details on the collection process in Section A.1 . We will release the environment and dataset . Dataset Analysis : Between the 14 crafts , we collected 6,322 games on AMT . In total , this dataset contains 195,405 state-action pairs and 35,901 total instructions . In the supplementary material we present relevant summary statistics about the data , including the number of instructions provided for each n-step task . The number of instructions , and consequently , actions required increases with the number steps as shown in Table 4 .

Summary: This paper proposes to use natural language to aid reinforcement learning by generating instructions for sub-goals that allow the agent to complete tasks with delayed rewards. The authors first design a multi-task crafting environment and collect step-by-step human demonstrations along with sub-goal instructions using crowdsourced workers. The proposed method then involves a model architecture that includes a recurrent neural network for generating instructions, and a policy network that produces actions conditioned on the state and the instruction. The model is trained using a combination of imitation learning and reinforcement learning. Experiments reveal that the proposed method outperforms several baselines and also generalizes well to zero-shot unseen tasks (where the goal is to craft new combinations of seen components).