Guiding Policies with Language via Meta-Learning

1 INTRODUCTION . Behavioral skills or policies for autonomous agents are typically specified in terms of reward functions ( in the case of reinforcement learning ) or demonstrations ( in the case of imitation learning ) . However , both reward functions and demonstrations have downsides as mechanisms for communicating goals . Reward functions must be engineered manually , which can be challenging in real-world environments , especially when the learned policies operate directly on raw sensory perception . Sometimes , simply defining the goal of the task requires engineering the very perception system that end-to-end deep learning is supposed to acquire . Demonstrations sidestep this challenge , but require a human demonstrator to actually be able to perform the task , which can be cumbersome or even impossible . When humans must communicate goals to each other , we often use language . Considerable research has also focused on building autonomous agents that can follow instructions provided via language ( Branavan et al . ( 2009 ) ; Chen & Mooney ( 2011 ) ) . However , a single instruction may be insufficient to fully communicate the full intent of a desired behavior . For example , if we would like a robot to position an object on a table in a particular place , we might find it easier to guide it by telling it which way to move , rather than verbally defining a coordinate in space . Furthermore , an autonomous agent might be unable to deduce how to perform a task from a single instruction , even if it is very precise . In both cases , interactive and iterative corrections can help resolve confusion and ambiguity , and indeed humans often employ corrections when communicating task goals to each other . In this paper , our goal is to enable an autonomous agent to accept instructions and then iteratively adjust its policy by incorporating interactive corrections ( illustrated in Figure 1 ) . This type of in-theloop supervision can guide the learner out of local optima , provide fine-grained task definition , and is natural for humans to provide to the agent . As we discuss in Section 2 , iterative language corrections can be substantially more informative than simpler forms of supervision , such as preferences , while being substantially easier and more natural to provide than reward functions or demonstrations . ∗jcoreyes @ eecs.berkeley.edu In order to effectively use language corrections , the agent must be able to ground these corrections to concrete behavioral patterns . We propose an end-to-end algorithm for grounding iterative language corrections by using a multi-task setup to meta-train a model that can ingest its own past behavior and a correction , and then correct its behavior to produce better actions . During a meta-training phase , this model is iteratively retrained on its own behavior ( and the corresponding correction ) on a wide distribution of known tasks . The model learns to correct the types of mistakes that it actually tends to make in the world , by interpreting the language input . At meta-test time , this model can then generalize to new tasks , and learn those tasks quickly through iterative language corrections . The main contributions of our work are the formulation of guided policies with language ( GPL ) via meta-learning , as well as a practical GPL meta-learning algorithm and model . We train on English sentences sampled from a hand-designed grammar as a first step towards real human-in-the-loop supervision . We evaluate our approach on two simulated tasks - multi-room object manipulation and robotic object relocation . The first domain involves navigating a complex world with partial observation , seeking out objects and delivering them to user-specified locations . The second domain involves controlling a robotic gripper in a continuous state and action space to move objects to precise locations in relation to other objects . This requires the policy to ground the corrections in terms of objects and places , and to control and correct complex behavior . 2 RELATED WORK . Tasks for autonomous agents are most commonly specified by means of reward functions ( Sutton & Barto , 1998 ) or demonstrations ( Argall et al. , 2009 ) . Prior work has studied a wide range of different techniques for both imitation learning ( Ziebart et al. , 2008 ; Abbeel & Ng , 2004 ) and reward specification , including methods that combine the two to extract reward functions and goals from user examples ( Fu et al. , 2018 ; Thomaz et al. , 2006 ) and demonstrations ( Fu et al. , 2017 ; Liu et al. , 2017 ) . Other works have proposed modalities such as preferences ( Christiano et al. , 2017 ) or numerical scores ( Warnell et al. , 2017 ) . Natural language presents a particularly appealing modality for task specification , since it enables humans to communicate task goals quickly and easily . Unlike demonstrations , language commands do not require being able to perform the task . Unlike reward functions , language commands do not require any manual engineering . Finally , in comparison to low-bandwidth supervision modalities , such as examples of successful outcomes or preferences , language commands can communicate substantially more information , both about the goals of the task and how it should be performed . A considerable body of work has sought to ground natural language commands in meaningful behaviors . These works typically use a large supervised corpus in order to learn policies that are conditioned on natural language commands ( MacMahon et al. , 2006 ; Branavan et al. , 2009 ; Vogel & Jurafsky , 2010 ; Chen & Mooney , 2011 ; Tellex et al. , 2011 ; Artzi & Zettlemoyer , 2013 ; Kim & Mooney , 2013 ; Andreas & Klein , 2015 ; Misra et al. , 2017a ; Andreas et al. , 2018 ; Oh et al. , 2017 ) . Other works consider using a known reward function in order to learn how to ground language into expert behaviors ( Janner et al. , 2018 ; Andreas & Klein , 2015 ) . Most of these works consider the case of instruction following . However , tasks can often be quite difficult to specify with a single language description , and may require interactive guidance in order to be achieved . We focus on this setting in our work , where the agent improves its behavior via iterative language corrections . While the focus in our work is on incorporating language corrections , several prior works have also studied reinforcement learning and related problems with in-the-loop feedback of other forms Akrour et al . ( 2011 ) ; Pilarski et al . ( 2011 ) ; Akrour et al . ( 2012 ) ; El Asri et al . ( 2016 ) ; Wang et al . ( 2016b ) ; Warnell et al . ( 2017 ) ; Reddy et al . ( 2018 ) . In contrast , we study how to incorporate language corrections , which are more natural for humans to specify and can carry more information about the task . However , language corrections also present the challenge that the agent must learn how to ground them in behavior . To this end , we introduce an end-to-end algorithm that directly associates language with changes in behavior without intermediate supervision about object identities or word definitions . Our approach to learning to learn from language corrections is based on meta-reinforcement learning . In meta-reinforcement learning , a meta-training procedure is used to learn a procedure ( represented by initial network weights or a model that directly ingests past experience ) ( Schmidhuber , 1987 ) that can adapt to new tasks at meta-test time . However , while prior work has proposed meta-reinforcement learning for model-free RL ( Wang et al. , 2016a ; Duan et al. , 2016 ; Finn et al. , 2017 ; Mishra et al. , 2017 ) , model-based RL ( Clavera et al. , 2018 ) , a wide range of supervised tasks ( Snell et al. , 2017 ; Santoro et al. , 2016 ; Vinyals et al. , 2016 ; Sung et al. , 2017 ) , as well as goal specification ( Xu et al. , 2018 ; Xie et al. , 2018 ) , to our knowledge no prior work has proposed meta-training of policies that can acquire new tasks from iterative language corrections . 3 PROBLEM FORMULATION . We consider the sequential decision making framework , where an agent observes states s ∈ S , chooses to execute actions a ∈ A and transitions to a new state s′ via the transition dynamics T ( s′|s , a ) . For goal directed agents , the objective is typically to learn a policy πθ that chooses actions enabling the agent to achieve the desired goal . In this work , the agent ’ s goal is specified by a language instruction L. This instruction describes what the general objective of the task is , but may be insufficient to fully communicate the intent of a desired behavior . The agent can attempt the task multiple times , and after each attempt , the agent is provided with a language correction . Each attempt results in a trajectory τ = ( s0 , a0 , s1 , a1 , .... , sT , aT ) , the result of the agent executing its policy πθ ( a|s , L ) in the environment . After each attempt , the user generates a correction according to some unknown stochastic function of the trajectory c ∼ F ( τ ) . Here , cis a language phrase that indicates how to improve the current trajectory τ to bring it closer to accomplishing the goal . This process is repeated for multiple trials , and we will use τi to denote the trajectory on the ith trial , and ci to denote the corresponding correction . An effective model should be able to incorporate these corrections to come closer to achieving the goal . This process is illustrated in Figure 1 . In the next section , we will describe a model that can incorporate iterative corrections , and then describe a meta-training procedure that can train this model to incorporate corrections effectively . 4 THE LANGUAGE-GUIDED POLICY LEARNING MODEL . As described in Section 3 , our model for guiding policies with language ( GPL ) must take in an initial language instruction , and then iteratively incorporate corrections after each attempt at the task . This requires the model to ground the contents of the correction in the environment , and also interpret it in the context of its own previous trajectory so as to decide which actions to attempt next . To that end , we propose a deep neural network model , shown in Figure 2 , that can accept the instruction , correction , previous trajectory , and state as input . The model consists of three modules : an instruction following module , a correction module , and a policy module . The instruction following module interprets the initial language instruction . The instructions are provided as a sequence of words which is converted into a sequence of word-embeddings . A 1D CNN processes this sequence to generate an instruction embedding vector zim which is fed into the policy module . The correction module interprets the previous language corrections cn = ( c0 , . . . , cn ) in the context of the previous trajectories τn = ( τ0. . . . , τn ) . Each previous trajectory is processed by a 1D CNN to generate a trajectory embedding . The correction ci , similar to the language description , is converted into a sequence of word-embeddings which is then fed through a 1D CNN . The correction and trajectory embeddings are then concatenated and transformed by a MLP to form a single tensor . These tensors are then averaged to compute the full correction history tensor zcm = 1 n ∑n j=0 MLP ( 1dCNN ( τj ) , 1dCNN ( cj ) ) . The policy module has to integrate the high level description embedded into zim , the actionable changes from the correction module zcm , and the environment state s , to generate the right action . This module inputs zcm , zim and s and generates an action distribution p ( a|s ) that determine how the agent should act . Specific architecture details are described in the appendix . Note that , by iteratively incorporating language corrections , such a model in effect implements a learning algorithm , analogously to meta-reinforcement learning recurrent models proposed in prior work that read in previous trajectories and rewards ( Duan et al. , 2016 ; Wang et al. , 2016a ) . However , in contrast to these methods , our model has to use the language correction to improve , essentially implementing an interactive , user-guided reinforcement learning algorithm . As we will demonstrate in our experiments , iterative corrections cause this model to progressively improve its performance on the task very quickly . In the next section , we will describe a meta-learning algorithm that can train this model such that it is able to adapt to iterative corrections effectively at meta-test time .

Summary: This paper studies how to teach agents to complete tasks via natural language instructions in an iterative way, e.g., correct the behavior of agents. This is a very natural way to learn as humans. The basic idea is to learn a model that takes correction and history as inputs and output what action to take. This paper formulates this in meta-learning setting in which each task is drawn from a pre-designed task distribution and then the models are able to adapt to new tasks very fast. The proposed method is evaluated in a virtual environment where the task is to pick up a particular object in a room and bring it to a particular goal location in a different room. There are two baselines: 1) instruction only (missing information), 2) full information (not iterative), the proposed method outperforms 1) with higher task completion rate and 2) with fewer number of corrections.