Watch, Try, Learn: Meta-Learning from Demonstrations and Rewards

Imitation learning allows agents to learn complex behaviors from demonstrations . However , learning a complex vision-based task may require an impractical number of demonstrations . Meta-imitation learning is a promising approach towards enabling agents to learn a new task from one or a few demonstrations by leveraging experience from learning similar tasks . In the presence of task ambiguity or unobserved dynamics , demonstrations alone may not provide enough information ; an agent must also try the task to successfully infer a policy . In this work , we propose a method that can learn to learn from both demonstrations and trial-anderror experience with sparse reward feedback . In comparison to meta-imitation , this approach enables the agent to effectively and efficiently improve itself autonomously beyond the demonstration data . In comparison to meta-reinforcement learning , we can scale to substantially broader distributions of tasks , as the demonstration reduces the burden of exploration . Our experiments show that our method significantly outperforms prior approaches on a set of challenging , vision-based control tasks . 1 INTRODUCTION . Imitation learning enables autonomous agents to learn complex behaviors from demonstrations , which are often easy and intuitive for users to provide . However , learning expressive neural network policies from imitation requires a large number of demonstrations , particularly when learning from high-dimensional inputs such as images . Meta-imitation learning has emerged as a promising approach for allowing an agent to leverage data from previous tasks in order to learn a new task from only a handful of demonstrations ( Duan et al. , 2017 ; Finn et al. , 2017b ; James et al. , 2018 ) . However , in many practical few-shot imitation settings , there is an identifiability problem : it may not be possible to precisely determine a policy from one or a few demonstrations , especially in a new situation . And even if a demonstration precisely communicates what the task entails , it might not precisely communicate how to accomplish it in new situations . For example , it may be difficult to discern from a single demonstration where to grasp an object when it is in a new position or how much force to apply in order to slide an object without knocking it over . It may be expensive to collect more demonstrations to resolve such ambiguities , and even when we can , it may not be obvious to a human demonstrator where the agent ’ s difficulty is arising from . Alternatively , it is easy for the user to provide success-or-failure feedback , while exploratory interaction is useful for learning how to perform the task . To this end , our goal is to build an agent that can first infer a policy from ∗Work done as a Google AI Resident . one demonstration , then attempt the task using that policy while receiving binary user feedback , and finally use the feedback to improve its policy such that it can consistently solve the task . This vision of learning new tasks from a few demonstrations and trials inherently requires some amount of prior knowledge or experience , which we can acquire through meta-learning across a range of previous tasks . To this end , we develop a new meta-learning algorithm that incorporates elements of imitation learning with trial-and-error reinforcement learning . In contrast to previous meta-imitation learning approaches that learn one-shot imitation learning procedures through imitation ( Duan et al. , 2017 ; Finn et al. , 2017b ) , our approach enables the agent to improve at the test task through trial-and-error . Further , from the perspective of meta-RL algorithms that aim to learn efficient RL procedures ( Duan et al. , 2016 ; Wang et al. , 2016 ; Finn et al. , 2017a ) , our approach also has significant appeal : as we aim to scale meta-RL towards broader task distributions and learn increasingly general RL procedures , exploration and efficiency becomes exceedingly difficult . However , a demonstration can significantly narrow down the search space while also providing a practical means for a user to communicate the goal , enabling the agent to achieve few-shot learning of behavior . While the combination of demonstrations and reinforcement has been studied extensively in single task problems ( Kober et al. , 2013 ; Sun et al. , 2018 ; Rajeswaran et al. , 2018 ; Le et al. , 2018 ) , this combination is particularly important in meta-learning contexts where few-shot learning of new tasks is simply not possible without demonstrations . Further , we can even significantly improve upon prior methods that study this combination using meta-learning to more effectively integrate the information coming from both sources . The primary contribution of this paper is a meta-learning algorithm that enables learning of new behaviors with a single demonstration and trial experience . After receiving a demonstration illustrating a new goal , the meta-trained agent can learn to accomplish that goal through a small amount of trial-and-error with only binary success-or-failure labels . We evaluate our algorithm and several prior methods on a challenging , vision-based control problem involving manipulation tasks from four distinct families of tasks : button-pressing , grasping , pushing , and pick and place . We find that our approach can effectively learn tasks with new , held-out objects using one demonstration and a single trial , while significantly outperforming meta-imitation learning , meta-reinforcement learning , and prior methods that combine demonstrations and reward feedback . To our knowledge , our experiments are the first to show that meta-learning can enable an agent to adapt to new tasks with binary reinforcement signals from raw pixel observations , which we show with a single meta-model for a variety of distinct manipulation tasks . We have published videos of our experimental results1 and the experiment model code2 . 2 RELATED WORK . Learning to learn , or meta-learning , has a long-standing history in the machine learning literature ( Thrun & Pratt , 1998 ; Schmidhuber , 1987 ; Bengio et al. , 1992 ; Hochreiter et al. , 2001 ) . We particularly focus on meta-learning in the context of control . Our approach builds on and signifi- 1 https : //sites.google.com/view/watch-try-learn-project 2https : //github.com/google-research/tensor2robot/tree/master/research/ vrgripper cantly improves upon meta-imitation learning ( Duan et al. , 2017 ; Finn et al. , 2017b ; James et al. , 2018 ; Paine et al. , 2018 ) and meta-reinforcement learning ( Duan et al. , 2016 ; Wang et al. , 2016 ; Mishra et al. , 2018 ; Rakelly et al. , 2019 ) , extending contextual meta-learning approaches . Unlike prior work in few-shot imitation learning ( Duan et al. , 2017 ; Finn et al. , 2017b ; Yu et al. , 2018 ; James et al. , 2018 ; Paine et al. , 2018 ) , our method enables the agent to additionally improve upon trial-and-error experience . In contrast to work in multi-task and meta-reinforcement learning ( Duan et al. , 2016 ; Wang et al. , 2016 ; Finn et al. , 2017a ; Mishra et al. , 2018 ; Houthooft et al. , 2018 ; Sung et al. , 2017 ; Nagabandi et al. , 2019 ; Sæmundsson et al. , 2018 ; Hausman et al. , 2017 ) , our approach learns to use one demonstration to address the meta-exploration problem ( Gupta et al. , 2018 ; Stadie et al. , 2018 ) . Our work also requires only one round of on-policy data collection , collecting only 1 , 500 trials for the vision-based manipulation tasks , while nearly all prior meta-learning works require thousands of iterations of on-policy data collection , amounting to hundreds of thousands of trials ( Duan et al. , 2016 ; Wang et al. , 2016 ; Finn et al. , 2017a ; Mishra et al. , 2018 ) . Combining demonstrations and trial-and-error experience has long been explored in the machine learning and robotics literature ( Kober et al. , 2013 ) . This ranges from simple techniques such as demonstration-based pre-training and initialization ( Peters & Schaal , 2006 ; Kober & Peters , 2009 ; Kormushev et al. , 2010 ; Kober et al. , 2013 ; Silver et al. , 2016 ) to more complex methods that incorporate both demonstration data and reward information in the loop of training ( Taylor et al. , 2011 ; Brys et al. , 2015 ; Subramanian et al. , 2016 ; Hester et al. , 2018 ; Sun et al. , 2018 ; Rajeswaran et al. , 2018 ; Nair et al. , 2018 ; Le et al. , 2018 ) . The key contribution of this paper is an algorithm that can learn how to learn from both demonstrations and rewards . This is quite different from RL algorithms that incorporate demonstrations : learning from scratch with demonstrations and RL involves a slow , iterative learning process , while fast adaptation with a meta-trained policy involves extracting inherently distinct pieces of information from the demonstration and the trials . The demonstration provides information about what to do , while the small number of RL trials can disambiguate the task and how it s refinement . As a result , we get a procedure that significantly exceeds the efficiency of prior approaches , requiring only one demonstration and one trial to adapt to a new test task , even from pixel observations , by leveraging previous data . In comparison , single-task methods for learning from demonstrations and rewards typically require hundreds or thousands of trials to learn tasks of comparable difficulty ( Rajeswaran et al. , 2018 ; Nair et al. , 2018 ) . 3 META-LEARNING FROM DEMONSTRATIONS AND REWARDS . We first introduce some meta-learning preliminaries , then formalize our particular problem statement before finally describing our approach and its implementation . 3.1 PRELIMINARIES . Meta-learning , or learning to learn , aims to learn new tasks from very little data . To achieve this a meta-learning algorithm can first meta-train on a set of tasks { Ti } , called the meta-train tasks . We then evaluate how quickly the meta-learner can learn an unseen meta-test task Tj . We typically assume that the meta-train and meta-test tasks are drawn from some unknown task distribution p ( T ) ( Finn , 2018 ) . Through meta-training , a meta-learner can learn some common structure between the tasks in p ( T ) which it can use to more quickly learn a new meta-test task . We define a task Ti as a finite-horizon Markov decision process ( MDP ) , { S , A , ri , Pi , H } , with continuous state space S , continuous action space A , reward function ri : S × A → R , unknown dynamics Pi ( st+1|st , at ) , and horizonH . In our manipulation experiments we will restrict the tasks to sparse binary rewards ri : S → { 0 , 1 } . The state space S , reward ri , and dynamics Pi may vary across tasks . 3.2 PROBLEM STATEMENT . Our goal is to meta-train an agent such that can quickly learn a new test task Tj in two phases : Phase I : The agent observes and learns from k = 1 , ... , K task demonstrations D∗j : = { dj , k } . It can then attempt the task in ` = 1 , · · · , L trial episodes { τj , ` } , for which it receives reward labels . Phase II : The agent learns from those trial episodes and the original demos to succeed at Tj . A demonstration is a trajectory d = { ( st , at ) } of T states-action tuples that succeeds at the task , while a trial episode is a trajectory τ = { ( st , at , ri ( st , at ) ) } that also contains reward information .

Summary: The paper introduces a meta-learning approach that can learn from demonstrations and subsequent reinforcement learning trials. The approach works in two phases. First, multiple demonstrations are collected for each task and a meta-learning policy is obtained by embedding the demonstrations of a single task in a context vector that is given to the actor policy.  Subsequently, the meta-policy is used to collect trajectories which are also evaluated with a (sparse) reward. These trajectories, along with the demonstrations, are again used by a second meta-learning policy (with a concatenated context from demonstrations and trajectories) to obtain the final policy. The algorithm is tested on a simple point reaching task as well as on a more complex 3d physics simulation that contains pick-and-place tasks, pushing tasks and button pressing.