Learning to Reinforcement Learn by Imitation

1 INTRODUCTION . Meta-learning holds the promise of enabling learning systems to compile a diverse set of prior experiences and use this compiled prior knowledge to efficiently learn new skills or rapidly adapt to new environments . Meta reinforcement learning seeks to enable fast learning of new skills through trial-and-error , akin to how humans can rapidly learn how to walk when on ice for the first time , or more quickly learn how to play a new sport given experience with other sports . Hence , meta learning is an important aspect of how humans learn , and is particularly useful in real-world situations with diverse and dynamic environments . Unfortunately , it is challenging to develop meta-reinforcement learning methods that are practical , due to the immense sample complexity of reinforcement learning ( RL ) methods that is exacerbated by learning-to-learn . For example , prior meta-reinforcement learning methods have reported using more than 250 million transitions for learning to reinforcement learn in tabular MDPs ( Duan et al. , 2016 ) . Thus , if we want to endow machines with the ability use prior experience to quickly and autonomously adapt to new situations in real world settings , we need to develop more practical algorithms for learning fast reinforcement learning procedures . We make the following observation in this work : while the goal of meta-reinforcement learning is to acquire fast and efficient reinforcement learning procedures , those procedures themselves do not need to be acquired through reinforcement learning . In the same way that humans can be coached ( by other humans ) , a meta-reinforcement learning algorithm can receive more direct supervision during meta-training , in the form of example demonstrations , and then optimize for a purely rewarddriven reinforcement learning procedure that converges to solutions that are as good as the provided demonstrations . At meta-test time , when faced with a new task , the method solves the same problem as conventional meta-reinforcement learning : acquiring the new skill using only reward signals . But during meta-training , the availability of demonstrations dramatically improves the efficiency and effectiveness of the method . Our main contribution is a method that learns fast reinforcement learning via imitation . As illustrated in Figure 1 , we optimize for a set of parameters such that only one or a few gradient steps leads to a policy that matches the expert ’ s actions . By using demonstrations during meta-training , there is no challenge with exploration in the meta-optimization , making it possible to effectively learn how to learn in sparse reward environments , including from binary success/failure feedback . While the combination of imitation and RL has been explored before ( Peters & Schaal , 2006 ; Taylor et al. , 2011 ; Brys et al. , 2015 ; Subramanian et al. , 2016 ; Hester et al. , 2018 ; Sun et al. , 2018 ; Rajeswaran et al. , 2018 ; Nair et al. , 2018 ) , the particular combination of imitation and RL in a meta-learning context has not been studied in prior work . As we show in our experiments , this combination is in fact extremely powerful : compared to meta-reinforcement learning , our method can meta-learn comparable adaptation skills with up to 10x fewer interaction episodes , making meta-RL much more viable for real-world learning . Further , our experiments indicate that our method can be used to acquire reinforcement learning procedures that effectively learn from sparse rewards in constrained settings . 2 RELATED WORK . Our work builds upon prior work on meta-learning ( Schmidhuber , 1987 ; Bengio et al . ; Thrun & Pratt , 2012 ) , where the goal is to learn how to efficiently learn . We focus on the particular case where the goal is to learn an efficient reinforcement learner , as typical in the meta-reinforcement learning setting ( Schmidhuber , 1987 ; Duan et al. , 2016 ; Wang et al. , 2016 ; Finn et al. , 2017a ) . Prior works have sought to solve this problem by optimizing for an efficient reinforcement learner through reinforcement learning , where the learner is represented by a recurrent network ( Duan et al. , 2016 ; Wang et al. , 2016 ; Mishra et al. , 2018 ; Stadie et al. , 2018 ) , gradient descent from a learned initialization ( Finn et al. , 2017a ; Gupta et al. , 2018 ) , a learned critic that provides gradients to the policy ( Sung et al. , 2017 ; Houthooft et al. , 2018 ) , or a planner using an adaptable model ( Clavera et al. , 2018 ) . The key differentiating factor with this work is that our approach aims to leverage demonstrations for learning an efficient RL procedure . Our algorithm is able to learn from a reasonable number of demonstrations with a very small number of real world trials , making it practical for applications in the physical world , where humans can provide demonstrations efficiently . The demonstrations can also come from an algorithmic expert , yielding an alternative interpretation of our method as a meta-reinforcement learning algorithm that optimizes for expert policies using RL and then separately uses such experts for meta-training . Our approach is also related to few-shot imitation learning ( Duan et al. , 2017 ; Finn et al. , 2017b ) , in that we leverage demonstration data for meta-learning . However , unlike these approaches , we are learning a trial-and-error procedure , and demonstrations are only required during meta-training . Prior work has also sought to learn planning procedures through imitation ( Tamar et al. , 2016 ; Karkus et al. , 2017 ; Okada et al. , 2017 ; Srinivas et al. , 2018 ; Choudhury et al. , 2018 ; Lee et al. , 2018 ) . We instead learn a trial-and-error RL procedure through imitation . Meta-learning is closely related to multi-task learning ( Caruana , 1998 ) , where the goal is to master a fixed set of predefined tasks ( whereas meta-learning seeks to use experience from multiple tasks to quickly master new tasks ) . In this respect , our approach is related to multi-task learning methods that seek to distill policies for multiple tasks into a single policy , such as guided policy search ( Levine et al. , 2016 ) , policy distillation ( Rusu et al. , 2016 ; Parisotto et al. , 2016 ) , and related approaches ( Teh et al. , 2017 ; Omidshafiei et al. , 2017 ; Ghosh et al. , 2018 ) . Like these prior works , we use a separate expert for each condition or task , but unlike these approaches , we use these experts to train a meta-learner , rather than a single policy . Prior methods have also sought to use demonstrations to make standard reinforcement learning more efficient in the single-task setting ( Peters & Schaal , 2006 ; Kober & Peters , 2009 ; Kormushev et al. , 2010 ; Taylor et al. , 2011 ; Brys et al. , 2015 ; Subramanian et al. , 2016 ; Hester et al. , 2018 ; Sun et al. , 2018 ; Rajeswaran et al. , 2018 ; Nair et al. , 2018 ; Kober et al. , 2013 ; Silver et al. , 2016 ) . These prior methods aim to learn a policy from demonstrations and rewards . Our approach instead aims to leverage demonstrations to learn how to efficiently reinforcement learn without demonstrations . We use a simple behavioral cloning objective in the outer loop . Ideas from the aforementioned papers , as well as more sophisticated imitation algorithms like DAgger ( Ross et al. , 2011 ) and BCO ( Torabi et al. , 2018 ) , are complementary to our approach and could be used as a drop-in replacement for the outer optimization . 3 PRELIMINARIES . As our goal is to learn an RL algorithm in an efficient way , we will build on the model-agnostic meta-learning ( MAML ) algorithm ( Finn et al. , 2017a ) , which has previously been shown to be more data efficient than recurrent meta-learners ( Finn et al. , 2017b ) . In this section , we will introduce notation and overview the MAML algorithm . Meta-learning algorithms optimize for the ability to learn new tasks quickly and efficiently . To do so , they use data collected across a wide range of meta-training tasks and are evaluated based on their ability to learn new meta-test tasks . Meta-learning assumes that the meta-training and meta-test tasks are drawn from some distribution p ( T ) . Generally , meta-learning can be viewed as discovering the structure that exists between tasks such that , when the model is presented with a new task from the meta-test set , it can use the known structure to quickly learn the task . MAML achieves this by optimizing for a deep network ’ s initial parameter setting such that one or a few steps of gradient descent on a few datapoints leads to effective generalization ( referred to as few-shot generalization ) ( Finn et al. , 2017a ) . Then , after meta-training , the learned parameters are fine-tuned on data from a new task . Concretely , consider a supervised learning problem with a loss function denoted as L ( θ , D ) , where θ denotes the model parameters and D denotes the labeled data . For a few-shot supervised learning problem , MAML assumes access to a small amount of data for a large number of tasks . During meta-training , a task T is sampled , along with data from that task , which is randomly partitioned into two sets , Dtr and Dval . We will assume that Dtr has K examples . MAML optimizes for a set of model parameters θ such that one or a few gradient steps on Dtr produces good performance on Dval . Effectively , MAML optimizes for generalization from K examples . Thus , using φT to denote the updated parameters , the MAML objective is the following : min θ ∑ T L ( θ − α∇θL ( θ , DtrT ) , DvalT ) = min θ ∑ T L ( φT , DvalT ) . where α is a step size that can be set as a hyperparameter or learned . Moving forward , we will refer to the outer objective as the meta-objective . Subsequently , at meta-test time , K examples from a new , held-out task Ttest are presented and we can run gradient descent starting from θ to infer model parameters for the new task : φTtest = θ − α∇θL ( θ , DtrTtest ) . For convenience , we will use only one inner gradient step in the equations . However , using multiple inner gradient steps is straight-forward , and frequently done in practice . Finn et al . ( 2017a ) also applied the MAML algorithm to the meta-reinforcement learning setting , where each dataset DTi consists of trajectories of the form s1 , a1 , ... , aH−1 , sH and where the inner and outer loss function corresponds to the negative expected reward : LRL ( φ , DTi ) = − 1 |DTi | ∑ st , at∈DTi ri ( st , at ) = −Est , at∼πφ , qTi [ 1 H H∑ t=1 ri ( st , at ) ] . ( 1 ) Here , qTi is used to denote the transition dynamics of task Ti . Policy gradients ( Williams , 1992 ) were used to estimate the gradient of this loss function . Thus , the algorithm proceeded as follows : for each task Ti , first collect samples DtrTi from the policy πθ , then compute the updated parameters using the policy gradient evaluated on DtrTi , then collect new samples D val Ti via the updated policy parameters , and finally update the initial parameters θ by taking a gradient step on the meta-objective . In the next section , we will introduce a new approach to meta-reinforcement learning using ideas from the MAML algorithm .

Summary: This paper proposes a meta-learning algorithm for reinforcement learning that incorporates expert demonstrations. The goal is to reduce the sample complexity of meta-RL algorithms in the validation phase. The paper provides a good discussion of the background literature. Experimental results are provided on multi-goal planning problems for three prototypical simulated systems, namely, a 2D point-mass robot, a 7-DOF manipulator and a quadruped crawler.