Few-Shot Intent Inference via Meta-Inverse Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) algorithms have the potential to automate a wide range of decisionmaking and control tasks across a variety of different domains , as demonstrated by successful recent applications ranging from robotic control ( Kober & Peters , 2012 ; Levine et al. , 2016 ) to game playing ( Mnih et al. , 2015 ; Silver et al. , 2016 ) . A key assumption of the RL problem statement is the availability of a reward function that accurately describes the desired tasks . For many real world tasks , reward functions can be challenging to manually specify , while being crucial to good performance ( Amodei et al. , 2016 ) . Most real world tasks are multifaceted and require reasoning over multiple factors in a task ( e.g . an autonomous vehicle navigating a city at night ) , while simultaneously providing appropriate reward shaping to make the task feasible with tractable exploration ( Ng et al. , 1999 ) . These challenges are compounded by the inherent difficulty of specifying rewards for tasks with high-dimensional observation spaces such as images . Inverse reinforcement learning ( IRL ) is an approach that aims to address this problem by instead inferring the reward function from demonstrations of the task ( Ng & Russell , 2000 ) . This has the appealing benefit of taking a data-driven approach to reward specification in place of hand engineering . In practice however , rewards functions are rarely learned as it can be prohibitively expensive to provide demonstrations that cover the variability common in real world tasks ( e.g. , collecting demonstrations of opening every type of door knob ) . In addition , while learning a complex function from high dimensional observations might make an expressive function approximator seem like a reasonable modelling assumption , in the “ few-shot ” domain it is notoriously difficult to unambiguously recover a good reward function with expressive function approximators . Prior solutions have thus instead relied on low-dimensional linear models with handcrafted features that effectively encode a strong prior on the relevant features of a task . This requires engineering a set of features by hand that work well for a specific problem . In this work , we propose an approach that instead explicitly learns expressive features that are robust even when learning with limited demonstrations . Our approach relies on the key observation that related tasks share common structure that we can leverage when learning new tasks . To illustrate , considering a robot navigating through a home . While the exact reward function we provide to the robot may differ depending on the task , there is a structure amid the space of useful behaviours , such as navigating to a series of landmarks , and there are certain behaviors we always want to encourage or discourage , such as avoiding obstacles or staying a reasonable distance from humans . This notion agrees with our understanding of why humans can easily infer the intents and goals ( i.e. , reward functions ) of even abstract agents from just one or a few demonstrations Baker et al . ( 2007 ) , as humans have access to strong priors about how other humans accomplish similar tasks accrued over many years . Similarly , our objective is to discover the common structure among different tasks , and encode the structure in a way that can be used to infer reward functions from a few demonstrations . More specifically , in this work we assume access to a set of tasks , along with demonstrations of the desired behaviors for those tasks , which we refer to as the meta-training set . From these tasks , we then learn a reward function parameterization that enables effective few-shot learning when used to initialize IRL in a novel task . Our method is summarized in Fig . 1 . Our key contribution is an algorithm that enables efficient learning of new reward functions by using meta-training to build a rich “ prior ” for goal inference . Using our proposed approach , we show that we can learn deep neural network reward functions from raw pixel observations with substantially better data efficiency than existing methods and standard baselines . 2 RELATED WORK . Inverse reinforcement learning ( IRL ) ( Ng & Russell , 2000 ) is the problem of inferring an expert ’ s reward function directly from demonstrations . Prior methods for performing IRL range from margin based approaches ( Abbeel & Ng , 2004 ; Ratliff et al. , 2006 ) to probabilistic approaches ( Ramachandran & Amir , 2007 ; Ziebart et al. , 2008 ) . Although it is possible to extend our approach to any other IRL method , in this work we base on work on the maximum entropy ( MaxEnt ) framework ( Ziebart et al. , 2008 ) . In addition to allowing for sub-optimality in the expert demonstrations , MaxEnt-IRL can be re-framed as a maximum likelihood estimation problem . 4 ) . In part to combat the under-specified nature of IRL , prior work has often used low-dimensional linear parameterizations with handcrafted features ( Abbeel & Ng , 2004 ; Ziebart et al. , 2008 ) . In order to learn from high dimensional input , Wulfmeier et al . ( 2015 ) proposed applying fully convolutional networks ( Shelhamer et al. , 2017 ) to the MaxEnt IRL framework ( Ziebart et al. , 2008 ) for several navigation tasks ( Wulfmeier et al. , 2016a ; b ) . Other methods that have incorporated neural network rewards include guided cost learning ( GCL ) ( Finn et al. , 2017a ) , which uses importance sampling and regularization for scalability to high-dimensional spaces , and adversarial IRL ( Fu et al. , 2018 ) . Several other methods have also proposed imitation learning approaches based on adversarial frameworks that resemble IRL , but do not aim to directly recover a reward function ( Ho & Ermon , 2016 ; Li et al. , 2017 ; Hausman et al. , 2017 ; Kuefler & Kochenderfer , 2018 ) . In this work , instead of improving the ability to learn reward functions on a single task , we focus on the problem of effectively learning to use prior demonstration data from other IRL tasks , allowing us to learn new tasks from a limited number demonstrations even with expressive non-linear reward functions . Prior work has explored the problem of multi-task IRL , where the demonstrated behavior is assumed to have originated from multiple experts achieving different goals . Some of these approaches include those that aim to incorporate a shared prior over reward functions through extending the Bayesian IRL ( Ramachandran & Amir , 2007 ) framework to the multi-task setting ( Dimitrakakis & Rothkopf , 2012 ; Choi & Kim , 2012 ) . Other approaches have clustered demonstrations while simultaneously inferring reward functions for each cluster ( Babeş-Vroman et al. , 2011 ) or introduced regularization between rewards to a common “ shared reward ” ( Li & Burdick , 2017 ) . Our work is similar in that we also seek to encode prior information common to the tasks . However , a critical difference is that our method specifically aims to distill the meta-training tasks into a prior that can then be used to learn rewards for new tasks efficiently . The goal therefore is not to acquire good reward functions that explain the meta-training tasks , but rather to use them to learn efficiently on new tasks . Our approach builds on work on the broader problem of meta-learning ( Schmidhuber , 1987 ; Bengio et al . ; Naik & Mammone , 1992 ; Thrun & Pratt , 2012 ) and generative modelling ( Rezende et al. , 2016 ; Reed et al. , 2018 ; Mordatch , 2018 ) . Prior work has proposed a variety of solutions for learning to learn including memory based methods ( Duan et al. , 2016 ; Santoro et al. , 2016 ; Wang et al. , 2016 ; Mishra et al. , 2017 ) , methods that learn an optimizer and/or initialization ( Andrychowicz et al. , 2016 ; Ravi & Larochelle , 2016 ; Finn et al. , 2017a ; Li & Malik , 2017 ) , and methods that compare new datapoints in a learned metric space ( Koch , 2015 ; Vinyals et al. , 2016 ; Shyam et al. , 2017 ; Snell et al. , 2017 ) . Our work is motivated by the goal of broadening the applicability of IRL , but in principle it is possible to adapt many of these meta-learning approaches for our problem statement . We leave it to future work to do a comprehensive investigation of different meta-learning approaches which could broaden the applicability of IRL . 3 PRELIMINARIES AND OVERVIEW . In this section , we introduce our notation and describe the IRL and meta-learning problems . 3.1 LEARNING REWARDS VIA INVERSE REINFORCEMENT LEARNING . The standard Markov decision process ( MDP ) is defined by the tuple ( S , A , ps , r , γ ) where S and A denote the set of possible states and actions respectively , r : S × A → R is the reward function , γ ∈ [ 0 , 1 ] is the discount factor and ps : S ×S ×A → [ 0 , 1 ] denotes the transition distribution over the next state st+1 , given the current state st and current action at . Typically , the goal of “ forward ” RL is to maximize the expected discounted return R ( τ ) = ∑T t=1 γ t−1r ( st , at ) . In IRL , we instead assume that the reward function is unknown but that we instead have access to a set of expert demonstrations D = { τ1 , . . . , τK } , where τk = { s1 , a1 , . . . , sT , aT } . The goal of IRL is to recover the unknown reward function r from the set of demonstrations . We build on the maximum entropy ( MaxEnt ) IRL framework by Ziebart et al . ( 2008 ) , which models the probability of the trajectories as being distributed proportional to their exponentiated return p ( τ ) = 1 Z exp ( R ( τ ) ) , ( 1 ) where Z is the partition function , Z = ∫ τ exp ( R ( τ ) ) dτ . This distribution can be shown to be induced by the optimal policy in entropy regularized forward RL problem : π∗ =π Eτ∼π [ R ( τ ) − log π ( τ ) ] . ( 2 ) This formulation allows us to pose the reward learning problem as a maximum likelihood estimation ( MLE ) problem in an energy-based model rφ : min φ Eτ∼D [ LIRL ( τ ) ] =min φ Eτ∼D [ − log pφ ( τ ) ] . ( 3 ) Learning in general energy-based models of this form is common in many applications such as structured prediction . However , in contrast to applications where learning can be supervised by millions of labels ( e.g . semantic segmentation ) , the learning problem in Eq . 3 must typically be performed with a relatively small number of example demonstrations . In this work , we seek to address this issue in IRL by providing a way to integrate information from prior tasks to constrain the optimization in Eq . 3 in the regime of limited demonstrations . 3.2 META-LEARNING . The goal of meta-learning algorithms is to optimize for the ability to learn efficiently on new tasks . Rather than attempt to generalize to new datapoints , meta-learning can be understood as attempting to generalize to new tasks . It is assumed in the meta-learning setting that there are two sets of tasks that we refer to as the meta-training set { Ti ; i = 1 .. N } and meta-test set { Tj ; j = 1 .. M } , which are both drawn from a distribution p ( T ) . During meta-training time , the meta-learner attempts to learn the structure of the tasks in the meta-training set , such that when it is presented with a new test task , it can leverage this structure to learn efficiently from a limited number of examples . To illustrate this distinction , consider the case of few-shot learning setting . Let fθ denote the learner , and let a task be defined by learning from K training examples X trT = { x1 . . . , xK } , Y trT = { y1 . . . , yK } , and evaluating on K ′ test examples X testT = { x1 . . . , xK′ } , Y testT = { y1 . . . , yK′ } . One approach to meta-learning is to directly parameterize the meta-learner with an expressive model such as a recurrent or recursive neural network ( Duan et al. , 2016 ; Mishra et al. , 2017 ) conditioned on the task training data and the inputs for the test task : fθ ( Y |X testT , X trT , Y trT ) . Such a model is optimized using log-likelihood across all tasks . In this approach to meta-learning , since neural networks are known to be universal function approximators ( Siegelmann & Sontag , 1995 ) , any desired structure between tasks can be implicitly encoded . Rather than learn a single black-box function , another approach to meta-learning is to learn components of the learning procedure such as the initialization ( Finn et al. , 2017a ) or the optimization algorithm ( Ravi & Larochelle , 2016 ; Andrychowicz et al. , 2016 ) . In this work we extend the approach of model agnostic meta-learning ( MAML ) introduced by Finn et al . ( 2017a ) , which learns an initialization that is adapted by gradient descent . Concretely , in the supervised learning case , given a loss function L ( θ , XT , YT ) ( e.g . cross-entropy ) , MAML performs the following optimization min θ ∑ T L ( φT , X testT , Y testT ) = min θ ∑ T L ( θ − α∇θL ( θ , X trT , Y trT ) , X testT , Y testT ) , ( 4 ) where the optimization is over an initial set of parameters θ and the loss on the held out tasks XtestT becomes the signal for learning the initial parameters for gradient descent on X tr T . This optimization is analogous to adding a constraint in a multi-task setting , which we show in later sections is analogous in our setting to learning a prior over reward functions .

Summary: This paper attempts to the solve  data-set coverage issue common with Inverse reinforcement learning based approaches - by introducing a meta-learning framework trained on a smaller number of basic tasks. The primary insight here is that there exists a smaller set of unique tasks, the knowledge from which is transferable to new tasks and using these to learn an initial parametrized reward function improves the coverage for IRL. With experiments on the SpriteWorld synthetic data-set, the authors confirm this hypothesis and demonstrate performance benefits - showcasing better correlation with far fewer  number of demonstrations.