Few-Shot Intent Inference via Meta-Inverse Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) algorithms have the potential to automate a wide range of decisionmaking and control tasks across a variety of different domains , as demonstrated by successful recent applications ranging from robotic control ( Kober & Peters , 2012 ; Levine et al. , 2016 ) to game playing ( Mnih et al. , 2015 ; Silver et al. , 2016 ) . A key assumption of the RL problem statement is the availability of a reward function that accurately describes the desired tasks . For many real world tasks , reward functions can be challenging to manually specify , while being crucial to good performance ( Amodei et al. , 2016 ) . Most real world tasks are multifaceted and require reasoning over multiple factors in a task ( e.g . an autonomous vehicle navigating a city at night ) , while simultaneously providing appropriate reward shaping to make the task feasible with tractable exploration ( Ng et al. , 1999 ) . These challenges are compounded by the inherent difficulty of specifying rewards for tasks with high-dimensional observation spaces such as images . Inverse reinforcement learning ( IRL ) is an approach that aims to address this problem by instead inferring the reward function from demonstrations of the task ( Ng & Russell , 2000 ) . This has the appealing benefit of taking a data-driven approach to reward specification in place of hand engineering . In practice however , rewards functions are rarely learned as it can be prohibitively expensive to provide demonstrations that cover the variability common in real world tasks ( e.g. , collecting demonstrations of opening every type of door knob ) . In addition , while learning a complex function from high dimensional observations might make an expressive function approximator seem like a reasonable modelling assumption , in the “ few-shot ” domain it is notoriously difficult to unambiguously recover a good reward function with expressive function approximators . Prior solutions have thus instead relied on low-dimensional linear models with handcrafted features that effectively encode a strong prior on the relevant features of a task . This requires engineering a set of features by hand that work well for a specific problem . In this work , we propose an approach that instead explicitly learns expressive features that are robust even when learning with limited demonstrations . Our approach relies on the key observation that related tasks share common structure that we can leverage when learning new tasks . To illustrate , considering a robot navigating through a home . While the exact reward function we provide to the robot may differ depending on the task , there is a structure amid the space of useful behaviours , such as navigating to a series of landmarks , and there are certain behaviors we always want to encourage or discourage , such as avoiding obstacles or staying a reasonable distance from humans . This notion agrees with our understanding of why humans can easily infer the intents and goals ( i.e. , reward functions ) of even abstract agents from just one or a few demonstrations Baker et al . ( 2007 ) , as humans have access to strong priors about how other humans accomplish similar tasks accrued over many years . Similarly , our objective is to discover the common structure among different tasks , and encode the structure in a way that can be used to infer reward functions from a few demonstrations . More specifically , in this work we assume access to a set of tasks , along with demonstrations of the desired behaviors for those tasks , which we refer to as the meta-training set . From these tasks , we then learn a reward function parameterization that enables effective few-shot learning when used to initialize IRL in a novel task . Our method is summarized in Fig . 1 . Our key contribution is an algorithm that enables efficient learning of new reward functions by using meta-training to build a rich “ prior ” for goal inference . Using our proposed approach , we show that we can learn deep neural network reward functions from raw pixel observations with substantially better data efficiency than existing methods and standard baselines . 2 RELATED WORK . Inverse reinforcement learning ( IRL ) ( Ng & Russell , 2000 ) is the problem of inferring an expert ’ s reward function directly from demonstrations . Prior methods for performing IRL range from margin based approaches ( Abbeel & Ng , 2004 ; Ratliff et al. , 2006 ) to probabilistic approaches ( Ramachandran & Amir , 2007 ; Ziebart et al. , 2008 ) . Although it is possible to extend our approach to any other IRL method , in this work we base on work on the maximum entropy ( MaxEnt ) framework ( Ziebart et al. , 2008 ) . In addition to allowing for sub-optimality in the expert demonstrations , MaxEnt-IRL can be re-framed as a maximum likelihood estimation problem . 4 ) . In part to combat the under-specified nature of IRL , prior work has often used low-dimensional linear parameterizations with handcrafted features ( Abbeel & Ng , 2004 ; Ziebart et al. , 2008 ) . In order to learn from high dimensional input , Wulfmeier et al . ( 2015 ) proposed applying fully convolutional networks ( Shelhamer et al. , 2017 ) to the MaxEnt IRL framework ( Ziebart et al. , 2008 ) for several navigation tasks ( Wulfmeier et al. , 2016a ; b ) . Other methods that have incorporated neural network rewards include guided cost learning ( GCL ) ( Finn et al. , 2017a ) , which uses importance sampling and regularization for scalability to high-dimensional spaces , and adversarial IRL ( Fu et al. , 2018 ) . Several other methods have also proposed imitation learning approaches based on adversarial frameworks that resemble IRL , but do not aim to directly recover a reward function ( Ho & Ermon , 2016 ; Li et al. , 2017 ; Hausman et al. , 2017 ; Kuefler & Kochenderfer , 2018 ) . In this work , instead of improving the ability to learn reward functions on a single task , we focus on the problem of effectively learning to use prior demonstration data from other IRL tasks , allowing us to learn new tasks from a limited number demonstrations even with expressive non-linear reward functions . Prior work has explored the problem of multi-task IRL , where the demonstrated behavior is assumed to have originated from multiple experts achieving different goals . Some of these approaches include those that aim to incorporate a shared prior over reward functions through extending the Bayesian IRL ( Ramachandran & Amir , 2007 ) framework to the multi-task setting ( Dimitrakakis & Rothkopf , 2012 ; Choi & Kim , 2012 ) . Other approaches have clustered demonstrations while simultaneously inferring reward functions for each cluster ( Babeş-Vroman et al. , 2011 ) or introduced regularization between rewards to a common “ shared reward ” ( Li & Burdick , 2017 ) . Our work is similar in that we also seek to encode prior information common to the tasks . However , a critical difference is that our method specifically aims to distill the meta-training tasks into a prior that can then be used to learn rewards for new tasks efficiently . The goal therefore is not to acquire good reward functions that explain the meta-training tasks , but rather to use them to learn efficiently on new tasks . Our approach builds on work on the broader problem of meta-learning ( Schmidhuber , 1987 ; Bengio et al . ; Naik & Mammone , 1992 ; Thrun & Pratt , 2012 ) and generative modelling ( Rezende et al. , 2016 ; Reed et al. , 2018 ; Mordatch , 2018 ) . Prior work has proposed a variety of solutions for learning to learn including memory based methods ( Duan et al. , 2016 ; Santoro et al. , 2016 ; Wang et al. , 2016 ; Mishra et al. , 2017 ) , methods that learn an optimizer and/or initialization ( Andrychowicz et al. , 2016 ; Ravi & Larochelle , 2016 ; Finn et al. , 2017a ; Li & Malik , 2017 ) , and methods that compare new datapoints in a learned metric space ( Koch , 2015 ; Vinyals et al. , 2016 ; Shyam et al. , 2017 ; Snell et al. , 2017 ) . Our work is motivated by the goal of broadening the applicability of IRL , but in principle it is possible to adapt many of these meta-learning approaches for our problem statement . We leave it to future work to do a comprehensive investigation of different meta-learning approaches which could broaden the applicability of IRL . 3 PRELIMINARIES AND OVERVIEW . In this section , we introduce our notation and describe the IRL and meta-learning problems . 3.1 LEARNING REWARDS VIA INVERSE REINFORCEMENT LEARNING . The standard Markov decision process ( MDP ) is defined by the tuple ( S , A , ps , r , γ ) where S and A denote the set of possible states and actions respectively , r : S × A → R is the reward function , γ ∈ [ 0 , 1 ] is the discount factor and ps : S ×S ×A → [ 0 , 1 ] denotes the transition distribution over the next state st+1 , given the current state st and current action at . Typically , the goal of “ forward ” RL is to maximize the expected discounted return R ( τ ) = ∑T t=1 γ t−1r ( st , at ) . In IRL , we instead assume that the reward function is unknown but that we instead have access to a set of expert demonstrations D = { τ1 , . . . , τK } , where τk = { s1 , a1 , . . . , sT , aT } . The goal of IRL is to recover the unknown reward function r from the set of demonstrations . We build on the maximum entropy ( MaxEnt ) IRL framework by Ziebart et al . ( 2008 ) , which models the probability of the trajectories as being distributed proportional to their exponentiated return p ( τ ) = 1 Z exp ( R ( τ ) ) , ( 1 ) where Z is the partition function , Z = ∫ τ exp ( R ( τ ) ) dτ . This distribution can be shown to be induced by the optimal policy in entropy regularized forward RL problem : π∗ =π Eτ∼π [ R ( τ ) − log π ( τ ) ] . ( 2 ) This formulation allows us to pose the reward learning problem as a maximum likelihood estimation ( MLE ) problem in an energy-based model rφ : min φ Eτ∼D [ LIRL ( τ ) ] =min φ Eτ∼D [ − log pφ ( τ ) ] . ( 3 ) Learning in general energy-based models of this form is common in many applications such as structured prediction . However , in contrast to applications where learning can be supervised by millions of labels ( e.g . semantic segmentation ) , the learning problem in Eq . 3 must typically be performed with a relatively small number of example demonstrations . In this work , we seek to address this issue in IRL by providing a way to integrate information from prior tasks to constrain the optimization in Eq . 3 in the regime of limited demonstrations . 3.2 META-LEARNING . The goal of meta-learning algorithms is to optimize for the ability to learn efficiently on new tasks . Rather than attempt to generalize to new datapoints , meta-learning can be understood as attempting to generalize to new tasks . It is assumed in the meta-learning setting that there are two sets of tasks that we refer to as the meta-training set { Ti ; i = 1 .. N } and meta-test set { Tj ; j = 1 .. M } , which are both drawn from a distribution p ( T ) . During meta-training time , the meta-learner attempts to learn the structure of the tasks in the meta-training set , such that when it is presented with a new test task , it can leverage this structure to learn efficiently from a limited number of examples . To illustrate this distinction , consider the case of few-shot learning setting . Let fθ denote the learner , and let a task be defined by learning from K training examples X trT = { x1 . . . , xK } , Y trT = { y1 . . . , yK } , and evaluating on K ′ test examples X testT = { x1 . . . , xK′ } , Y testT = { y1 . . . , yK′ } . One approach to meta-learning is to directly parameterize the meta-learner with an expressive model such as a recurrent or recursive neural network ( Duan et al. , 2016 ; Mishra et al. , 2017 ) conditioned on the task training data and the inputs for the test task : fθ ( Y |X testT , X trT , Y trT ) . Such a model is optimized using log-likelihood across all tasks . In this approach to meta-learning , since neural networks are known to be universal function approximators ( Siegelmann & Sontag , 1995 ) , any desired structure between tasks can be implicitly encoded . Rather than learn a single black-box function , another approach to meta-learning is to learn components of the learning procedure such as the initialization ( Finn et al. , 2017a ) or the optimization algorithm ( Ravi & Larochelle , 2016 ; Andrychowicz et al. , 2016 ) . In this work we extend the approach of model agnostic meta-learning ( MAML ) introduced by Finn et al . ( 2017a ) , which learns an initialization that is adapted by gradient descent . Concretely , in the supervised learning case , given a loss function L ( θ , XT , YT ) ( e.g . cross-entropy ) , MAML performs the following optimization min θ ∑ T L ( φT , X testT , Y testT ) = min θ ∑ T L ( θ − α∇θL ( θ , X trT , Y trT ) , X testT , Y testT ) , ( 4 ) where the optimization is over an initial set of parameters θ and the loss on the held out tasks XtestT becomes the signal for learning the initial parameters for gradient descent on X tr T . This optimization is analogous to adding a constraint in a multi-task setting , which we show in later sections is analogous in our setting to learning a prior over reward functions .

Summary: This paper aims to address the problem of lacking sufficient demonstrations in inverse reinforcement learning (IRL) problems. They propose to take a meta learning approach, in which a set of i.i.d. IRL tasks are provided to the learner and the learner aims to learn a strategy to quickly recover a good reward function for a new task that is assumed to be sampled from the same task distribution. Particularly, they adopt the gradient-based meta learning algorithm, MAML, and the maximal entropy (MaxEnt) IRL framework, and derive the required meta gradient expression for parameter update. The proposed algorithm is evaluated on a synthetic grid-world problem, SpriteWorld. The experimental results suggest the proposed algorithm can learn to mimic the optimal policy under the true reward function that is unknown to the learner.