Abstract: A significant challenge for the practical application of reinforcement learning to real world problems is the need to specify an oracle reward function that correctly defines a task. Inverse reinforcement learning (IRL) seeks to avoid this challenge by instead inferring a reward function from expert demonstrations. While appealing, it can be impractically expensive to collect datasets of demonstrations that cover the variation common in the real world (e.g. opening any type of door). Thus in practice, IRL must commonly be performed with only a limited set of demonstrations where it can be exceedingly difficult to unambiguously recover a reward function. In this work, we exploit the insight that demonstrations from other tasks can be used to constrain the set of possible reward functions by learning a "prior" that is specifically optimized for the ability to infer expressive reward functions from limited numbers of demonstrations. We demonstrate that our method can efficiently recover rewards from images for novel tasks and provide intuition as to how our approach is analogous to learning a prior.

Kelvin Xu
Ellis Ratner
Anca Dragan
Sergey Levine
Chelsea Finn

Learning a Prior over Intent via Meta-Inverse Reinforcement Learning

Introduction Reinforcement learning (RL) algorithms have the potential to automate a wide range of decision-making and control tasks across a variety of different domains, as demonstrated by successful recent applications ranging from robotic control  (Kober & Peters, 2012; Levine et al., 2016)  to game playing  (Mnih et al., 2015; Silver et al., 2016) . A key assumption of the RL problem statement is the availability of a reward function that accurately describes the desired task. For many real world tasks, reward functions can be chal-Figure  1 . A diagram of our meta-inverse RL approach. Our approach attempts to remedy over-fitting in few-shot IRL by learning a "prior" that constraints the set of possible reward functions to lie within a few steps of gradient descent. Standard IRL attempts to recover the reward function directly from the available demonstrations. The shortcoming of this approach is that there is little reason to expect generalization as it is analogous to training a density model with only a few examples. lenging to manually specify, while being crucial for good performance  (Amodei et al., 2016) . Most real world tasks are multifaceted and require reasoning over multiple factors in a task (e.g. a robot cleaning in a house with children), while simultaneously providing appropriate reward shaping to make the task feasible with tractable exploration  (Ng et al., 1999) . These challenges are compounded by the inherent difficulty of specifying rewards for tasks with highdimensional observation spaces such as images. Inverse reinforcement learning (IRL) is an approach that aims to address this problem by instead inferring the reward function from demonstrations of the task  (Ng & Russell, 2000) . This has the appealing benefit of taking a datadriven approach to reward specification in place of hand engineering. In practice however, rewards functions are rarely learned as it can be prohibitively expensive to provide demonstrations that cover the variability common in real world tasks (e.g., collecting demonstrations of opening every type of door knob). In addition, while learning a complex function from high dimensional observations might make an expressive function approximator seem like a reasonable modelling assumption, in the "few-shot" domain it is notoriously difficult to unambiguously recover a good reward function with expressive function approximators. Many prior approaches have thus relied on low-dimensional linear models with handcrafted features that effectively encode a strong prior on the relevant features of a task. This requires engineering a set of features by hand that work well for a specific problem. In this work, we propose an approach that instead explicitly learns expressive features that are robust even when learning with limited demonstrations. Our approach relies on the key observation that related tasks share a common structure that we can leverage when learning new tasks. To illustrate, considering a robot navigating through a home. While the exact reward function we provide to the robot may differ depending on the task, there is a structure amid the space of useful behaviours, such as navigating to a series of landmarks, and there are certain behaviors we always want to encourage or discourage, such as avoiding obstacles or staying a reasonable distance from humans. This notion agrees with our understanding of why humans can easily infer the intents and goals (i.e., reward functions) of even abstract agents from just one or a few demonstrations  (Baker et al., 2007) , as humans have access to strong priors about how other humans accomplish similar tasks accrued over many years. Similarly, our objective is to discover the common structure among different tasks, and encode that structure in a way that can be used to infer reward functions from a few demonstrations. More specifically, in this work we assume access to a set of tasks, along with demonstrations of the desired behaviors for those tasks, which we refer to as the meta-training set. From these tasks, we then learn a reward function parameterization that enables effective few-shot learning when used to initialize IRL in a novel task. Our method is summarized in Fig.  1 . Our key contribution is an algorithm that enables efficient learning of new reward functions by using metatraining to build a rich "prior" for goal inference. Using our proposed approach, we show that we can learn deep neural network reward functions from raw pixel observations on two distinct domains with substantially better data efficiency than existing methods and standard baselines.
 Related Work Inverse reinforcement learning (IRL)  (Ng & Russell, 2000)  is the problem of inferring an expert's reward function directly from demonstrations. Prior methods for performing IRL range from margin based approaches  (Abbeel & Ng, 2004; Ratliff et al., 2006)  to probabilistic approaches  (Ramachandran & Amir, 2007; Ziebart et al., 2008) . Although it is possible to extend our approach to any other IRL method, in this work we base on work on the maximum entropy (MaxEnt) framework  (Ziebart et al., 2008) . In addition to allowing for sub-optimality in the expert demonstrations, MaxEnt-IRL can be re-framed as a maximum likelihood estimation problem. (Sec. 3). In part to combat the under-specified nature of IRL, prior work has often used low-dimensional linear parameterizations with handcrafted features  (Abbeel & Ng, 2004; Ziebart et al., 2008) . In order to learn from high dimensional input,  Wulfmeier et al. (2015)  proposed applying fully convolutional networks  (Shelhamer et al., 2017)  to the MaxEnt IRL framework  (Ziebart et al., 2008)  for several navigation tasks  (Wulfmeier et al., 2016b; a) . Other methods that have incorporated neural network rewards include guided cost learning (GCL)  (Finn et al., 2017a) , which uses importance sampling and regularization for scalability to highdimensional spaces, and adversarial IRL  (Fu et al., 2018) . Several other methods have also proposed imitation learning approaches based on adversarial frameworks that resemble IRL, but do not aim to directly recover a reward function  (Ho & Ermon, 2016; Hausman et al., 2017; Kuefler & Kochenderfer, 2018) . In this work, instead of improving the ability to learn reward functions on a single task, we focus on the problem of effectively learning to use prior demonstration data from other IRL tasks, allowing us to learn new tasks from a limited number demonstrations even with expressive non-linear reward functions. Prior work has explored the problem of multi-task IRL, where the demonstrated behavior is assumed to have originated from multiple experts achieving different goals. Some of these approaches include those that aim to incorporate a shared prior over reward functions through extending the Bayesian IRL  (Ramachandran & Amir, 2007)  framework to the multi-task setting  (Dimitrakakis & Rothkopf, 2012; Choi & Kim, 2012) . Other approaches have clustered demonstrations while simultaneously inferring reward functions for each cluster  (Babes ¸-Vroman et al., 2011)  or introduced regularization between rewards to a common "shared reward"  (Li & Burdick, 2017) . Our work is similar in that we also seek to encode prior information common to the tasks. However, a critical difference is that our method specifically aims to distill the meta-training tasks into a prior that can then be used to learn rewards for new tasks efficiently. The goal therefore is not to acquire good reward functions that explain the meta-training tasks, but rather to use them to learn efficiently on new tasks. Our approach builds on work on the broader problem of meta-learning  (Schmidhuber, 1987; Bengio et al.; Naik & Mammone, 1992; Thrun & Pratt, 2012)  and generative modelling  (Rezende et al., 2016; Reed et al., 2018; Mordatch, 2018) . Prior work has proposed a variety of solutions for learning to learn including memory based methods  (Duan et al., 2016; Santoro et al., 2016; Mishra et al., 2017) , methods that learn an optimizer and/or initialization  (Andrychowicz et al., 2016; Ravi & Larochelle, 2016; Finn et al., 2017a; Li & Malik, 2017) , and methods that compare new datapoints in a learned metric space  (Koch, 2015; Wang & Hebert, 2016; Vinyals et al., 2016; Shyam et al., 2017; Snell et al., 2017) . Our work is motivated by the goal of broadening the applicability of IRL, but in principle it is possible to adapt many of these metalearning approaches for our problem statement. We build upon  Finn et al. (2017a) , which has also been previously applied to the related problems of imitation learning and human motion prediction  Finn et al., 2017b; Alet et al., 2018) . We leave it to future work to do a comprehensive investigation of different meta-learning approaches which could broaden the applicability of IRL.
 Preliminaries and Overview In this section, we introduce our notation and describe the IRL and meta-learning problems.
 Learning Rewards via Maximum Entropy Inverse Reinforcement Learning The standard Markov decision process (MDP) is defined by the tuple (S, A, p s , r, ) where S and A denote the set of possible states and actions respectively, r : S ⇥ A ! R is the reward function, 2 [0, 1] is the discount factor and p s : S ⇥ S ⇥ A ! [0, 1] denotes the transition distribution over the next state s t+1 , given the current state s t and current action a t . Typically, the goal of "forward" RL is to maximize the expected discounted return R(⌧ ) = P T t=1 t 1 r(s t , a t ). In IRL, we instead assume that the reward function is unknown but that we instead have access to a set of expert demonstrations D = {⌧ 1 , . . . , ⌧ K }, where ⌧ k = {s 1 , a 1 , . . . , s T , a T }. The goal of IRL is to recover the unknown reward function r from the set of demonstrations. We build on the maximum entropy (MaxEnt) IRL framework by  Ziebart et al. (2008) , which models trajectories as being distributed proportional to their exponentiated return p(⌧ ) = 1 Z exp (R(⌧ )) , (1) where Z is the partition function, Z = R ⌧ exp(R(⌧ ))d⌧ . This distribution can be shown to be induced by the optimal policy in entropy regularized forward RL problem: ⇡ ⇤ = arg max ⇡ E ⌧ ⇠⇡ [R(⌧ ) log ⇡(⌧ )] . (2) This formulation allows us to pose the reward learning problem as a maximum likelihood estimation (MLE) problem in an energy-based model r by defining the following loss: min E ⌧ ⇠D [L IRL (⌧ )] := min E ⌧ ⇠D [ log p (⌧ )] . (3) Learning in general energy-based models of this form is common in many applications such as structured prediction. However, in contrast to applications where learning can be supervised by millions of labels (e.g. semantic segmentation), the learning problem in Eq. 3 must typically be performed with a relatively small number of example demonstrations. In this work, we seek to address this issue in IRL by providing a way to integrate information from prior tasks to constrain the optimization in Eq. 3 in the regime of limited demonstrations.
 Meta-Learning The goal of meta-learning algorithms is to optimize for the ability to learn efficiently on new tasks. Rather than attempting to generalize to new datapoints, meta-learning can be understood as attempting to generalize to new tasks. It is assumed in the meta-learning setting that there are two disjoint sets of tasks that we refer to as the meta-training set {T i ; i = 1..N } and meta-test set {T j ; j = 1..M }, which are both drawn from a distribution p(T ). During meta-training time, the meta-learner attempts to learn the structure of the tasks in the meta-training set, such that when it is presented with a test task, it can leverage this structure to learn efficiently from a limited number of examples. To illustrate this distinction, consider the case of fewshot learning setting. Let f ✓ denote the learner, and let a task be defined by learning from K training examples X tr T = {x 1 . . . , x K }, Y tr T = {y 1 . . . , y K }, and evaluat- ing on K 0 test examples X test T = {x 1 . . . , x K 0 }, Y test T = {y 1 . . . , y K 0 }. One approach to meta-learning is to directly parameterize the meta-learner with an expressive model such as a recurrent or recursive neural network  (Duan et al., 2016; Mishra et al., 2017)  conditioned on the task training data and the inputs for the test task: f ✓ (Y |X test T , X tr T , Y tr T ) . Such a model is optimized using log-likelihood across all tasks. In this approach to meta-learning, since neural networks are known to be universal function approximators  (Siegelmann & Sontag, 1995) , any desired structure between tasks can be implicitly encoded. Rather than learn a single black-box function, another approach to meta-learning is to learn components of the learning procedure such as the initialization  (Finn et al., 2017a)  or the optimization algorithm  (Ravi & Larochelle, 2016; Andrychowicz et al., 2016) . In this work we extend the approach of model agnostic meta-learning (MAML) introduced by  Finn et al. (2017a) , which learns an initialization that is adapted by gradient descent. Concretely, in the supervised learning case, given a loss function L(✓, X T , Y T ) (e.g. cross-entropy), MAML performs the following optimization min ✓ X T L( T , X test T , Y test T ) = min ✓ X T L ✓ ↵r ✓ L(✓, X tr T , Y tr T ), X test T , Y test T , (4) where the optimization is over an initial set of parameters ✓ and the loss on the held out tasks X test T becomes the signal for learning the initial parameters for gradient descent (with step size ↵) on X tr T . This optimization is analogous to adding a constraint in a multi-task setting, which we show in later sections is analogous in our setting to learning a prior over reward functions.
 Learning to Learn Rewards Our goal in meta-IRL is to learn how to learn reward functions across many tasks such that the model can infer the reward function for a new task using only one or a few expert demonstrations. Intuitively, we can view this problem as aiming to learn a prior over the rewards of expert demonstrators, such that when given just one or a few demonstrations of a new task, we can combine the learned prior with the new data to effectively determine the expert's intent. Such a prior is helpful in inverse reinforcement learning settings, since the space of reward functions with are relevant to particular task is much smaller than the space of all possible rewards definable on the raw observations. During meta-training, we have a set of tasks {T i ; i = 1..N }. Each task T i has a set of demonstrations D T = {⌧ 1 , . . . , ⌧ K } from an expert policy which we partition into disjoint D tr T and D test T sets. The demonstrations for each meta-training task are assumed to be produced by the expert according to the maximum entropy model in Section 3.1. During meta-training, these tasks will be used to encodes common structure so that our model can quickly acquire rewards for new tasks from just a few demonstrations. After meta-training, our method is presented with a new task. During this meta-test phase, the algorithm must infer the parameters of the reward function r (s t , a t ) for the new task from a few demonstrations. As is standard in meta-learning, we assume that the test task is from the same distribution of tasks seen during meta-training, a distribution that we denote as p(T ).
 Meta Reward and Intention Learning (MandRIL) In order to meta-learn a reward function that can act as a prior for new tasks and new environments, we first formalize the notion of a good reward by defining a loss L T (✓) on the reward function r ✓ for a particular task T . We use the MaxEnt IRL loss L IRL discussed in Section 3, which, for a given D T , leads to the following gradient  (Ziebart et al., 2008) : r ✓ L T (✓) = @r ✓ @✓ ⇥ E ⌧ [µ ⌧ ] µ D T ⇤ . ( 5 ) where µ ⌧ are the state-action visitations under the optimal maximum entropy policy under r ✓ , and µ D are the mean state visitations under the demonstrated trajectories. If our end goal were to achieve a single reward function that # Compute Max-Ent state visitations 7: E ⌧ [µ ⌧ ] = STATE-VISITATIONS-POLICY(r ✓ , T ) 8: # MaxEntIRL gradient  (Ziebart et al., 2008)  9: @L @r ✓ = E ⌧ [µ ⌧ ] µ D 10: Return @L @r ✓ 11: 12: Randomly initialize ✓ 13: while not done do 14: Sample batch of tasks T i ⇠ {T } meta-train 15: for all T i do 16: Sample demos D tr = {⌧ 1 , . . . , ⌧ K } ⇠ T i 17: # Inner loss computation 18: @L tr T i (✓) @r ✓ = MAXENTIRL-GRAD(r ✓ , T i , D tr ) 19: Compute r ✓ L tr Ti (✓) from @L tr T i (✓) @r ✓ 20: Compute Ti = ✓ ↵r ✓ L tr Ti (✓) 21: Sample demos D test = {⌧ 0 1 , . . . , ⌧ 0 K 0 } ⇠ T i 22: # Outer loss computation 23: @L test T i @r ✓ = MAXENTIRL-GRAD(r T i , T i , D test )) 24: # Compute meta-gradient 25: Compute r ✓ L test Ti from @L test T i @r ✓ via chain rule 26: Compute update to ✓ ✓ P i r ✓ L test Ti works as well as possible across all tasks in {T i ; i = 1..N }, then we could simply follow the mean gradient across all tasks. However, our objective is different: instead of optimizing performance on the meta-training tasks, we aim to learn a reward function that can be quickly and efficiently adapted to new tasks at meta-test time. In doing so, we aim to encode prior information over the task distribution in this learned reward prior. We propose to implement such a learning algorithm by finding the parameters ✓, such that starting from ✓ and taking a small number of gradient steps on a few demonstrations from given task leads to a reward function for which a set of test demonstrations have high likelihood, with respect to the MaxEnt IRL model. In particular, we would like to find a ✓ such that the parameters T = ✓ ↵r ✓ L tr T (✓) (6) lead to a reward function r T for task T , such that the IRL loss (corresponding to negative log-likelihood) for a disjoint set of test demonstrations, given by L T ,test IRL , is minimized. The corresponding optimization problem for ✓ can therefore be written as follows: min ✓ N X i=1 L test Ti ( Ti ) = N X i=1 L test Ti ✓ ↵r ✓ L tr Ti (✓) . (7) Our method acquires this prior ✓ over rewards in the task distribution p(T ) by optimizing this loss. This amounts to an extension of the MAML algorithm in Section 3.2 to the inverse reinforcement learning setting. This extension is quite challenging, because computing the MaxEnt IRL gradient requires repeatedly solving for the current maximum entropy policy and visitation frequencies, and the MAML objective requires computing derivatives through this gradient step. Next, we describe in detail how this is done. An overview of our method is also outlined in Alg. 1.
 Meta-training. The computation of the meta-gradient for the objective in Eq. 7 can be conceptually separated into two parts. First, we perform the update in Eq. 6 by computing the expected state visitations µ, which is the expected number of times an agent will visit each state. We denote this overall procedure as STATE-VISITATIONS-POLICY, and follow  Ziebart et al. (2008)  by first computing the maximum entropy optimal policy in Eq. 2 under the current r ✓ , and then approximating µ using dynamic programming. Next, we compute the state visitation distribution of the expert using a procedure which we denote as STATE-VISITATIONS-TRAJ. This can be done either empirically, by averaging the state visitation of the experts demonstrations, or by using STATE-VISITATIONS-POLICY if the true reward is available at meta-training time. This allows us to recover the IRL gradient according to Eq. 5, which we can then apply to compute T according to Eq. 6. Second, we need to differentiate through this update to compute the gradient of the meta-loss in Eq. 7. Note that the meta-loss itself is the IRL loss evaluated with a different set of test demonstrations. We follow the same procedure as above to evaluate the gradient of L T ,test IRL with respect to the post-update parameters T , and then apply the chain rule to compute the meta-gradient: r ✓ L test T (✓) = @ @✓ (✓ ↵r ✓ L tr T (✓)) @r T @ T @L test T @r T = I ↵ @ 2 L tr T (✓) @✓ 2 ↵ @ r ✓ @ ✓ D @ r ✓ @ ✓ > ! @r T @ T @L test T @r T (8) where on the second line we differentiate through the MaxEnt-IRL update, and we define the |S||A|-dimensional diagonal matrix D as  D := diag ⇢ @ @ r ✓,i (E ⌧ [µ ⌧ ]) i |S||A| i=1 ! . a 1 a 2 s 1 s 2 . . . . . . (⌧ ) = 1 Z Q T t=1 r( T , st, at) dyn(st+1, st, at) (above) where we learn a prior over T , which during meta-test is used for MAP inference over new expert demonstrations. A detailed derivation of this expression is provided in the supplementary Appendix E. Meta-testing. Once we have acquired the meta-trained parameters ✓ that encode a prior over p(T ), we can leverage this prior to enable fast, few-shot IRL of novel tasks in {T j ; j = 1..M }. For each task, we first compute the state visitations from the available set of demonstrations for that task. Next, we use these state visitations to compute the gradient, which is the same as the inner loss gradient computation of the meta-training loop in Alg. 1. We apply this gradient to adapt the parameters ✓ to the new task. Even if the model was trained with only one inner gradient steps, we found in practice that it was beneficial to take substantially more gradient steps during meta-testing; performance continued to improve with up to 20 steps.
 Connection to Learning a Prior over Intent The objective in Eq. 6 optimizes for parameters that enable the reward function to generalize efficiently on a wide range of tasks. Intuitively, constraining the space of reward functions to lie within a few steps of gradient descent can be interpreted as expressing a "locality" prior over reward function parameters. This intuition can be made more concrete by the following analysis. By viewing IRL as maximum likelihood estimation in a particular graphical model (Fig.  2 ), we can take the perspective of  Grant et al. (2018)  who showed that for a linear model, fast adaptation via a few steps of gradient descent in MAML is performing MAP inference over , under a Gaussian prior with the mean ✓ and a covariance that depends on the step size, number of steps and hessian of the loss. This is based on the connection between early stopping and regularization previously discussed in  Santos (1996) , which we refer the readers to for a more detailed discussion. The interpretation of MAML as imposing a Gaussian prior on the parameters is exact in the case of a likelihood that is quadratic in the parameters (such as the log-likelihood of a Gaussian in terms of its mean). For any non-quadratic likelihood, this is an approximation in a local neighborhood around ✓ (i.e. up to convex quadratic approximation). In the case of complex parameterizations, such as deep function approximators, this is a coarse approximation and unlikely to be the mode of a posterior. However, we can still frame the effect of early stopping and initialization as serving as a prior in a similar way as prior work  (Sjöberg & Ljung, 1995; Duvenaud et al., 2016; Grant et al., 2018) . More importantly, this interpretation hints at future extensions to our approach that could benefit from employing more fully Bayesian approaches to reward and goal inference.
 Experiments Our evaluation seeks to answer two questions. First, we aim to test our core hypothesis that using prior task experience enables reward learning for new tasks with just a few demonstrations. Second, we compare our method with alternative approaches that make use of multi-task experience. We test our core hypothesis by comparing learning performance on a new tasks starting from the initialization produced by MandRIL with learning a separate model for every task starting either from a random initialization or from an initialization obtained by supervised pre-training. We refer to these approaches as learning "FROM SCRATCH" and "AVERAGE GRADIENT" pretraining respectively. Our supervised pre-training baseline follows the average gradient during meta-training tasks and finetunes at meta-test time (as discussed in Section 4). Unlike our method, supervised pre-training does not optimize for a model that performs well under fine tuning, but does use the same prior data to pre-train. We additionally compare to pre-training on a single task as well as all the meta-training tasks. To our knowledge, there is no prior work that addresses the specific meta-inverse reinforcement learning problem introduced in this paper. Thus, to provide a point of comparison and calibrate the difficulty of the tasks, we adapt two alternative black-box meta-learning methods to the IRL setting. The comparisons to both of the black-box methods described below evaluate the importance of incorporating the IRL gradient into the meta-learning process, rather than learning the adaptation process entirely from scratch. Demo conditional model: Our method implicitly conditions on the demonstrations through the gradient update. In principle, a conditional deep model with sufficient capacity could implicitly implement a similar learning rule. Thus, we consider a conditional model (often referred to as a "contextual model"  (Finn et al., 2017b) ), which receives the demonstration as an additional input. Recurrent meta-learner: We additionally compare to an RNN-based meta-learner  (Santoro et al., 2016; Duan et al., 2017) . Specifically, we implement a conditional model by feeding both images and sequences of states visited by the demonstrations to an LSTM. When learning a task, the agent has access to the image (left) and demonstrations (red arrows). To evaluate learning (right), the agent is tested for its ability to recover the reward for the task when the objects have been rearranged. The reward structure we wish to capture can be illustrated by considering the initial state in blue. An policy acting optimally under a correctly inferred reward should interpret the other objects as obstacles, and prefer a path on dirt. We consider two environments: (1) an image-based navigation task with an aerial viewpoint, (2) a first-person navigation task in a simulated home environment with object interaction. We describe here the environments and evaluation protocol and provide detailed experimental settings and hyperparameters for both domains in Appendices A and B. (1) SpriteWorld navigation domain. Since most prior IRL work (and multi-task IRL work) studied settings where linear reward function approximators suffice (i.e., lowdimensional state spaces and hand-designed features), we design an experiment that is significantly more challengingthat requires learning rewards on raw pixels. We consider a navigation problem where we must learn a convolutional neural network that directly maps image pixels to rewards. We introduce a family of tasks called "SpriteWorld." Some example tasks are shown in Fig.  3 . Tasks involve navigating to goal objects while exhibiting preference over terrain types (e.g., the agent prefers to traverse dirt tiles over traversing grass tiles). At meta-test time, we provide one or a few demonstrations in a single training environment and evaluate the reward learned using these demonstrations in a new, test environment that contains the same objects as the training environment, but arranged differently. Evaluating in a new test environment is critical to measure that, after adapting to the training environment from a few demonstrations, the reward learned the correct visual cues, rather than simply memorizing the demonstration trajectory. We generate unique tasks in this domain as follows. First, we randomly choose a set of three sprites from one hundred sprites from the original game (creating a total of 161,700 unique tasks). We randomly place these three sprites within a randomly generated terrain tiling; we designate one of the sprites to be the goal landmark of the navigation task. The other two objects are treated as obstacles for which the agent incurs a large negative reward for not avoiding. In each task, we optimize our model on a training world and generalization in a test world, as described below. The agent must complete either a "NAV" task (blue line) where the goal is to navigate from the start to the cup or a "PICK" task (blue + green line) where the agent must also bring the cup to the bed. The agent's observation is a panoramic first-person viewpoint (see bottom left for RGB). Following the convention in prior work  (Fu et al., 2019) , we provide to the reward function the corresponding semantic images (bottom right). These images are 32 ⇥ 24 containing 61 channels corresponding to each object class. (2) SUNCG navigation domain: In addition to the SpriteWorld domain, we evaluate our approach on a first person image-based navigation task in an indoor house environment where the agent must interact with objects. We use an environment built on top of the SUNCG dataset  (Song et al., 2017)  which has previously been used in the context of IRL  (Fu et al., 2019)  with language instructions. We follow a similar task setup as  Fu et al. (2019) , although we omit the language instructions. In this domain, we consider tasks that can be categorized into two types.
 Navigation (NAV): In this task, the agent must navigate to a location in the house that corresponds either to a target object or location. For example, in the blue line of Fig.  4 , the agent must navigate to the "cup" object. Pick-and-place (PICK): In this more difficult task, the agent moves an object between two locations. For example, in Fig  4 , the agent must navigate to the "cup", perform a pick action and then navigate to the "bedroom". Evaluation protocol: We evaluate on held-out tasks that were unseen during meta-training. In the SpriteWorld domain, we consider two settings: (1) tasks involving new combinations and placements of sprites, with sprites that were present during meta-training, and (2) tasks with combinations of unseen sprites which we refer to as "out of domain objects." For each task, we generate one environment (a set of sprite positions) along with demonstrations for adapting the reward, and generate a second environment (with new sprite positions) for evaluating the adapted reward. In the SUNCG domain, we similarly evaluate on both novel combinations of objects and locations. We follow the evaluation protocol of  Fu et al. (2019)  and evaluate on "TEST" tasks which consist of tasks within the same houses as training, but with novel combinations of objects and locations. In addition, we evaluate on environments which consists of new houses not in the training set. We refer to these as "UNSEEN-HOUSES." This evaluation adds complexity by testing the models ability to successfully infer rewards in an entirely new scene. In total, the dataset consists of 1413 tasks (716 PICK, 697 NAV). The meta-train set is composed of 1004 tasks, the "TEST" set contains 236 tasks, and the "UNSEEN-HOUSES" set contains 173 tasks. Evaluation Metrics. We measure performance using the expected value difference, which measures the suboptimality of a policy learned under the learned reward; this is a performance metric used in prior IRL work  (Levine et al., 2011; Wulfmeier et al., 2015) . The metric is computed by taking the difference between the value of the optimal policy under the learned reward and the value of the optimal policy under the true reward. On the SUNCG domain, we follow  Fu et al. (2019)  and report the success rate of the optimal policy under the learned reward function. . We find that pre-training on the full set of tasks leads to negative transfer, while pre-training on a single task is comparable to random initialization. ManDRIL outperforms both alternative initialization approaches, which shows that optimizing for initial weights for fine-tuning robustly improves performance. Shaded regions show 95% confidence intervals. Results. The results for SpriteWorld are shown in Fig.  6 , which illustrate test performance with in-distribution and out-of-distribution sprites. Our approach, MandRIL, achieves consistently better performance in both settings. Most significantly, our approach performs well even with single-digit numbers of demonstrations. By comparison, alternative meta-learning methods generally overfit considerably, attaining good training performance (see Appendix. C for curves) but poor test performance. Learning the reward function from scratch is in fact the most competitive baseline -as the number of demonstrations increases, simply training the reward function from scratch on the new task is the only method that matches the performance of MandRIL when provided 20 or more demonstrations. With only a few demonstrations however, MandRIL has substantially lower value difference. It is worth noting the performance of MandRIL on the out of distribution test setting (Fig.  6 , bottom): although the evaluation is on new sprites, Man-dRIL is still able to adapt via gradient descent and exceed the performance all other methods. In both domains, we perform a comparison to representations finetuned from a supervised pre-training phase in Fig.  6  and Table  5 . We compare against an approach that follows the mean gradient across the tasks at meta-training time and is fine-tuned at meta-test time which we find consistently leads to negative transfer. We conclude that fine tuning reward functions learned in this manner is not an effective way of using prior task information. In contrast, we find that our approach, which explicitly optimizes for initial weights for fine-tuning, robustly improves performance on all task types and test settings. By visualizing the value under the learned reward function (see Fig.  5 ), we see that even with a small number of gradient steps, the reward function can be effectively adapted to an unseen home layout. Note that the SUNCG task is substantially more challenging, requiring the reward function to interpret first-person images. Indeed, the pretrained MaxEntIRL algorithm that does not use meta-learning exhibits negative transfer, as illustrated by the lower performance of this method on all tasks as compared to the learning "FROM SCRATCH" version, which learns each task entirely from random initialization. Training from scratch is a strong baseline here, because the method still sees every single first-person image in the house -2257.7 images on average. This provides sufficient variety to learn effective visual features in many cases. Nonetheless, our method (last row in Table  1 ) produces a substantial improvement, especially on the much harder "PICK" task, demonstrating that meta-learning can produce positive transfer even when pre-training does not.
 Conclusion In this work, we present an approach that enables few-shot learning of reward functions. We achieve this through a novel formulation of IRL that learns to encode common structure across tasks. Using our meta-IRL approach, we show that we can leverage data from previous tasks to effectively learn reward functions from raw pixel observations for new tasks, from only a handful of demonstrations. Our work paves the way for future work that considers unknown dynamics, or work that employs more fully probabilistic approaches to reward and goal inference. Algorithm 1 1 Meta Reward and Intention Learning (Man- dRIL)    1: Input: Set of meta-training tasks {T } meta-train 2: Input: hyperparameters ↵, 3: function MAXENTIRL-GRAD(r ✓ , T , D)
 Figure 2 . 2 Figure2. Our approach can be understood as approximately learning a distribution over the demonstrations ⌧ , in the factor graph p(⌧ ) = 1
 Figure 3 . 3 Figure 3. An example task on the SpriteWorld domain. When learning a task, the agent has access to the image (left) and demonstrations (red arrows).To evaluate learning (right), the agent is tested for its ability to recover the reward for the task when the objects have been rearranged. The reward structure we wish to capture can be illustrated by considering the initial state in blue. An policy acting optimally under a correctly inferred reward should interpret the other objects as obstacles, and prefer a path on dirt.
 Figure 4 . 4 Figure 4. An example task in the SUNCG environment (top).The agent must complete either a "NAV" task (blue line) where the goal is to navigate from the start to the cup or a "PICK" task (blue + green line) where the agent must also bring the cup to the bed. The agent's observation is a panoramic first-person viewpoint (see bottom left for RGB). Following the convention in prior work (Fu et al., 2019) , we provide to the reward function the corresponding semantic images (bottom right). These images are 32 ⇥ 24 containing 61 channels corresponding to each object class.
 Figure 5 . 5 Figure 5. An example adaptation in an UNSEEN-HOUSE (best viewed in color). The agent starts in one room (blue square) and is required to pick up the vase (green square) and take it to the living room (red square). The value function (blue is high, red is low) under the learned reward (bottom) exhibits no "PICK" task structure pre-adaptation (bottom left-column). Post-adaptation (bottom right-column), the reward function successfully leads the agent to the vase (bottom figure, top-right plot) and after the pick action (bottom figure, bottom-right plot) is performed, navigates the agent to the goal location.
 Figure 6 . 6 Figure6. Meta-test performance on the SpriteWorld domain (lower is better): held-out tasks performance (top) and held-out tasks with novel sprites (bottom). The recurrent meta-learner has a value difference > 60 in both test settings. In both test settings, Man-dRIL achieves comparable performance to the training environment, while the other methods overfit until they receive at least 10 demonstrations (see Appendix C for training environment performance). We find that pre-training on the full set of tasks leads to negative transfer, while pre-training on a single task is comparable to random initialization. ManDRIL outperforms both alternative initialization approaches, which shows that optimizing for initial weights for fine-tuning robustly improves performance. Shaded regions show 95% confidence intervals.
 Table 1 . 1 Success rate (%) on heldout tasks with 5 demonstrations. ManDRIL achieves consistently better performance on all task/environment types. Results are averaged over 3 random seeds. METHOD TEST PICK NAV TOTAL PICK NAV TOTAL UNSEEN HOUSES BEHAVIORAL CLONING 0.4 8.2 4.3 3.7 12.0 9.4 MAXENT IRL (AVG GRADIENT) 37.3 83.7 60.8 38.3 89.7 73.3 MAXENT IRL (FROM SCRATCH) 42.4 87.9 65.4 48.1 89.9 76.5 MANDRIL(OURS) 52.3 90.7 77.3 56.3 91.0 82.6 MANDRIL (PRE-ADAPTATION) 6.0 35.3 20.7 4.3 34.6 25.3
			 Department of Electrical Engineering and Computer Science, University of California, Berkeley, USA. Correspondence to: Kelvin Xu <<EMAIL>.


Abstract: Once isolated behind safety fences, robots are now making their way into spaces shared with people: not only on manufacturing production lines, but also our houses, museums, or hospitals. In these spaces, collaboration between humans and robots becomes vital, but also raises new challenges for robotics algorithms. A collaborative robot must be able to assist humans in a large diversity of tasks, understand its collaborator's intentions as well as communicate its own, predict human actions to adapt its behavior accordingly, and decide when it can lead the task or instead follow the user. All these aspects require the robot to adapt. It needs to execute different tasks, and rapidly adapt to the user's actions and requirements. This adaptation requirement makes learning a crucial feature for collaborative robots. The goal of this special issue is to document and highlight recent progress in robot learning for human-robot collaboration (HRC), covering a diversity of articles that reflect the state-of-the-art in the field. Following an open call for papers,

Leonel Rozo
Heni Ben Amor
Sylvain Calinon
Anca Dragan
Dongheui Lee

Special issue on learning for human-robot collaboration

we received more than fifty submissions, among which ten articles were selected for the special issue, covering exciting applications spanning co-manipulation, medical robotics, and social behaviors. These works and five additional papers compose a fascinating online topical collection that reports a larger diversity of research advancements in learning for HRC. In Progress and Prospects of the Physical Human-Robot Collaboration, Ajoudani, Zanchettin, Ivaldi, Albu-Schäffer, Kosuge and Khatib review the state-of-the-art on intermediate human-robot bi-directional interfaces, robot control modalities, systems stability, benchmarking, and relevant use cases. The article extends views on the required future developments in the realm of physical human-robot collaboration. The authors provide a thorough overview of the pioneering methodologies that aim at achieving intuitive and seamless HRC, potentially relying on a minimum degree of task-related pre-programming, where the relevance of robot learning stands out. They also provide a list of potential applications and relevant use cases ranging from domestic to industrial environments. Turn-taking prediction, a synchronization aspect in HRC, is the capability to comprehend the on-going task progress and predict where, when and how to seize the next turn during multi-agent collaborations. In Early Prediction for Physical Human-Robot Collaboration in the Operating Room, Zhou and Wachs show the importance of fluent and natural turn-taking regulations in team performance, which lead to better social connections among team members. The requirement for early turn-taking prediction stands out more clearly in high-risk and high-paced tasks like surgery, where the scrub nurse and the surgeon perform fast, accurate and highly coordinated turn-taking actions when exchanging surgical instruments. In this context, the authors tackle the problem of designing a fully functional robotic scrub nurse, and propose a computational framework for early turn-taking prediction built on Long Short-Term Memory networks and Dempster-Shafer theory for uncertain sensor fusion. The approach is evaluated on a simulated surgical procedure dataset, and is found to be superior to its algorithmic counterparts, and better than the human baseline when little partial input is given. Shared autonomy is on its own a relevant form of collaboration, and medical robotics has also inspired the research on shared autonomy reported in the paper Skill-Based Human-Robot Cooperation in Teleoperated Path Tracking by Enayati, Ferrigno and De Momi. The paper argues that a robot's assistance during shared autonomy should depend on the operator's skill level. The robot should aid novices, but not unnecessarily restrict experts. This work introduces a skill estimation system for path tracking tasks, and puts the idea to the test in a user study. The results support that customizing assistance based on operator's skill improves the shared autonomy experience. Human-robot collaborative manipulation entails bi-directional force exchange, mutual adaptation, variable compliance, among others. Such scenario served as an inspiration for several works in this special issue. For example, in Robot Adaptation to Human Physical Fatigue in Human-Robot Co-Manipulation, Peternel, Tsagarakis, Caldwell and Ajoudani propose to estimate human fatigue, and to adjust how much a robot is helping in a human-robot collaborative manipulation task based on how tired the person gets. The robot begins by acting as a follower, and observes the trajectory, stiffness, and/or forces that the user prefers. As the user is performing the task, the robot collects these examples and also monitors the estimated human shoulder muscle activation via electromyography (EMG). Once the user's fatigue reaches a predefined threshold, the robot takes control of the task and follows the learned stiffness/force and trajectories. In Human-Robot Cooperation with Compliance Adaptation along the Motion Trajectory, Nemec, Likar, Gams and Ude introduce a learning from demonstration interface for kinesthetic teaching of compliant co-manipulation tasks. Since it is difficult to demonstrate precise motions at high speed, the paper proposes to split the teaching procedure into two parts. First, the human demonstrates the desired trajectory, and the robot infers the desired stiffness from the variance in the demonstrations. Second, the human demonstrates the desired speed profile, with trajectory guidance provided by the robot based on the first training step. The proposed scheme was experimentally verified in a humanrobot collaborative transportation task, which may be further exploited in assembly processes in production plants or in civil engineering for transportation of heavy and bulky objects. In Co-manipulation with a Library of Virtual Guiding Fixtures, Raiola, Sanchez Restrepo, Chevalier, Rodriguez-Ayerbe, Lamy, Tliba and Stulp address the problem of learning virtual guiding fixtures in HRC. These guides are built on a probabilistic representation based on Gaussian mixture regression, and are aimed at constraining the movements of a robot to task-relevant trajectories defined by the user. By using the analogy of a ruler used as a physical guide to draw lines, they emphasize that robots can be used to define virtual guides of complex shapes, enabling safer co-manipulation for industrial tasks. Whereas previous work mostly considered guiding fixtures for single tasks, this paper addresses the problem of creating a library of guiding fixtures for multiple tasks, by selecting the guides online and incrementally refining them. The approach is demonstrated in a user study with an industrial pick-and-place task, showing that a library of guiding fixtures can provide an intuitive haptic interface for joint human-robot completion of tasks, improving task execution time, and reducing mental workload and errors. Handovers are complex interactions where agents coordinate in time and space to transfer control of an object. This challenging interaction is the focus of the paper One-Shot Learning of Human-Robot Handovers with Triadic Interaction Meshes by Vogt, Stepputtis, Jung and Ben Amor, where it is proposed to learn human-robot handover interaction from observing one demonstration of humanhuman handover. The relevant information, such as joint correlation and spatial relationships of two humans and a handing-over object, is extracted from a single demonstration. Triadic interaction meshes are constructed to model interaction between two agents and an object, and are subsequently used for robot's adaptive motion generation in human-robot handover scenarios. Human-Robot social interactions also play an essential role in extending the use of robots in daily life, where robot learning can be exploited to master different types of behaviors. In this context, the paper Learning Proactive Behavior for Interactive Social Robots by Liu, Glas, Kanda and Ishiguro, addresses the learning of both reactive and proactive robot behaviors from human-human interactions. The authors first introduce the concept of a "yield action" that enables the robot to identify opportunities for a proactive action to be generated. Since proactive behaviors are often sensitive to the context of the interaction, interaction history is incorporated into the inputs of a deep neural network, along with variables describing natural language and the state of the interactive agents. An attention mechanism is also used in the learning system, which has the ability to "attend" and learn which parts of the interaction history are relevant. Offline analysis and live interactions with users in a camera shop scenario showed that the proposed system can effectively reproduce proactive behaviors that were perceived as socially-appropriate by the participants. The paper Hierarchical Emotional Episodic Memory for Social Human-Robot Collaboration by Lee and Kim investigates how human emotional states can be represented and anticipated in order to improve the HRC experience. A critical component to achieve these goals is a computational model of episodic memory based on Adaptive Resonance Theory. The resulting framework allows robots to learn correlations between the user's emotional state and the executed collaborative actions. In turn, the introduced framework and algorithms can be used in a variety of tasks, in particular in domestic environments, in order to endow robots with social capabilities. Finally, a fresh look at interaction learning methods for collaborative tasks is given in the paper Efficient Behavior Learning in Human-Robot Collaboration by Munzer, Toussaint and Lopes. In particular, the paper discusses a relational approach to learning task requirements and user preferences. A subset of first-order logic is used to explicitly represent knowledge about the collaborative task. At the same time, a reinforcement learning method is used to iteratively update the representation, so as to increase efficiency during the joint task. The approach aims at bridging the divide between symbolic and numerical approaches to artificial intelligence for robot systems. Collectively, these ten papers illustrate a broad range of human-robot collaboration applications. They provide a compilation of the large variety of challenges that are currently being investigated, where robot learning emerges as a promising approach to tackle this diversity.


Abstract: Our goal is to infer reward functions from demonstrations. In order to infer the correct reward function, we must account for the systematic ways in which the demonstrator is suboptimal. Prior work in inverse reinforcement learning can account for specific, known biases, but cannot handle demonstrators with unknown biases. In this work, we explore the idea of learning the demonstrator's planning algorithm (including their unknown biases), along with their reward function. What makes this challenging is that any demonstration could be explained either by positing a term in the reward function, or by positing a particular systematic bias. We explore what assumptions are sufficient for avoiding this impossibility result: either access to tasks with known rewards which enable estimating the planner separately, or that the demonstrator is sufficiently close to optimal that this can serve as a regularizer. In our exploration with synthetic models of human biases, we find that it is possible to adapt to different biases and perform better than assuming a fixed model of the demonstrator, such as Boltzmann rationality.

Under review as a conference paper at ICLR 2019 INFERRING REWARD FUNCTIONS FROM DEMONSTRATORS WITH UNKNOWN BIASES

INTRODUCTION Our ultimate goal is to enable the design of agents that optimize for the right reward function. Unfortunately, designing reward functions is challenging  (Amodei et al., 2017)  and can have unintended side-effects  (Hadfield-Menell et al., 2017) . Inverse Reinforcement Learning  (Russell, 1998; Ng et al., 2000; Abbeel & Ng, 2004)  aims to bypass the need for reward design by learning the reward from observed demonstrations of good behavior. Existing IRL algorithms typically make the assumption that the demonstrator is either optimal, or Boltzmann rational, i.e. taking better actions with higher probability  (Ziebart et al., 2008; Finn et al., 2016) . However, there is a rich literature showing that humans are not optimal, and are biased in systematic ways. Consider a grad student who starts writing a paper a month in advance, expecting it to take two weeks, but then misses the deadline. Should we infer that they prefer to lose sleep to pursue a deadline that they then miss? Of course not -this is a classic case of the planning fallacy  (Buehler et al., 1994) . It is not clear what exactly went wrong -perhaps the grad student did not have a good dynamics model (not realizing that Amazon spot instances can be suddenly terminated), or they failed to account for the uncertainty in their dynamics model (expecting that it wouldn't happen to them), or they optimized too much for short term reward (watching Netflix), and so on -we do not know what the bias is. Ideally, even with this unknown bias, we would like our system to infer that the grad student failed to plan appropriately, and that they would prefer to meet the deadline. This leads to a tradeoff between the expressivity of our model of the demonstrator, and the ability of the algorithm to learn the true reward function with few samples. On one extreme, if we know exactly what bias the demonstrator has, we can account for it to infer the true reward function quickly and accurately, as in  Evans et al. (2016) ;  Evans & Goodman (2015) ;  Zheng et al. (2014) ;  Majumdar et al. (2017)  (myopia and hyperbolic time discounting, sparse noise, and risk-sensitivity respectively). Even suboptimal trajectories or failures  (Shiarlis et al., 2016)  can be thought of as a biased demonstrator, where the bias is the specific model of failure. Unfortunately, people are complex, and it is unlikely that we will know exactly which bias a person is displaying, or even the space of possible biases that people might have. If we are incorrect about the bias, we will have a mis-specified model, and the reward function will get garbage values that explain the "noise" resulting from the mis-specification  (Steinhardt, 2017; Steinhardt & Evans, 2017) . On the other extreme, when the demonstrator could be any function mapping reward functions to policies, it is impossible to learn the true reward function even with infinite data, because there are always alternative explanations for the observed policy  (Armstrong & Mindermann, 2017; Christiano, 2015) . We could consider asking the human to rate each demonstration  (Burchfiel et al., 2016) , but the human could be biased in these ratings as well. In this paper, we focus on learning rewards when the demonstrator's bias is unknown. We seek to avoid impossibility results by introducing a set of assumptions that are weaker than assuming a known bias, the core of which is that the demonstrator plans their actions similarly in similar tasks. This leads us to investigate the idea that one could actually learn, rather than assume, the demonstrator's planning algorithm across multiple tasks, including any biases they may have. We analyze the feasibility of this idea in two settings, via two algorithms. In both algorithms, we can navigate the tradeoff between model expressivity and sample efficiency through the choice of architecture for the differentiable planner we learn, and the space of reward functions to consider. In the first setting, we assume that we have access to the true reward functions in some subset of tasks -that means we can learn the demonstrator's planning algorithm on those tasks, after which we can infer rewards from demonstrations in new tasks. In the more realistic setting where we have no access to any reward function, we assume that the demonstrator is "close" to optimal. This is a weaker version of the assumption of Boltzmann rationality that still allows us to learn particular systematic biases. We operationalize this assumption by initializing the differentiable planner to mimic the optimal agent, and then finetuning it to allow it to account for specific biases. We show that this initialization is essential for good performance, as we would expect from the impossibility result. In summary, this paper makes three contributions: 1. Assumptions that are more realistic than the assumption of Boltzmann rationality that allow us to avoid known pitfalls of reward inference. 2. Algorithms that leverage these assumptions to infer reward functions from suboptimal demonstrations. 3. An experimental evaluation demonstrating there is hope in the idea of learning better rewards by learning biases too, despite the theoretical difficulty of the problem.
 EXAMPLES OF BIASES To put this work in context, we start with some examples of the kind of biases a general algorithm should be able to capture and account for. While we use these for illustrative purposes, the whole point of our work is that humans might have systematic suboptimalities completely different from these examples. We don't know all the possible biases a priori. Running Example. We illustrate the effects of these biases on a simple 2D navigation task in figure  1 . There are multiple salient locations, each of which has a desirability score (which can be negative, in which case the agent wants to avoid those locations). The agent can move in any of the four cardinal directions, or stay in its current position. Every movement action has a chance of failing and causing the agent to move in a direction orthogonal to the one it chose. Despite their simplicity, there are several ways in which human-like suboptimal behavior can manifest in these environments. Time inconsistency. Would you prefer to get $100 in 30 days, or $110 in 31 days? Faced with this question, people typically choose the latter. However, thirty days later, when faced with the choice of getting $100 now, or $110 tomorrow, they sometimes choose to take the $100. This reversal of preferences over time would never happen with an optimal agent that maximizes expected sum of discounted rewards. Researchers model this phenomenon using hyperbolic time discounting, in which future rewards are discounted more aggressively than exponentially. This leads to a followup question -how do humans make long-term plans, given that their future self will have different preferences? Prior work has considered a spectrum from naive agents that assume their future self will have the same preferences as they do, to sophisticated agents that perfectly understand how their preferences will change over time and make plans that take such change into account  (Frederick et al., 2002) . In figure  1 , when going to a high reward, both the naive and sophisticated hyperbolic time discounters can be "tempted" by a proximate smaller reward. The naive agent fails to anticipate the temptation, and so once it gets near the smaller positive reward, it caves in to the temptation and stays there. The sophisticated agent explicitly plans to avoid the temptation -it does not collect the smaller reward and instead takes a longer, more dangerous path around the smaller reward to get to the large reward. Incorrect estimates of probabilities. Humans are notoriously bad at judging probabilities. The availability heuristic  (Tversky & Kahneman, 1973)  refers to the human tendency to rate events as more likely if they are easier to recall. The recency effect is a similar effect where recent events are judged to be more probable. These biases are complicated and depend heavily on context and don't obviously transfer to our task. So, we use two simplified models -an overconfident agent, which expects that the most likely next state is more likely than it actually is, leading it to take risks, and an underconfident agent, which is overly cautious. In figure  1 , the overconfident agent takes the shortest path to the reward, underestimating the risk of slipping into the large region of negative reward, while the underconfident agent plans a circuitous route around negative reward that it is unlikely to have actually encountered. Bounded computation. Researchers have studied models of bounded rationality, where humans are assumed to be rational subject to the constraint that they have a bounded amount of computation. This can be thought of as an explanation that many other heuristics and biases are actually computational shortcuts that allow us to reach reasonably good decisions without too much cost  (Kahneman, 2003) . In our task, we model computation bounds as a small time horizon for planning, leading to myopic behavior. In figure  1 , the myopic agent can only see close rewards, and goes directly to them, never even realizing the possibility of going to the highest reward.
 PROBLEM: LEARNING REWARDS OF DEMONSTRATORS WITH UNKNOWN BIASES Notation. A (finite-horizon) Markov Decision Process (MDP) (Puterman, 2014) is a tuple S, A, T, r, H . S is a set of states. A is a set of actions. T is a probability distribution over the next state, given the previous state and action. We write this as T (s t+1 |s t , a). r is a reward function that maps states and actions to rewards r : S × A → R. H ∈ Z + is the finite planning horizon for the agent. Since we are interested in the setting where the reward function r is unknown, we will factor MDPs into world models w = S, A, T, H and reward functions r. Instead of having access to a reward function, we observe the behavior of an demonstrator, who performs the task well but may be suboptimal in systematic ways. In particular, we observe the demonstrator's policy π : S → A. Estimating Biases and Rewards. Given a world model w and the demonstrator's policy π D which may exhibit an unknown bias, determine the reward r * that the demonstrator is optimizing. We might hope to solve this problem without any additional assumptions. However, this problem is unsolvable -  Armstrong & Mindermann (2017)  prove an impossibility result showing that for any potential reward function r , there is some planner D such that D (w, r ) = D(w, r * ). The proof is simple -simply set D (w, r ) = π for any r , that is D always returns π regardless of reward. Inverse reinforcement learning assumes that the demonstrator is (approximately) optimal to get around this issue. It is common to assume Boltzmann rationality, where the probability of an action is proportional to the exponent of its expected value  (Baker et al., 2006) , i.e. P (a|s) ∝ e  Q(s,a)  , where Q is the optimal Q function that satisfies the Bellman equation: Q(s, a) = R(s, a) + γ s T (s |s, a) max a Q(s , a ) (1) However, we know that humans are systematically suboptimal, and so we would like to relax this assumption and try other, more realistic assumptions. The pathological solutions in the impossibility result occur partly because the demonstrator can have arbitrary behavior on different environments. While we certainly want the demonstrator to adapt to different environments, the algorithm that the demonstrator uses to determine their policy should stay fixed across similar environments. This imposes structure on the demonstrator's planner that can eliminate some possibilities. Assumption 1: The demonstrator plans in the same way for sufficiently similar environments. To formalize this, we assume that there is a space of world models W with the same set of states S and actions A, and a space of reward functions R ⊆ S × A → R. The demonstrator can plan a stochastic policy given a world model and reward, that is, they are modeled as D : (W × R) → (S → A → [0, 1]). We call D the planning algorithm used by the demonstrator, or planner for short. Of course, if D can be any function with this type signature, it can still map any arbitrary (w, r) pair to any arbitrary policy D(w, r), but we will further ensure that D is simple (through regularization). Given a list of world models W = [w 1 . . . w n ] and reward functions R = [r 1 . . . r n ], we define D(W, R) to be the list of the demonstrator's policies [D(w 1 , r 1 ) . . . D(w n , r n )]. Note that this is a strong assumption and it does limit the scope of our work: while it is reasonable to believe that people plan in the same way for variations of the same task, they likely have different biases for different tasks, because they may have domain-specific heuristics. The setting of multiple tasks has been studied before  (Gleave & Habryka, 2018; Dimitrakakis & Rothkopf, 2011; Choi & Kim, 2012) , though not for the purpose of inferring systematic biases. This assumption leads to a slightly easier problem, of recovering rewards from multiple tasks: Estimating Biases and Rewards for Multiple Tasks. Given a list of world models W and the demonstrator's policies Π D = D(W, R) which may exhibit an unknown bias, determine the list of reward functions R (one for each w ∈ W ) that the demonstrator was optimizing. Since the person uses the same planner across all tasks, an agent can have an easier time recovering rewards for each task by leveraging the common structure across the tasks. This is especially appealing for agents that would get to observe people for some period of time before trying to assist them. Unfortunately, Assumption 1 is not sufficient to solve this problem. Consider the case where the demonstrator is optimal. Given the assumptions so far, we could infer that the demonstrator is minimizing expected reward, i.e. that the reward they are optimizing is −r * , since that perfectly predicts π D . This is very bad, as we could infer a reward that incentivizes the worst possible behavior! When humans take action, we typically assume that they are doing something that is reasonable for achieving their goals, even if it is not optimal: Assumption 2a: The demonstrator is "close" to optimal. We explore this assumption in section 4.3 to solve the problem of estimating biases and rewards for multiple tasks. We also explore an alternative approach, based on the fact that we have strong priors about what humans are trying to optimize for, which allow us to infer how good they are at achieving their goals. If we see some tasks where we know the demonstrator's reward function and policy, we can infer that the demonstrator is not minimizing expected reward.
 Assumption 2b: We know what reward function the demonstrator is optimizing for some tasks. Estimating Biases and Rewards with Access to Tasks with Known Rewards. Given a list of world models W , a list of the demonstrator's policies Π D = D(W, R), a list of world models W known with known rewards R known and a list of the demonstrator's policies Π known = D(W known , R known ), determine the reward functions R that D was optimizing.  We will present algorithms for both settings in the next section. While our problem formulations above assume that we have access to full policies Π D , none of our algorithms rely on this assumption -it is easy to modify them to work with trajectories instead.
 ALGORITHMS TO ESTIMATE BIASES AND REWARDS The key idea behind our algorithms is to learn a model of how the demonstrator plans, and invert the model's "understanding" using backpropagation to infer the reward from actions.
 ARCHITECTURE We model the demonstrator planning algorithm D using a differentiable planner f θ , which is a neural net that can express planning algorithms whose parameters θ can be updated using gradient descent. f has the same type as the demonstrator's planner D, namely (W × R) → (S → A → [0, 1]). Thus, the inputs to the differentiable planner f are a world model w ∈ W and a reward function r ∈ R; the output is a stochastic policy π ∈ (S → A → [0, 1]). We determine how well f, R matches the demonstrator's policy π D with the cross entropy loss L(f θ (W, R), Π D ) = i L(f θ (w i , r i ), π D,i ). While our algorithms can work with any differentiable planner, in this work we use a value iteration network (VIN)  (Tamar et al., 2016) . A VIN is a fully differentiable neural network that embeds an approximate value iteration algorithm inside a feed-forward classification network. For environments where transitions only depend on "nearby" states (as in navigation tasks), the Bellman update can be performed using an appropriate convolution, and the computation of values from Q-values can be done with a max-pooling layer. By leaving the filters for the convolutions unspecified, the VIN can automatically learn the transition probabilities. Of course, the VIN is merely one architecture for a differentiable planner; we could equally well use other planners  (Srinivas et al., 2018; Pascanu et al., 2017; Guez et al., 2018) . As research in this area advances, our work stands to benefit. By choosing this particular architecture we are making another assumption: Assumption 3: The inductive bias of f θ is well-suited to the task of learning the planner D. In our experiments, this assumption says that the demonstrator's planner is well-modeled by a Value Iteration Network. This is not a good assumption currently, but we expect that as more research is done on differentiable planners, they will become more sophisticated and will become better able to learn human planning algorithms. The components of our algorithms. There are two main operations that we make use of on this architecture, which we illustrate in figure  2 . First, given world models W , reward functions R (either known or hypothesized), and the demonstrator's policies Π D , we can train a corresponding planner using gradient descent (figure  2a ): θ = min θ L(f θ (W, R), Π D ) (TRAIN-PLANNER) Second, given world models W , demonstrator's policies Π D , and some planner parameters θ, we can infer the corresponding reward functions using gradient descent (figure  2b ): R = min R L(f θ (W, R ), Π D ) (TRAIN-REWARD) We can also perform both of these at the same time by training the planner parameters and rewards jointly given world models W and the demonstrator's policies Π D : R, θ = min R ,θ L(f θ (W, R ), Π D ) (TRAIN-JOINTLY) 4.2 LEARNING THE PLANNER FROM KNOWN REWARDS FIRST (ASSUMPTION 2B) First, we tackle the simpler setting when we have access to a set of tasks with known rewards. We illustrate this in Algorithm 1. We first train the planner on the world models for which we have rewards, which lets us learn a model of how the demonstrator plans, including any systematic biases they may have. Then, we use the learned planner weights to infer the reward on the world models for which we don't know the reward. Algorithm 1 Estimating biases and rewards with access to tasks with known rewards. 1: function IRL-WITH-REWARDS(W , Π D , W known , R known , Π known ) 2: θ ← TRAIN-PLANNER(W known , R known , Π known ) 3: return TRAIN-REWARD(W, θ, Π D )
 LEARNING THE PLANNER AND REWARDS SIMULTANEOUSLY (ASSUMPTION 2A) The assumption that we have some tasks with known rewards is very strong, and may not hold in practice. How could we infer rewards without making this assumption? The obvious answer is to train θ and R jointly as in Equation TRAIN-JOINTLY. However, since we have discarded Assumption 2b, we can once again learn the planner that minimizes expected reward, and infer that the reward is −r * . We must instead use Assumption 2a, that the demonstrator is "close" to optimal. To use this assumption in our algorithm, we simulate data from an optimal agent with randomly generated world models and rewards, and use this to train the planner to mimic an optimal agent. After initializing this way, we can then train the planner and reward jointly. This gives us Algorithm 2. We show in section 5.3 that the initialization, enacting assumption 2a, is crucial for good performance. Algorithm 2 Estimating biases and rewards for multiple tasks with no known rewards. 1: function IRL-WITHOUT-REWARDS(W , Π D ) 2: W sim , R sim ← Generate random world models and rewards 3: Π sim ← Run optimal agent on W sim , R sim 4: θ init ← TRAIN-PLANNER(W sim , R sim , Π sim ) 5: R init ← TRAIN-REWARD(W, θ, Π D ) 6: θ, R ← TRAIN-JOINTLY(W, Π D ) using θ init and R init as initializations 7: return R
 EVALUATION We evaluate our algorithms by simulating demonstrators with different biases, and testing whether the same method can correctly infer reward for all these demonstrators. Details on the experiment setup are provided in the supplementary material.
 EVALUATING REWARD INFERENCE Hypothesis. The key idea behind this work is that accounting for unknown systematic bias should outperform the assumption of a particular inaccurate bias, e.g. noisy rationality or the lack thereof. Manipulated variables. In order to test this, we manipulate whether we learn the demonstrator model or assume it. To avoid confounds introduced by changing the inference algorithm, we use the same algorithm for both. In the learning case, we train the planner on the ground truth demonstrator data; in the assume case, we train it on data generated from a) a Boltzmann-rational demonstrator; and b) an optimal demonstrator -these are the two models commonly assumed by IRL algorithms. Keeping the algorithm the same enables us to isolate the effect of adapting to an unknown model from the effect of having to use an approximate differentiable planner rather than a perfect one. We will quantify the second effect, i.e. the approximation error introduced by the VIN, in section 5.2 In the setting where we learn the bias, we further manipulate whether we have access to known rewards for some tasks or not -i.e. whether we use Algorithm 1 or Algorithm 2. Finally, we manipulate the actual bias of the demonstrator. Following  Evans et al. (2016) , we implement the myopic, naive and sophisticated synthetic demonstrators as modifications of the value iteration algorithm. Similarly, we implement the overconfident and underconfident demonstrators by modifying the transition probability distributions used to plan in value iteration. We also include an optimal demonstrator, and stochastic (Boltzmann) versions of all demonstrators. Dependent measures. We measure the reward obtained by planning optimally with the inferred reward function, as a percentage of the maximum possible reward that could be obtained. Findings. Figure  3  shows our results comparing learning a demonstrator model with assuming an optimal or a Boltzmann demonstrator. The top left subfigure plots what happens on average, across all synthetic demonstrators we tested. The results provide support to our hypothesis: both learning methods (orange) outperform assuming a model (gray). Looking at the breakdown per demonstrator, we see that assuming optimal does not do well when the demonstrator has any noise (bottom graph). Similarly, assuming Boltzmann does not do well when the demonstrator is not noisy (top graph). The learning methods tend to perform on par with the best of two choices. In some cases, like the naive and sophisticated hyperbolic discounters, especially the noisy ones, the learning methods outperform both optimal and Boltzmann assumptions. The optimal assumption performs well in some of the non-noisy cases, because our demonstrator bias models are almost deterministic (they only break ties randomly). So, as long as the demonstrator eventually reaches the best reward location, the reward inference works well. Interestingly, Algorithm 1 does not always outperform Algorithm 2, despite it having access to known rewards. We believe this has to do with the fact that Algorithm 2 exploits Assumption 2a (demonstrator close to optimal) and initializes from training on simulated optimal demonstrator data. Algorithm 1 does not rely on this assumption and therefore does not benefit from this initialization, whereas the assumption is correct for most of the models we test.
 TRADEOFF BETWEEN BEING ADAPTIVE TO BIAS VS. USING EXACT PLANNING Our paper is not about a practical solution to be used right now, but rather an investigation of the viability of an idea. The core reason behind this is that to be adaptive to different kinds of biases we might see, we learn a model of the demonstrator's planning algorithm via a differentiable planner. Unfortunately, this causes our planning to be approximate -whatever benefit we get from the adapting to biases, we lose because of the approximation. But these planners will become more practical, they can make this idea practical as well. To quantify this loss, we replace the VIN with a differentiable exact model of the demonstrator, and infer the reward by backpropagating through the exact model. Since value iteration is not differentiable, we implement soft value iteration, where max operations are replaced with logsumexp operations, and measure percent reward obtained when inferring rewards for an optimal demonstrator.
 Results . With an exact model of the demonstrator, we get (98.1±0.1)% of the maximum reward when performing optimal planning on the inferred rewards, while we get (86.9 ± 1.6)% with Algorithm 1 and (86.2 ± 1.6)% with Algorithm 2. Again, better planners would improve both algorithms.
 HOW IMPORTANT ARE THE VARIOUS PARTS OF THE ALGORITHM? Algorithm 2 was predicated on Assumption 2a, that the demonstrator's planner was "close" to rational, which motivated the initialization step where the planner is trained to mimic an optimal agent. We test how important this is by modifying Algorithm 2 to infer rewards without an initialization (removing lines 2-5). We include versions of the algorithm where we perform coordinate ascent by alternating planner training and reward training instead of training the planner and reward jointly. Results. Figure  4  shows the results for a subset of demonstrators (full results are in the supplementary material). We can see that the initialization is indeed crucial for good performance, as expected. It also turns out that the joint training outperforms coordinate ascent.
 DISCUSSION Summary. In this work, we considered the very general problem of inverse reinforcement learning when the demonstrator has an unknown bias. In this problem we face a severe tradeoff between the ability to adapt to unknown biases, and the ability to infer any reward function. We introduced assumptions that are weaker than the typical assumption of noisy rationality, and tested algorithms that leverage these assumptions to perform better on average across different kinds of biases than assuming a specific bias known a priori. Limitations and future work. We see this paper as taking a step towards exploring the idea of learning planners from demonstrations to learn reward functions. This is not yet a practical idea, due to the practical limitations of differentiable planners. Thus, to analyze how feasible this is with different possible biases and with ground truth reward functions, we resorted to synthetic demonstrator data as opposed to human data. Further, our assumption that the demonstrator has the same bias across many tasks is key to our work, but is very strong. We could extend this work by using meta-learning to learn a prior over planners. We will also need to infer the demonstrator's beliefs as in  Baker & Tenenbaum (2014) , for which we could use TOMNets  (Rabinowitz et al., 2018) . We are excited to look into this, and into what additional inductive bias we could leverage, in our future work.
 A ADDITIONAL EXPERIMENTAL DATA In section 5, we presented data on the results of running various algorithms against a set of demonstrators, reporting the reward obtained according to the true reward function when using the inferred reward with an optimal planner, as a percentage of the maximum possible true reward. Table  1  shows the percentage reward obtained for all combinations of algorithms and demonstrators. We also measure the accuracy of the planner and reward at predicting the demonstrator's actions in new gridworlds where the rewards are the same but the wall locations have changed. These results are presented in Table  2 . Note that there are often multiple optimal actions at a given state, which makes it challenging to get high accuracy. Table  1 : Percent reward obtained when the algorithm (column) is used to infer the bias of the demonstrator (row). The optimal and Boltzmann algorithms assume a fixed model of the demonstrator and train the VIN to mimic the model before performing reward inference (and were used in figure  3 ). We also include the four flavors of Algorithm 2 that were plotted in figure  4 . The VI algorithm uses a differentiable implementation of soft value iteration as the planner instead of a VIN (used in section 5.2). The demonstrators are the optimal agent, the biased agents of figure  1 , and versions of each of these agents with Boltzmann noise.
 B EXPERIMENT DETAILS All results are averaged over 10 runs with different seeds, on randomly generated 14x14 gridworlds that have 7 squares with non-zero rewards. We ensure that all such squares can be reached from the start state, and that at least half of the positions in grid are not walls. We use a Value Iteration Network with 10 iterations as our differentiable planner, and set the space of rewards to be S → R; that is, any state can be mapped to any reward, but the reward is assumed not to depend on the action. We added an extra convolutional layer to the initial part of the VIN (which learns the proxy reward) as initial experiments showed that this could better learn an optimal planner for our gridworlds. We apply L2 regularization to the VIN with scale 0.0001, and do not regularize the reward. For all experiments, we kept the number of demonstrations fixed to 8000. For Algorithm 1, this was split into 7000 policies with rewards that were used to train the planner, and 1000 on which rewards had to be inferred. Note that this does not include any simulated data -for example, Algorithm 2 would get 8000 biased policies, and would also simulate a further 7000 policies from an optimal agent in order to initialize the planner and reward. Figure 1 : 1 Figure 1: The plans of our synthetic agents on two navigation environments. Actual trajectories could differ due to randomness in the transitions. Green squares indicate positive reward while red squares indicate negative reward, with darker colors indicating higher magnitude of reward.
 ( a ) a Training the planner f θ . We hold the world model w, reward r, and policy π fixed, and update the planner f θ with gradient descent.(b) Training the reward R. We hold the world model w, planner f θ , and policy π fixed, and update the reward r with gradient descent.
 Figure 2 : 2 Figure 2: The architecture and operations on it that we use in our algorithms.
 Figure 3 : 3 Figure 3: Reward obtained when planning with the inferred reward, as a percentage of the maximum possible reward, for different bias models and algorithms.
 Figure 4 : 4 Figure4: Percent reward obtained for different bias models using variations of Algorithm 2, which does not get access to any known rewards. These algorithms can vary along two dimensions -whether they are initialized with the assumption that the demonstrator is rational, and whether they train the planner and reward jointly or with coordinate ascent. The original version of Algorithm 2 does initialize, and trains jointly.