Related Work
The IRL problem is under-defined, so approaches use different means of identifying which reward function best explains the observed trajectories. Among these, maximum margin IRL methods [ 29 , 1 ] choose the reward function that most separates the optimal policy and the second-best policy. Variants of these methods have allowed for suboptimal demonstrations [ 32 ] , non-linear reward functions [ 35 ] , and game-theoretic learning [ 37 ] . Bayesian IRL approaches [ 31 , 30 ] use prior knowledge to bias the search over reward functions, and can support suboptimal demonstrations [ 33 ] . Gradient-based algorithms optimize a loss to learn the reward while, for instance, penalizing deviations from the expert’s policy [ 28 ] . Maximum entropy models [ 43 , 21 , 42 ] find the most likely reward function given the demonstrations, and produce a policy that matches the user’s expected performance without making further assumptions on the preference over trajectories. Other work has avoided learning the reward altogether and focuses instead on modeling the user’s policy via classification [ 27 ] .

Note, however, that all these approaches focus on learning a single IRL task, and do not consider sharing knowledge between multiple tasks. Although other work has focused on multi-task IRL, existing methods either assume that the tasks share a state and action space, or scale poorly due to their computational cost; our approach differs in both respects. An early approach to multi-task IRL [ 12 ] learned different tasks by sampling from a joint prior on the rewards and policies, assuming that the state-action spaces are shared. Tanwani and Billard [ 38 ] studied knowledge transfer for learning from multiple experts, by using previously learned reward functions to bootstrap the search when a new expert demonstrates trajectories. Although efficient, their approach does not optimize performance across all tasks, and only considers learning different experts’ approaches to one task.

The notion of transfer in IRL was also studied in an unsupervised setting [ 2 , 11 ] , where each task is assumed to be generated from a set of hidden intentions. These methods cluster an initial batch of tasks, and upon observing each new task, use the clusters to rapidly learn the corresponding reward function. However, they do not address how to update the clusters after observing a new task. Moreover, these methods assume the state-action space is shared across tasks, and, as an inner loop in the optimization, learn a single policy for all tasks. If the space was not shared, the repeated policy learning would become computationally infeasible for numerous tasks. Most recently, transfer in IRL has been studied for solving the one-shot imitation learning problem [ 13 , 17 ] . In this setting, the agent is tasked with using knowledge from an initial set of tasks to generalize to a new task given a single demonstration of the new task. The main drawback of these methods is that they require a large batch of tasks available at training time, and so cannot handle tasks arriving sequentially.

Our work is most similar to that by Mangin and Oudeyer [ 25 ] , which poses the multi-task IRL problem as batch dictionary learning of primitive tasks, but appears to be incomplete and unpublished. Finn et al. [ 16 ] used IRL as a step for transferring knowledge in a lifelong RL setting, but they do not explore lifelong learning specifically for IRL. In contrast to existing work, our method can handle distinct state-action spaces. It is fully online and computationally efficient, enabling it to rapidly learn the reward function for each new task via transfer and then update a shared knowledge repository. New knowledge is transferred in reverse to improve the reward functions of previous tasks (without retraining on these tasks), thereby optimizing all tasks. We achieve this by adapting ideas from lifelong learning in the supervised setting [ 34 ] , which we show achieves similar benefits in IRL.