Introduction
In many applications, such as personal robotics or intelligent virtual assistants, a user may want to teach an agent to perform some sequential decision-making task. Often, the user may be able to demonstrate the appropriate behavior, allowing the agent to learn the customized task through imitation. Research in inverse reinforcement learning (IRL) [ 29 , 1 , 43 , 21 , 31 , 28 ] has shown success with framing the learning from demonstration (LfD) problem as optimizing a utility function from user demonstrations. IRL assumes that the user acts to optimize some reward function in performing the demonstrations, even if they cannot explicitly specify that reward function as in typical reinforcement learning (RL). 1 1 1 Complex RL tasks require similarly complex reward functions, which are often hand-coded. This hand-coding would be very cumbersome for most users, making demonstrations better for training novel behavior. IRL seeks to recover this reward function from demonstrations, and then use it to train an optimal policy. Learning the reward function instead of merely copying the user’s policy provides the agent with a portable representation of the task. Most IRL approaches have focused on an agent learning a single task. However, as AI systems become more versatile, it is increasingly likely that the agent will be expected to learn multiple tasks over its lifetime. If it learned each task in isolation, this process would cause a substantial burden on the user to provide numerous demonstrations.

To address this challenge, we introduce the novel problem of lifelong learning from demonstration , in which an agent will face multiple consecutive LfD tasks and must optimize its overall performance. By building upon its knowledge from previous tasks, the agent can reduce the number of user demonstrations needed to learn a new task. As one illustrative example, consider a personal service robot learning to perform household chores from its human owner. Initially, the human might want to teach the robot to load the dishwasher by providing demonstrations of the task. At a later time, the user could teach the robot to set the dining table. These tasks are clearly related since they involve manipulating dinnerware and cutlery, and so we would expect the robot to leverage any relevant knowledge obtained from loading the dishwasher while setting the table for dinner. Additionally, we would hope the robot could improve its understanding of the dishwasher task with any additional knowledge it gains from setting the dining table. Over the robot’s lifetime of many tasks, the ability to share knowledge between demonstrated tasks would substantially accelerate learning.

We frame lifelong LfD as an online multi-task learning problem, enabling the agent to accelerate learning by transferring knowledge among tasks. This transfer can be seen as exploiting the underlying relations among different reward functions (e.g., breaking a wine glass is always undesired). Although lifelong learning has been studied in classification, regression, and RL [ 10 , 34 , 6 ] , this is the first study of lifelong learning for IRL. Our framework wraps around existing IRL methods, performing lifelong function approximation of the learned reward functions. As an instantiation of our framework, we propose the Efficient Lifelong IRL (ELIRL) algorithm, which adapts Maximum Entropy (MaxEnt) IRL [ 43 ] into a lifelong learning setting. We show that ELIRL can successfully transfer knowledge between IRL tasks to improve performance, and this improvement increases as it learns more tasks. It significantly outperforms the base learner, MaxEnt IRL, with little additional cost, and can achieve equivalent or better performance than IRL via Gaussian processes with far less computational cost.