The Efficient Lifelong IRL Algorithm
As described in Section 4 , the lifelong IRL agent must optimize its performance over all IRL tasks observed so far. Using the MaxEnt assumption that the reward function \mathbf{r}^{(t)}_{s_{i}}=\boldsymbol{\theta}^{\top}\mathbf{x}^{(t)}_{s_{i}} for each task is linear and parameterized by \boldsymbol{\theta}^{(t)}\in\mathbb{R}^{d} , we can factorize these parameters into a linear combination \boldsymbol{\theta}^{(t)}=\mathbf{L}\mathbf{s}^{(t)} to facilitate transfer between parametric models, following Kumar and Daumé [ 19 ] and Maurer et al. [ 26 ] . The matrix {\mathbf{L}\in\mathbb{R}^{d\times k}} represents a set of k latent reward vectors that are shared between all tasks, with sparse task-specific coefficients {\mathbf{s}^{(t)}\in\mathbb{R}^{k}} to reconstruct \boldsymbol{\theta}^{(t)} .

Using this factorized representation to facilitate transfer between tasks, we place a Laplace prior on the \mathbf{s}^{(t)} ’s to encourage them to be sparse, and a Gaussian prior on \mathbf{L} to control its complexity, thereby encouraging the reward functions to share structure. This gives rise to the following reward prior:

P\left(\mathbf{r}^{(1)},\ldots,\mathbf{r}^{(N)}\right)=\frac{1}{Z\left(\lambda,\mu\right)}\exp\left(-N\lambda\|\mathbf{L}\|_{\mathsf{F}}^{2}\right)\prod_{t=1}^{N}\exp\left(-\mu\|\mathbf{s}^{(t)}\|_{1}\right)\enspace, (2)

where Z(\lambda,\mu) is the partition function, which has no effect on the optimization. We can substitute the prior in Equation 2 along with the MaxEnt likelihood into Equation 1 . After taking logs and re-arranging terms, this yields the equivalent objective:

\min_{\mathbf{L}}\frac{1}{N}\sum_{t=1}^{N}\min_{\mathbf{s}^{(t)}}\Biggl{\{}-\frac{1}{n_{t}}\sum_{\ \ \ \ \ \ \zeta^{(t)}_{j}\in\mathcal{Z}^{(t)}}\log P\left(\zeta_{j}^{(t)}\mid\mathbf{L}\mathbf{s}^{(t)},T^{(t)}\right)+\mu\|\mathbf{s}^{(t)}\|_{1}\Biggr{\}}+\lambda\|\mathbf{L}\|_{\mathsf{F}}^{2}\enspace

Note that Equation 3 is separably, but not jointly, convex in \mathbf{L} and the \mathbf{s}^{(t)} ’s; typical multi-task approaches would optimize similar objectives [ 19 , 26 ] using alternating optimization.

To enable Equation 3 to be solved online when tasks are observed consecutively, we adapt concepts from the lifelong learning literature. Ruvolo and Eaton [ 34 ] approximate a multi-task objective with a similar form to Equation 3 online as a series of efficient online updates. Note, however, that their approach is designed for the supervised setting, using a general-purpose supervised loss function in place of the MaxEnt negative log-likelihood in Equation 3 , but with a similar factorization of the learned parametric models. Following their approach but substituting in the IRL loss function, for each new task t , we can take a second-order Taylor expansion around the single-task point estimate of \boldsymbol{\alpha}^{(t)}=\operatorname*{argmin}_{\boldsymbol{\alpha}}-\!\sum_{\zeta_{j}^{(t)}\in\mathcal{Z}^{(t)}}\!\log P\bigl{(}\zeta_{j}^{(t)}\mid\boldsymbol{\alpha},T^{(t)}\bigr{)} , and then simplify to reformulate Equation 3 as

\min_{\mathbf{L}}\frac{1}{N}\sum_{t=1}^{N}\min_{\mathbf{s}^{(t)}}\left\{\left(\boldsymbol{\alpha}^{(t)}-\mathbf{L}\mathbf{s}^{(t)}\right)^{\top}\mathbf{H}^{(t)}\left(\boldsymbol{\alpha}^{(t)}-\mathbf{L}\mathbf{s}^{(t)}\right)+\mu\|\mathbf{s}^{(t)}\|_{1}\right\}+\lambda\|\mathbf{L}\|_{\mathsf{F}}^{2}\enspace, (4)

where the Hessian \mathbf{H}^{(t)} of the MaxEnt negative log-likelihood is given by (derivation in Appendix A ):

\mathbf{H}^{(t)}\!=\frac{1}{n_{t}} abla^{2}_{\boldsymbol{\theta},\boldsymbol{\theta}}\mathcal{L}\!\left(\!\mathbf{r}\!\left(\!\mathbf{L}\mathbf{s}^{(t)}\!\right)\!,\mathcal{Z}^{(t)}\!\right)=\Bigg{(}-\sum_{\ \ \ \ \ \tilde{\zeta}\in\mathcal{Z}_{\mathit{MDP}}}\mathbf{x}_{\tilde{\zeta}}P(\tilde{\zeta}|\boldsymbol{\theta})\!\!\Bigg{)}\!\Bigg{(}\sum_{\ \ \ \ \ \tilde{\zeta}\in\mathcal{Z}_{\mathit{MDP}}}\mathbf{x}_{\tilde{\zeta}}^{\top}\!P(\tilde{\zeta}|\boldsymbol{\theta})\!\!\Bigg{)}+\sum_{\ \ \ \ \ \tilde{\zeta}\in\mathcal{Z}_{\mathit{MDP}}}\mathbf{x}_{\tilde{\zeta}}\mathbf{x}_{\tilde{\zeta}}^{\top}\!P(\tilde{\zeta}|\boldsymbol{\theta})\enspace

Since \mathbf{H}^{(t)} is non-linear in the feature counts, we cannot make use of the state visitation frequencies obtained for the MaxEnt gradient in the lifelong learning setting. This creates the need for obtaining a sample-based approximation. We first solve the MDP for an optimal policy \pi^{\boldsymbol{\alpha}^{(t)}} from the parameterized reward learned by single-task MaxEnt. We compute the feature counts for a fixed number of finite horizon paths by following the stochastic policy \pi^{\boldsymbol{\alpha}^{(t)}} . We then obtain the sample covariance of the feature counts of the paths as an approximation of the true covariance in Equation 5 .

Given each new consecutive task t , we first estimate \boldsymbol{\alpha}^{(t)} as described above. Then, Equation 4 can be approximated online as a series of efficient update equations [ 34 ] :

\mathbf{s}^{(t)}\!\leftarrow \operatorname*{argmin}_{\mathbf{s}}\ell\!\left(\!\mathbf{L}_{N},\mathbf{s},\boldsymbol{\alpha}^{(t)}\!,\mathbf{H}^{(t)}\!\right) \mathbf{L}_{N+1}\!\leftarrow \operatorname*{argmin}_{\mathbf{L}}\lambda\|\mathbf{L}\|_{\mathsf{F}}^{2}+\frac{1}{N}\!\sum_{t=1}^{N}\!\ell\!\left(\!\mathbf{L},\mathbf{s}^{(t)}\!,\boldsymbol{\alpha}^{(t)}\!,\mathbf{H}^{(t)}\!\right)\enspace, (6)

where \ell\left(\mathbf{L},\mathbf{s},\boldsymbol{\alpha},\mathbf{H}\right)=\mu\|\mathbf{s}\|_{1}+(\boldsymbol{\alpha}-\mathbf{L}\mathbf{s})^{\top}\mathbf{H}(\boldsymbol{\alpha}-\mathbf{L}\mathbf{s}) , and \mathbf{L} can be built incrementally in practice (see [ 34 ] for details). Critically, this online approximation removes the dependence of Equation 3 on the numbers of training samples and tasks, making it scalable for lifelong learning, and provides guarantees on its convergence with equivalent performance to the full multi-task objective [ 34 ] . Note that the \mathbf{s}^{(t)} coefficients are only updated while training on task t and otherwise remain fixed.

This process yields the estimated reward function as \mathbf{r}^{(t)}_{s_{i}}=\mathbf{L}\mathbf{s}^{(t)}\mathbf{x}_{s_{i}} . We can then solve the now-complete MDP for the optimal policy using standard RL. The complete ELIRL algorithm is given as Algorithm 1 . ELIRL can either support a common feature space across tasks, or can support different feature spaces across tasks by making use of prior work in autonomous cross-domain transfer [ 7 ] , as shown in Appendix C .