RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Inverse Reinforcement Learning
We first describe IRL and the MaxEnt IRL method, before introducing the lifelong IRL problem.
The Inverse RL Problem
A Markov decision process (MDP) is defined as a tuple \langle\mathcal{S},\mathcal{A},T,\mathbf{r},\gamma\rangle , where \mathcal{S} is the set of states, \mathcal{A} is the set of actions, the transition function T:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\mapsto[0,1] gives the probability P(s_{i+1}\mid s_{i},a_{i}) that being in state s_{i} and taking action a_{i} will yield a next state s_{i+1} , \mathbf{r}:\mathcal{S}\mapsto\mathbb{R} is the reward function 2 2 2 Although we typically notate functions as uppercase non-bold symbols, we notate the reward function as \mathbf{r} , since primarily it will be represented as a parameterized function of the state features and a target for learning. , and \gamma\in[0,1) is the discount factor. A policy \pi:\mathcal{S}\times\mathcal{A}\mapsto[0,1] models the distribution P(a_{i}\mid s_{i}) over actions the agent should take in any state. When fully specified, an MDP can be solved via linear or dynamic programming for an optimal policy \pi^{*} that maximizes the rewards earned by the agent: \pi^{*}=\operatorname*{argmax}_{\pi}V^{\pi} , with V^{\pi}=\mathbb{E}_{\pi}\left[\sum_{i}\gamma^{i}\mathbf{r}(s_{i})\right] .

In IRL [ 29 ] , the agent does not know the MDP’s reward function, and must infer it from demonstrations \mathcal{Z}=\{\zeta_{1},\dots,\zeta_{n}\} given by an expert user. Each demonstration \zeta_{j} is a sequence of state-action pairs [\mathbf{s}_{0:H},\mathbf{a}_{0:H}] that is assumed to be generated by the user’s unknown policy \hat{\pi}^{*} . Once the reward function is learned, the MDP is complete and so can be solved for the optimal policy \pi^{*} .

Given an MDP\ \mathbf{r}=\langle\mathcal{S},\mathcal{A},T,\gamma\rangle and expert demonstrations \mathcal{Z} , the goal of IRL is to estimate the unknown reward function \mathbf{r} of the MDP. Previous work has defined the optimal reward such that the policy enacted by the user be (near-)optimal under the learned reward ( {V^{\pi^{*}}=V^{\hat{\pi}^{*}}} ), while (nearly) all other actions would be suboptimal. This problem is unfortunately ill-posed, since it has numerous solutions, and so it becomes necessary to make additional assumptions in order to find solutions that generalize well. These various assumptions and the strategies to recover the user’s policy have been the focus of previous IRL research. We next focus on the MaxEnt approach to the IRL problem.
Maximum Entropy IRL
In the maximum entropy (MaxEnt) algorithm for IRL [ 43 ] , each state s_{i} is represented by a feature vector \mathbf{x}_{s_{i}}\in\mathbb{R}^{d} . Each demonstrated trajectory \zeta_{j} gives a feature count \mathbf{x}_{\zeta_{j}}=\sum_{i=0}^{H}\gamma^{i}\mathbf{x}_{s_{i}} , giving an approximate expected feature count \tilde{\mathbf{x}}=\frac{1}{n}\sum_{j}\mathbf{x}_{\zeta_{j}} that must be matched by the agent’s policy to satisfy the condition V^{\pi^{*}}=V^{\hat{\pi}^{*}} . The reward function is represented as a parameterized linear function with weight vector \boldsymbol{\theta}\in\mathbb{R}^{d} as \mathbf{r}_{s_{i}}=\mathbf{r}(\mathbf{x}_{s_{i}},\boldsymbol{\theta})=\boldsymbol{\theta}^{\top}\mathbf{x}_{s_{i}} and so the cumulative reward of a trajectory \zeta_{j} is given by \mathbf{r}_{\zeta_{j}}=\mathbf{r}(\mathbf{x}_{\zeta_{j}},\boldsymbol{\theta})=\sum_{s_{i}\in\zeta_{j}}\gamma^{i}\boldsymbol{\theta}^{\top}\mathbf{x}_{s_{i}}=\boldsymbol{\theta}^{\top}\mathbf{x}_{\zeta_{j}} .

The algorithm deals with the ambiguity of the IRL problem in a probabilistic way, by assuming that the user acts according to a MaxEnt policy. In this setting, the probability of a trajectory is given as: P(\zeta_{j}\mid\boldsymbol{\theta},T)\approx\frac{1}{Z(\boldsymbol{\theta},T)}\exp(\mathbf{r}_{\zeta_{j}})\prod_{(s_{i},a_{i},s_{i+1})\in\zeta_{j}}T(s_{i+1}\mid s_{i},a_{i}) , where Z(\boldsymbol{\theta},T) is the partition function, and the approximation comes from assuming that the transition uncertainty has little effect on behavior. This distribution does not prefer any trajectory over another with the same reward, and exponentially prefers trajectories with higher rewards. The IRL problem is then solved by maximizing the likelihood of the observed trajectories {\boldsymbol{\theta}^{*}=\operatorname*{argmax}_{\boldsymbol{\theta}}\log P(\mathcal{Z}\mid\boldsymbol{\theta})=\operatorname*{argmax}_{\boldsymbol{\theta}}\!\sum_{\zeta_{j}\in\mathcal{Z}}\!\log P(\zeta_{j}\mid\boldsymbol{\theta},T)} . The gradient of the log-likelihood is the difference between the user’s and the agent’s feature expectations, which can be expressed in terms of the state visitation frequencies D_{s} : \tilde{\mathbf{x}}-\sum_{\tilde{\zeta}\in\mathcal{Z}_{\mathit{MDP}}}P(\tilde{\zeta}\mid\boldsymbol{\theta},T)\mathbf{x}_{\tilde{\zeta}}=\tilde{\mathbf{x}}-\sum_{s\in\mathcal{S}}D_{s}\mathbf{x}_{s} ,
where \mathcal{Z}_{\mathit{MDP}} is the set of all possible trajectories. The D_{s} can be computed efficiently via a forward-backward algorithm [ 43 ] . The maximum of this concave objective is then achieved when the feature counts match, and so V^{\pi^{*}}=V^{\hat{\pi}^{*}} .
The Lifelong Inverse RL Problem
We now introduce the novel problem of lifelong IRL. In contrast to most previous work on IRL, which focuses on single-task learning, this paper focuses on online multi-task IRL. Formally, in the lifelong learning setting, the agent faces a sequence of IRL tasks \mathcal{T}^{(1)},\dots,\mathcal{T}^{(N_{max})} , each of which is an MDP\ \mathbf{r} \,{\mathcal{T}^{(t)}=\left<\mathcal{S}^{(t)},\mathcal{A}^{(t)},T^{(t)},\gamma^{(t)}\right>} . The agent will learn tasks consecutively, receiving multiple expert demonstrations for each task before moving on to the next. We assume that a priori the agent does not know the total number of tasks N_{max} , their distribution, or the order of the tasks.

The agent’s goal is to learn a set of reward functions {\mathcal{R}=\bigl{\{}\mathbf{r}(\boldsymbol{\theta}^{(1)}),\dots,\mathbf{r}(\boldsymbol{\theta}^{(N_{max})})\bigr{\}}} with a corresponding set of parameters \varTheta=\bigl{\{}\boldsymbol{\theta}^{(1)},\dots,\boldsymbol{\theta}^{(N_{max})}\bigr{\}} . At any time, the agent may be evaluated on any previous task, and so must strive to optimize its performance for all tasks \mathcal{T}^{(1)},\dots,\mathcal{T}^{(N)} , where N denotes the number of tasks seen so far ( 1\leq N\leq N_{max} ). Intuitively, when the IRL tasks are related, knowledge transfer between their reward functions has the potential to improve the learned reward function for each task and reduce the number of expert demonstrations needed.

After N tasks, the agent must optimize the likelihood of all observed trajectories over those tasks:

\max_{\mathbf{r}^{(1)},\ldots,\mathbf{r}^{(N)}}P\left(\mathbf{r}^{(1)},\ldots,\mathbf{r}^{(N)}\right)\prod_{t=1}^{N}\left(\prod_{j=1}^{n_{t}}P\left(\zeta_{j}\mid\mathbf{r}^{(t)}\right)\right)^{\frac{1}{n_{t}}}\enspace, (1)

where P(\mathbf{r}^{(1)},\ldots,\mathbf{r}^{(N)}) is a reward prior to encourage relationships among the reward functions, and each task is given equal importance by weighting it by the number of associated trajectories n_{t} .
Lifelong Inverse Reinforcement Learning
The key idea of our framework is to use lifelong function approximation to represent the reward functions for all tasks, enabling continual online transfer between the reward functions with efficient per-task updates. Intuitively, this framework exploits the fact that certain aspects of the reward functions are often shared among different (but related) tasks, such as the negative reward a service robot might receive for dropping objects. We assume the reward functions \mathbf{r}^{(t)} for the different tasks are related via a latent basis of reward components \mathbf{L} . These components can be used to reconstruct the true reward functions via a sparse combination of such components with task-specific coefficients \mathbf{s}^{(t)} , using \mathbf{L} as a mechanism for transfer that has shown success in previous work [ 19 , 26 ] .

This section develops our framework for lifelong IRL, instantiating it following the MaxEnt approach to yield the ELIRL algorithm. Although we focus on MaxEnt IRL, ELIRL can easily be adapted to other IRL approaches, as shown in Appendix D . We demonstrate the merits of the novel lifelong IRL problem by showing that 1) transfer between IRL tasks can significantly increase their accuracy and 2) this transfer can be achieved by adapting ideas from lifelong learning in supervised settings.
Improving Performance on Earlier Tasks
As ELIRL is trained over multiple IRL tasks, it gradually refines the shared knowledge in \mathbf{L} . Since each reward function’s parameters are modeled as \boldsymbol{\theta}^{(t)}=\mathbf{L}\mathbf{s}^{(t)} , subsequent changes to \mathbf{L} after training on task t can affect \boldsymbol{\theta}^{(t)} . Typically, this process improves performance in lifelong learning [ 34 ] , but it might occasionally decrease performance through negative transfer, due to the ELIRL simplifications restricting that \mathbf{s}^{(t)} is fixed except when training on task t . To prevent this problem, we introduce a novel technique. Whenever ELIRL is tested on a task t , it can either directly use the \boldsymbol{\theta}^{(t)} vector obtained from \mathbf{L}\mathbf{s}^{(t)} , or optionally repeat the optimization step for \mathbf{s}^{(t)} in Equation 6 to account for potential major changes in the \mathbf{L} matrix since the last update to \mathbf{s}^{(t)} . This latter optional step only involves running an instance of the LASSO, which is highly efficient. Critically, it does not require either re-running MaxEnt or recomputing the Hessian, since the optimization is always done around the optimal single-task parameters, \boldsymbol{\alpha}^{(t)} . Consequently, ELIRL can pay a small cost to do this optimization when it is faced with performing on a previous task, but it gains potentially improved performance on that task by benefiting from up-to-date knowledge in \mathbf{L} , as shown in our results.
Computational Complexity
The addition of a new task to ELIRL requires an initial run of single-task MaxEnt to obtain \boldsymbol{\alpha}^{(t)} , which we assume to be of order \mathcal{O}(i\xi(d,|\mathcal{A}|,|\mathcal{S}|)) , where i is the number of iterations required for MaxEnt to converge. The next step is computing the Hessian, which costs \mathcal{O}(MH+Md^{2}) , where M is the number of trajectories sampled for the approximation and H is their horizon. Finally, the complexity of the update steps for \mathbf{L} and \mathbf{s}^{(t)} is \mathcal{O}(k^{2}d^{3}) [ 34 ] . This yields a total per-task cost of \mathcal{O}(i\xi(d,|\mathcal{A}|,|\mathcal{S}|)+MH+Md^{2}+k^{2}d^{3}) for ELIRL. The optional step of re-updating \mathbf{s}^{(t)} when needing to perform on task t would incur a computational cost of {\mathcal{O}(d^{3}+kd^{2}+dk^{2})} for constructing the target of the optimization and running LASSO [ 34 ] .

Notably, there is no dependence on the number of tasks N , which is precisely what makes ELIRL suitable for lifelong learning. Since IRL in general requires finding the optimal policy for different choices of the reward function as an inner loop in the optimization, the additional dependence on N would make any IRL method intractable in a lifelong setting. Moreover, the only step that depends on the size of the state and action spaces is single-task MaxEnt. Thus, for high-dimensional tasks (e.g., robotics tasks), replacing the base learner would allow our algorithm to scale gracefully.
Theoretical Convergence Guarantees
ELIRL inherits the theoretical guarantees showed by Ruvolo and Eaton [ 34 ] . Specifically, the optimization is guaranteed to converge to a local optimum of the approximate cost function in Equation 4 as the number of tasks grows large. Intuitively, the quality of this approximation depends on how much the factored representation \boldsymbol{\theta}^{(t)}=\mathbf{L}\mathbf{s}^{(t)} deviates from \boldsymbol{\alpha}^{(t)} , which in turn depends on how well this representation can capture the task relatedness. However, we emphasize that this approximation is what allows the method to solve the multi-task learning problem online, and it has been shown empirically in the contexts of supervised learning [ 34 ] and RL [ 6 ] that this approximate solution can achieve equivalent performance to exact multi-task learning in a variety of problems.

[ \capbeside \thisfloatsetup capbesideposition=left,top,capbesidewidth=0.3]figure[ \FBwidth ] (a) Objectworld (b) Highway
Experimental Results
We evaluated ELIRL on two environments, chosen to allow us to create arbitrarily many tasks with distinct reward functions. This also gives us known rewards as ground truth. No previous multi-task IRL method was tested on such a large task set, nor on tasks with varying state spaces as we do.

Objectworld: Similar to the environment presented by Levine et al. [ 21 ] , Objectworld is a 32\times 32 grid populated by colored objects in random cells. Each object has one of five outer colors and one of two inner colors, and induces a constant reward on its surrounding 5\times 5 grid. We generated 100 tasks by randomly choosing 2–4 outer colors, and assigning to each a reward sampled uniformly from [-10,5] ; the inner colors are distractor features. The agent’s goal is then to move toward objects with “good” (positive) colors and away from objects with “bad” (negative) colors. Ideally, each column of \mathbf{L} would learn the impact field around one color, and the \mathbf{s}^{(t)} ’s would encode how good or bad each color is in each task. There are d=31(5+2) features, representing the distance to the nearest object with each outer and inner color, discretized as binary indicators of whether the distance is less than 1–31 . The agent can choose to move along the four cardinal directions or stay in place.

Highway: Highway simulations have been used to test various IRL methods [ 1 , 21 ] . We simulate the behavior of 100 different drivers on a three-lane highway in which they can drive at four speeds. Each driver prefers either the left or the right lane, and either the second or fourth speed. Each driver’s weight for those two factors is sampled uniformly from [0,5] . Intuitively, each column of \mathbf{L} should learn a speed or lane, and the \mathbf{s}^{(t)} ’s should encode the drivers’ preferences over them. There are d=4+3+64 features, representing the current speed and lane, and the distances to the nearest cars in each lane in front and back, discretized in the same manner as Objectworld. Each time step, drivers can choose to move left or right, speed up or slow down, or maintain their current speed and lane.

In both environments, the agent’s chosen action has a 70\% probability of success and a 30\% probability of a random outcome. The reward is discounted with each time step by a factor of \gamma=0.9 .
Evaluation Procedure
For each task, we created an instance of the MDP by placing the objects in random locations. We solved the MDP for the true optimal policy, and generated simulated user trajectories following this policy. Then, we gave the IRL algorithms the MDP\ \mathbf{r} and the trajectories to estimate the reward \mathbf{r} . We compared the learned reward function with the true reward function by standardizing both and computing the \ell_{2} -norm of their difference. Then, we trained a policy using the learned reward function, and compared its expected return to that obtained by a policy trained using the true reward.

We tested ELIRL using \mathbf{L} trained on various subsets of tasks, ranging from 10 to 100 tasks. At each testing step, we evaluated performance of all 100 tasks; this includes as a subset evaluating all previously observed tasks, but it is significantly more difficult because the latent basis \mathbf{L} , which is trained only on the initial tasks, must generalize to future tasks. The single-task learners were trained on all tasks, and we measured their average performance across all tasks. All learners were given n_{t}=32 trajectories for Objectworld and n_{t}=256 trajectories for Highway, all of length H=16 . We chose the size k of \mathbf{L} via domain knowledge, and initialized \mathbf{L} sequentially with the \boldsymbol{\alpha}^{(t)} ’s of the first k tasks. We measured performance on a new random instance of the MDP for each task, so as not to conflate overfitting the training environment with high performance. Results were averaged over 20 trials, each using a random task ordering.

We compared ELIRL with both the original (ELIRL) and re-optimized (ELIRLre) \mathbf{s}^{(t)} vectors to MaxEnt IRL (the base learner) and GPIRL [ 21 ] (a strong single-task baseline). None of the existing multi-task IRL methods were suitable for this experimental setting—other methods assume a shared state space and are prohibitively expensive for more than a few tasks [ 12 , 2 , 11 ] , or only learn different experts’ approaches to a single task [ 38 ] . Appendix B includes a comparison to MTMLIRL [ 2 ] on a simplified version of Objectworld, since MTMLIRL was unable to handle the full version.
Results
Figure 1 shows the advantage of sharing knowledge among IRL tasks. ELIRL learned the reward functions more accurately than its base learner, MaxEnt IRL, after sufficient tasks were used to train the knowledge base \mathbf{L} . This directly translated to increased performance of the policy trained using the learned reward function. Moreover, the \mathbf{s}^{(t)} re-optimization (Section 5.2 ) allowed ELIRLre to outperform GPIRL, by making use of the most updated knowledge.

As shown in Table 1 , ELIRL requires little extra training time versus MaxEnt IRL, even with the optional \mathbf{s}^{(t)} re-optimization, and runs significantly faster than GPIRL. The re-optimization’s additional time is nearly imperceptible. This signifies a clear advantage for ELIRL when learning multiple tasks in real-time.

In order to analyze how ELIRL captures the latent structure underlying the tasks, we created new instances of Objectworld and used a single learned latent component as the reward of each new MDP (i.e., a column of \mathbf{L} , which can be treated as a latent reward function factor). Figure 2 shows example latent components learned by the algorithm, revealing that each latent component represents the 5\times 5 grid around a particular color or small subset of the colors.

We also examined how performance on the earliest tasks changed during the lifelong learning process. Recall that as ELIRL learns new tasks, the shared knowledge in \mathbf{L} continually changes. Consequently, the modeled reward functions for all tasks continue to be refined automatically over time, without retraining on the tasks. To measure this effect of “reverse transfer” [ 34 ] , we compared the performance on each task when it was first encountered to its performance after learning all tasks, averaged over 20 random task orders. Figure 3 reveals that ELIRL improves previous tasks’ performance as \mathbf{L} is refined, achieving reverse transfer in IRL. Reverse transfer was further improved by the \mathbf{s}^{(t)} re-optimization.

[ \capbeside \thisfloatsetup capbesideposition=left,top,capbesidewidth=0.4]figure[ \FBwidth ] (a) Cross-domain transfer (b) Continuous domains
ELIRL Extensions to Cross-Domain Transfer and Continuous State-Action Spaces
We performed additional experiments to show how simple extensions to ELIRL can transfer knowledge across tasks with different feature spaces and with continuous state-action spaces.

ELIRL can support transfer across task domains with different feature spaces by adapting prior work in cross-domain transfer [ 7 ] ; details of this extension are given in Appendix C . To evaluate cross-domain transfer, we constructed 40 Objectworld domains with different feature spaces by varying the grid sizes from 5 to 24 and letting the number of outer colors be either 3 or 5. We created 10 tasks per domain, and provided the agents with 16 demonstrations per task, with lengths varying according to the number of cells in each domain. We compared MaxEnt IRL, in-domain ELIRL with the original (ELIRL) and re-optimized (ELIRLre) \mathbf{s}^{(t)} ’s, and cross-domain ELIRL with the original (CD-ELIRL) and reoptimized (CD-ELIRLre) \mathbf{s}^{(t)} ’s, averaged over 10 random task orderings. Figure 4(a) shows how cross-domain transfer improved the performance of an agent trained only on tasks within each domain. Notice how the \mathbf{s}^{(t)} re-optimization compensates for the major changes in the shared knowledge that occur when the agent encounters tasks from different domains.

We also explored an extension of ELIRL to continuous state spaces, as detailed in Appendix D . To evaluate this extension, we used a continuous planar navigation task similar to that presented by Levine and Koltun [ 20 ] . Analogous to Objectworld, this continuous environment contains randomly distributed objects that have associated rewards (sampled randomly), and each object has an area of influence defined by a radial basis function. Figure 4(b) shows the performance of ELIRL on 50 continuous navigation tasks averaged over 20 different task orderings, compared against the average performance of the single-task AME-IRL algorithm [ 20 ] across all tasks. These results show that ELIRL is able to achieve better performance in the continuous space than the single-task learner, once a sufficient number of tasks has been observed.
Conclusion
We introduced the novel problem of lifelong IRL, and presented a general framework that is capable of sharing learned knowledge about the reward functions between IRL tasks. We derived an algorithm for lifelong MaxEnt IRL, and showed how it can be easily extended to handle different single-task IRL methods and diverse task domains. In future work, we intend to study how more powerful base learners can be used for the learning of more complex tasks, potentially from human demonstrations.
Acknowledgements
This research was partly supported by AFRL grant #FA8750-16-1-0109 and DARPA agreement #FA8750-18-2-0117. We would like
to thank the anonymous reviewers for their helpful feedback.
Figure 1 : Average reward and value difference in the lifelong setting. Reward difference measures the error between learned and true reward. Value difference compares expected return from the policy trained on the learned reward and the policy trained on the true reward. The whiskers denote std. error. ELIRL improves as the number of tasks increases, achieving better performance than its base learner, MaxEnt IRL.
Using re-optimization after learning all tasks allows earlier tasks to benefit from the latest knowledge, increasing ELIRL’s performance above GPIRL. (Best viewed in color.); Average reward and value difference in the lifelong setting. Reward difference measures the error between learned and true reward. Value difference compares expected return from the policy trained on the learned reward and the policy trained on the true reward. The whiskers denote std. error. ELIRL improves as the number of tasks increases, achieving better performance than its base learner, MaxEnt IRL.
Using re-optimization after learning all tasks allows earlier tasks to benefit from the latest knowledge, increasing ELIRL’s performance above GPIRL. (Best viewed in color.)
Figure B.1 : Comparison against MTMLIRL. The reward function learned by ELIRL was more accurate than that learned by MTMLIRL on the small-scale Objectworld experiment.; Figure B.1 : Comparison against MTMLIRL. The reward function learned by ELIRL was more accurate than that learned by MTMLIRL on the small-scale Objectworld experiment.
(a) Green and yellow; (b) Green, blue, yellow; (c) Orange; Example latent reward functions from Objectworld learned by ELIRL. Each column of \mathbf{L} can be visualized as a reward function, and captures a reusable chunk of knowledge.
The grayscale values show the learned reward and the arrows show the corresponding optimal policy. Each latent component has specialized to focus on objects of particular colors, as labeled. (Best viewed in color.)