SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[59] E. Klein, M. Geist, B. Piot, and O. Pietquin, "Inverse reinforcement learning through structured classification," Advances in neural information processing systems, vol. 25, 2012.
[60] D. Lee, S. Srinivasan, and F. Doshi-Velez, "Truly batch apprenticeship learning with deep successor features," arXiv preprint arXiv:1903.10077, 2019.
[61] K. Zolna, A. Novikov, K. Konyushkova, C. Gulcehre, Z. Wang, Y. Aytar, M. Denil, N. de Freitas, and S. Reed, "Offline learning from demonstrations and unlabeled experience," arXiv preprint arXiv:2011.13885, 2020.
[62] M. Abdulhai, N. Jaques, and S. Levine, "Basis for intentions: Efficient inverse reinforcement learning using past experience," arXiv preprint arXiv:2208.04919, 2022.
[63] T. Xu, Z. Li, and Y. Yu, "Error bounds of imitating policies and environments," Advances in Neural Information Processing Systems, vol. 33, pp. 15737-15749, 2020.
[64] D. Jarrett, I. Bica, and M. van der Schaar, "Strictly batch imitation learning by energy-based distribution matching," Advances in Neural Information Processing Systems, vol. 33, pp. 7354-7365, 2020.
[65] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn, "Visual adversarial imitation learning using variational models," Advances in Neural Information Processing Systems, vol. 34, pp. 3016-3028, 2021.
[66] Y. Liu, A. Swaminathan, A. Agarwal, and E. Brunskill, "Off-policy policy gradient with state distribution correction," arXiv preprint arXiv:1904.08473, 2019.
[67] O. Nachum, B. Dai, I. Kostrikov, Y. Chow, L. Li, and D. Schuurmans, "Algaedice: Policy gradient from arbitrary experience," arXiv preprint arXiv:1912.02074, 2019.
[68] C.-A. Cheng, T. Xie, N. Jiang, and A. Agarwal, "Adversarily trained actor critic for offline reinforcement learning," in International Conference on Machine Learning. PMLR, 2022, pp. 3852-3878.
[69] A. Agarwal, N. Jiang, S. M. Kakade, and W. Sun, "Reinforcement learning: Theory and algorithms," CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, pp. 10-4, 2019.
[70] T. Xu, Z. Wang, and Y. Liang, "Improving sample complexity bounds for (natural) actor-critic algorithms," Advances in Neural Information Processing Systems, vol. 33, pp. 4358-4369, 2020.
Figure 3: The performance of Offline ML-IRL in different environments given 1,000 expert demonstrations
Dataset type	Environment	Offline ML-IRL	BC	ValueDICE	CLARE	Expert Performance
medium	hopper	1750.59 ± 507.06	843.59 ± 503.83	2417.83 ± 1258.30	1559.00 ± 324.10	3533.90
medium	halfcheetah	7690.30 ± 119.17	2799.40 ± 1046.00	-175.87 ± 343.57	240.50 ± 143.25	12107.51
medium	walker2d	4121.68 ± 291.86	1248.13 ± 72.89	1794.97 ± 1400.70	320.51 ± 93.07	5284.33
medium-replay	hopper	2395.03 ± 593.16	843.59 ± 503.83	2417.83 ± 1258.30	2369.79 ± 204.23	3533.90
medium-replay	halfcheetah	9313.29 ± 261.94	2799.40 ± 1046.00	-175.87 ± 343.57	343.37 ± 108.00	12107.51
medium-replay	walker2d	4100.99 ± 293.88	1248.13 ± 72.89	1794.97 ± 1400.70	440.78 ± 88.05	5284.33
medium-expert	hopper	3366.23 ± 229.56	843.59 ± 503.83	2417.83 ± 1258.30	3071.31 ± 186.37	3533.90
medium-expert	halfcheetah	10812.15 ± 551.38	2799.40 ± 1046.00	-175.87 ± 343.57	292.95 ± 84.86	12107.51
medium-expert	walker2d	4049.43 ± 1046.61	1248.13 ± 72.89	1794.97 ± 1400.70	548.59 ± 146.09	5284.33
Table 2: MuJoCo Results. The performance versus different datasets and 1,000 expert demonstrations. Here, the expert demonstration dataset only includes a single expert trajectory (1,000 expert transition samples) and the value of expert performance corresponds to the cumulative reward value of the provided expert trajectory.
Limitations and broader impacts
Offline inverse reinforcement learning is a method designed to recover the reward function and the associated optimal policy from an observed expert dataset. However, potential negative societal impacts may arise if the demonstration dataset incorporates low-quality data. For safety-critical applications, such as autonomous driving and clinical decision support, we need to exercise particular caution to avoid the introduction of detrimental biases from the demonstration dataset. Ensuring safe adaptation is vital for real-world applications.
One limitation of our method is that we exclusively use expert demonstration in the process of reward learning. Ideally, reward learning should incorporate varied data sources, like expert demonstrations and preferences (pairwise comparisons). A possible direction for future work is to broaden our algorithm and theoretical analysis for a wider scope in reward learning. This would allow us to develop a more robust reward model through the integration of diverse data inputs.
In this section, we provide implementation details and additional numerical results.
A.1 Detailed experiment setting
In our experiment, we use two kinds of datasets: 1) transition dataset  $\mathcal{D} = \{(s,a,s')\}$  which includes diverse transition samples and is downloaded from D4RL V2; 2) expert demonstration dataset  $\mathcal{D}^{\mathrm{E}} = \{\tau^{\mathrm{E}}\}$  which consists of several expert trajectories and the expert trajectories are collected from an expert-level policy. All evaluations are based on a single NVIDIA GeForce RTX 2080 Ti.
The transition dataset. The transition datasets are downloaded from D4RL V2 and the transition datasets for each specific task have three different type: medium, medium-replay and medium-expert. In D4RL V2, the datasets are generated as follows: medium: use SAC to train a policy with medium-level performance, then use it to collected 1 million transition samples; medium-replay: use SAC to train a policy until an environment-specific performance is obtained, then save all transition samples in the replay buffer; medium-expert: combine 1 million transition samples collected from a
Figure 4: The performance of Offline ML-IRL in different environments given 10,000 expert demonstrations
Dataset type	Environment	Offline ML-IRL	BC	ValueDICE	CLARE	Expert Performance
medium	hopper	2461.45 ± 705.70	3236.70 ± 45.97	3442.82 ± 199.21	3357.52 ± 270.45	3512.64 ± 17.10
medium	halfcheetah	7706.43 ± 159.39	4935.60 ± 2835.55	3248.60 ± 2001.05	841.46 ± 344.06	12156.16 ± 88.01
medium	walker2d	4195.36 ± 352.86	2822.56 ± 978.97	3046.76 ± 2001.28	825.15 ± 738.33	5365.62 ± 55.79
medium-replay	hopper	2889.73 ± 542.65	3236.70 ± 45.97	3442.82 ± 199.21	3139.98 ± 478.75	3512.64 ± 17.10
medium-replay	halfcheetah	9383.34 ± 358.67	4935.60 ± 2835.55	3248.60 ± 2001.05	437.18 ± 182.99	12156.16 ± 88.01
medium-replay	walker2d	4092.58 ± 308.71	2822.56 ± 978.97	3046.76 ± 2001.28	869.07 ± 612.56	5365.62 ± 55.79
medium-expert	hopper	3350.79 ± 264.96	3236.70 ± 45.97	3442.82 ± 199.21	3166.69 ± 512.77	3512.64 ± 17.10
medium-expert	halfcheetah	11276.09 ± 551.94	4935.60 ± 2835.55	3248.60 ± 2001.05	2020.51 ± 520.46	12156.16 ± 88.01
medium-expert	walker2d	4363.54 ± 729.60	2822.56 ± 978.97	3046.76 ± 2001.28	3245.79 ± 1911.56	5365.62 ± 55.79
Table 3: MuJoCo Results. The performance versus different datasets and 10,000 expert demonstrations.
medium-level policy with another 1 million transition samples collected from an expert-level policy. As a remark, since we are considering the setting of offline IRL where the ground-truth reward is not accessible, we hide the reward information of those downloaded transition datasets from D4RL V2.
The expert demonstration dataset. The expert demonstration dataset includes the transition samples in several collected expert trajectories. Here, we first train a reinforcement learning agent by SAC under the ground-truth reward function to achieve expert-level performance. Then we save the well-train expert-level policy to collect expert trajectories. For each trajectory, it includes 1,000 consecutive transition samples  $(s,a,s^{\prime})$  in one episode.
In the model-based algorithms like Offline ML-IRL and CLARE, the estimated dynamic models are trained using the transition dataset. After the estimated dynamics model is constructed, the corresponding algorithms (Offline ML-IRL and CLARE) will further utilize the expert trajectories in the expert demonstration dataset  $\mathcal{D}^{\mathrm{E}}$  to recover the ground-truth reward function and imitate the expert behaviors.
For model-free offline imitation learning algorithms like BC and ValueDICE, they directly learn a policy to imitate the expert behaviors. Hence, those model-free offline imitation learning algorithms (BC and ValueDICE) will only utilize the expert demonstration dataset  $D^{\mathrm{E}}$ . Due to the fact that BC and ValueDICE do not use the transition dataset  $\mathcal{D}$ , their recorded performance in Table 1 - 3 is not related to the type of the transition dataset  $\mathcal{D}$ .
In our implementation of Offline ML-IRL, we parameterize the reward network by a (256, 256) MLP with ReLU activation function. The input of the reward network is the state-action pair  $(s, a)$  and the output is the estimated reward value  $r(s, a; \theta)$ . Moreover, we use Adam as the optimizer and the stepsize to update the reward network is set to be  $1 \times 10^{-4}$ . For the policy optimization subroutine (13) - (14), we consider it as a model-based offline RL subtask. Since it is under an entropy-regularized framework, SAC-based algorithm is used as the corresponding RL solver. More specifically, we use model-based offline policy optimization (MOPO) [25] in the RL subroutine (13) - (14). For the implementation of MOPO and the corresponding hyperparameters, we follow the setup provided in [37] which guarantees strong performance through fine-tuning the key hyperparameters of MOPO, including the number of estimated world models, the choice of the penalty function  $U(\cdot, \cdot)$ , the penalty coefficient and the imaginary rollout length in the estimated world model. The official code base of [37] is available in https://openreview.net/forum?id=zz9hXVhf40. During the training of Offline ML-IRL, each time we implement a policy improvement subroutine under the current reward estimator, we update the agent by running MOPO for 20 offline episodes in the estimated world model and under the current reward estimator. After that, we sample expert trajectory
Figure 5: Reward Transfer. The recovered reward by Offline ML-IRL in the medium-expert dataset is transferred to the medium-replay datasets for solving offline RL tasks.
from the expert demonstration dataset  $\mathcal{D}^{\mathrm{E}} = \{\tau^{\mathrm{E}}\}$ , and sample agent trajectory from the estimated world model  $\widehat{P}$  to construct the stochastic reward gradient estimator following the expressing given in (16). Then we are able to update the reward network by a stochastic gradient step.
For the benchmark algorithms, the official code base of CLARE is provided in https://openreview.net/forum?id=5aT4gan0d98. Moreover, the official implementation of ValueDICE is provided in https://github.com/google-research/google-research/tree/master/value_dice.
A.2 Additional numerical results
In Fig. 3 - 4, we show the convergence curves of Offline ML-IRL when expert demonstration datasets  $D^{\mathrm{E}}$  include 1,000 and 10,000 expert demonstrations (1 and 10 expert trajectories) respectively. Moreover, we compare the performance with benchmark algorithms in Table 2 - 3. According to the numerical results, we show that our proposed method Offline ML-IRL is insensitive to the number of expert demonstrations. Even when only 1,000 expert demonstrations are provided, Offline ML-IRL can achieve strong performance (close to the expert-level performance with the medium-expert dataset).
In Fig. 5, we show the numerical results of reward transfer experiment. Here, we first use the medium-expert dataset and expert demonstration dataset to generate a reward estimator by running Offline ML-IRL. Then we transfer the recovered reward function  $r(\cdot, \cdot; \theta)$  to the medium-replay dataset for labelling all transition samples with the estimated reward value. With the transferred reward function  $r(\cdot, \cdot; \theta)$  to label the transition dataset as  $\{(s, a, r(s, a; \theta), s')\}$ , we can treat the problem as an Offline RL task and run MOPO to solve it. The numerical result is shown in Fig. 5. Comparing Fig. 5 with the results in Fig 4 which are obtained by directly running Offline ML-IRL with expert demonstrations / trajectories, similar performance can be obtained by run Offline RL on the transition dataset  $\mathcal{D}$  with the transferred reward estimator. These results suggest that Offline ML-IRL can recover high-quality reward estimator, which can be transferred across different datasets to label those unlabeled data with the estimated reward value.
Inverse reinforcement learning (IRL) consists of estimating the reward function and the optimal policy that best fits the expert demonstrations [16, 17, 18, 20, 31]. In the seminal work [18], a formulation for IRL is proposed based on the principle of maximum entropy which enjoys the theoretical guarantees to recover the reward function [29, 30, 55, 56]. Furthermore, in [57, 20, 21], a series of sample-efficient algorithms are proposed to solve the the maximum entropy IRL formulation. Moreover, there still exists one major limitation in IRL, due to the fact that most existing IRL methods require extensive online trials and errors which could be impractical in real-life applications.
To side-step the online interactions with the environment, offline IRL is considered to recover the reward function from fixed datasets of expert demonstrations. In [13], by taking into account the bias of expert demonstrations, the authors propose a gradient-based IRL methods to jointly estimate the reward function and the transition dynamics. In [14], a model-free algorithm called IQ-Learn is proposed by implicitly representing the reward function and the policy from a soft Q-function.
Although avoiding the online interactions with the environment, IQ-Learn sacrifices the accuracy of the estimated reward function, since its recovered reward is highly dependent on the environment dynamics. In [9], the authors propose a variational Bayesian framework to estimate the approximate posterior distribution of the reward function from a collected demonstration dataset. Recently, in [15], a model-based offline IRL approach (called CLARE) is proposed, which implements IRL algorithm in an estimated dynamics model to learn the reward. To avoid distribution shift, CLARE incorporates conservatism into its estimated reward to ensure the corresponding policy generates a state-action visitation measure to match the joint data distribution of collected transition samples and expert demonstrations. However, when the number of expert demonstrations is limited and/or most of the transition samples about the state-action next state observations are collected from a low-quality behavior policy, matching the empirical state-action visitation measure of all collected data will force the recovered reward / policy to mimic the low-quality behavior policy, which is not enough to guarantee an accurate model of the expert. Moreover, matching the visitation measure is sensitive to the quality of the estimated dynamics model. Matching the visitation measure in an estimated dynamics model with poor prediction quality cannot guarantee high-quality recovered reward.
Here, we discuss the literature in several closely related areas as below.
Online IRL. [18] is a seminal work in which the expert policy is formulated as the model that maximizes entropy subject to a constraint requiring that the expected features under such policy match the empirical averages in the expert's observation dataset. [29, 19] propose algorithms with nested loop structure to solve such maximum entropy estimation problem. Those algorithms with a nested loop structure can suffer from computational burden, because they need to alternate between an outer loop with a reward update step and an inner loop that calculates the explicit policy estimates. In [58], the authors propose a method to estimate both reward functions and constraints from a group of experts. In [21], the authors propose an online IRL formulation based on maximum likelihood estimation perspective, which needs extensive interaction with the environment in order to learn the reward function. When the reward function is linearly parameterized, it is shown that the maximum likelihood IRL formulation is the dual problem to the classic maximum entropy IRL problem. Moreover, a theoretical analysis is provided to ensure the reward parameter can converge to a stationary solution in finite time. Despite the computational efficiency obtained in [21], it is not clear whether an optimal reward estimator can be recovered from the maximum likelihood formulation. As a remark, compared with the online IRL work [21], our paper extends the maximum likelihood formulation to the setting of offline IRL. We further provide a statistical guarantee to show that when the transition dataset has sufficient coverage over the expert-visited state-action space, an estimated world model can be constructed so that the optimal reward estimator can be recovered. Moreover, we provide a computational guarantee to show that the proposed algorithm can identify a stationary point in finite time. When the reward function is linearly parameterized, we show the optimal reward estimator of the maximum likelihood estimation formulation can be recovered by the proposed algorithm.
Offline IRL. In [59], the authors propose a method to perform IRL through constructing a linearly parameterized score function-based multi-class classification algorithm. With an estimated of the feature expectation of the expert, the proposed algorithm is able to avoid direct RL subroutine. Given appropriate heuristic and expert trajectories, the proposed algorithm could recover the underlying reward function without any online environment interactions. In [60], the authors propose a model-free method to construct the Deep Successor Feature Networks (DSFN) to estimate the feature expectations in an off-policy setting. By parameterizing the reward as a linear function and estimating the feature expectation, the offline IRL can be solved computation-friendly with limited expert demonstrations. In [61], a modular algorithm called Offline Reinforcement Imitation Learning (ORIL) is proposed. In ORIL, a reward function is first constructed by constrasting expert demonstrations with other transition samples  $\{(s,a,s')\}$  which are sampled from a behavior policy with unknown quality. With the constructed reward function, the quality of all unlabeled data can be evaluated and then an agent is trained via offline reinforcement learning. In [62], the authors proposed an IRL method which leverages multi-task RL pre-training and successor features to train IRL from past experience. When a set of related tasks can be provided for RL pre-training and the feature expectation is estimated accurately, the proposed method could benefit from the multi-task pre-training. However, in practice, it is difficult to obtain full knowledge of the feature structure in the ground-truth reward function.
Offline Imitation Learning. Different from offline IRL, offline imitation learning aims to directly imitate the expert behavior by learning a policy in the offline setting. In [28], an algorithm called
ValueDICE is proposed to leverage off-policy data to learn an imitation policy with the use of any explicit reward functions. Moreover, the numerical results of ValueDICE show that it could be easily implemented in the offline regime, where additional interactions with the environment is not allowed. In [63], the authors propose method to learn both policies and environment and analyze the corresponding error bounds. In [64], a model-free offline imitation learning algorithm is proposed by energy-based distribution matching (EDM). EDM provides an effective way to minimize the divergence between the state-action visitation measure of the demonstrator and the imitation policy. In [65], the authors develop a variational model-based adversarial imitation learning (V-MAIL) algorithm for learning from visual demonstrations. By constructing a variational latent-space dynamics model, V-MAIL is able to solve high-dimensional visual tasks without any additional environment interactions. In [22], the authors introduce Model-based Imitation Learning from Offline data (MILO), which extends model-based offline reinforcement learning to imitation learning. The theoretical analysis of MILO shows that full coverage of the offline data is not necessary. When the offline dataset provides sufficient coverage to cover the expert-visited state-actions, MILO can provably avoid distribution shift in offline imitation learning by leveraging a constructed dynamics model.
Offline RL. Offline RL considers the problem of learning a policy from a fixed datasets where the reward value is provided for each collected transition samples. For model-free offline RL algorithms, a world model is not estimated and the algorithms directly learn a policy from the collected dataset. In [66, 67], model-free offline RL algorithms are proposed to solve the importance sampling problem. In [23, 68], conservatism is incorporated into the value function to avoid overestimation in the offline RL setting. For the model-based offline RL algorithms, [24] constructs the estimated world model and sets hard threshold on the model uncertainty for constructing terminating states to avoid dangerous explorations. In [25], the authors proposes a model-based offline policy optimization algorithm (MOPO) which utilizes uncertainty estimation techniques to construct a penalty function to regularize the reward function. Therefore, MOPO can learn a conservative policy which stays in the low-uncertainty region to aviod the distribution shift issue. As a follow-up work, [37] revisits the design choices of several key hyperparameters in MOPO and fine-tune the corresponding hyperparameters in MOPO to guarantee strong performance. In [26], the authors propose a model-based offline RL algorithm called COMBO which does not rely on explicit uncertainty estimation. By regularizing the value function on out-of-distribution state-action pairs generated in the estimated world model, COMBO can benefit from the conservatism without requiring explicit uncertainty estimation techniques. As a remark, the algorithms proposed in [25, 24, 37, 26] all perform conservative policy optimization in a well-constructed dynamics model and the estimated dynamics model keeps fixed during the training of the RL agent. Different from those algorithms mentioned above, [45, 47] incorporate conservatism into the constructed dynamics model. By adversarially modifying the estimated dynamics model to minimize the value function under the current policy, the proposed methods can learn a robust policy with respect to the environment dynamics and can obtain probably approximately correct (PAC) performance guarantee.
C Auxiliary lemmas
Before we introduce the auxiliary lemmas, we re-write Assumptions 1 - 3 here for convenience.
Assumption 1 For any reward parameter  $\theta$  and any state-action pair  $(s, a)$ , the following conditions hold:
$$
\left| r (s, a; \theta) \right| \leq C _ {r}, \quad \left| U (s, a) \right| \leq C _ {u}
$$
Assumption 2 Given any policy  $\pi$ , the Markov chain under the estimated world model  $\widehat{P}$  is irreducible and aperiodic. There exist constants  $\kappa > 0$  and  $\rho \in (0,1)$  to ensure the following condition holds:
Assumption 3 Under any reward parameter  $\theta$ , the following conditions hold for any  $s \in S$  and  $a \in \mathcal{A}$ :
$$
\begin{array}{l} \left\| \nabla_ {\theta} r (s, a; \theta) \right\| \leq L _ {r}, \\ \left| \left| \nabla_ {\theta} r (s, a; \theta_ {1}) - \nabla_ {\theta} r (s, a; \theta_ {2}) \right| \right| \leq L _ {g} \| \theta_ {1} - \theta_ {2} \|, \\ \end{array}
$$
Then we introduce the auxiliary lemmas as below.