When Demonstrations meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning
Authors: Siliang Zeng
Chenliang Li
Alfredo Garcia
Mingyi Hong
Venue: NeurIPS 2023 oral (2023)
Keywords: Inverse Reinforcement Learning
Model-based Offline Inverse Reinforcement Learning

Abstract: Offline inverse reinforcement learning (Offline IRL) aims to recover the structure of rewards and environment dynamics that underlie observed actions in a fixed, finite set of demonstrations from an expert agent. Accurate models of expertise in executing a task has applications in safety-sensitive applications such as clinical decision making and autonomous driving. However, the structure of an expert's preferences implicit in observed actions is closely linked to the expert's model of the environment dynamics (i.e. the ``world''). Thus, inaccurate models of the world obtained from finite data with limited coverage could compound inaccuracy in estimated rewards. To address this issue, we propose a bi-level optimization formulation of the estimation task wherein the upper level is likelihood maximization based upon a conservative model of the expert's policy (lower level). The policy model is conservative in that it maximizes reward subject to a penalty that is increasing in the uncertainty of the estimated model of the world. We propose a new algorithmic framework to solve the bi-level optimization problem formulation and provide statistical and computational guarantees of performance for the associated optimal reward estimator. Finally,  we demonstrate that the proposed algorithm outperforms the state-of-the-art offline IRL and imitation learning benchmarks by a large margin, over the continuous control tasks in MuJoCo and different datasets in the D4RL benchmark.

When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning
Siliang Zeng" University Minnesota Minneapolis, MN, USA zeng01760umn.co edu
Chenliang Li Texas A&M University College Station, TX, USA chenliangl @tamu. edu
Alfredo Garcia Texas A&M University College Station, TX, USA alfredo garcia@tamu.co edu
Mingyi Hong
University of Minnesota Minncapolis, MN, USA mhong@umn edu
Abstract
Offline inverse reinforcement learning (Offline IRL) aims to recover the structure rewards and environment dynamics that underlie observed actions in fixed, finite set of demonstrations from an expert agent. Accurate models of expertise in executing task has applications in safety-sensitive applications such as clinical decision making and autonomous driving. However, the structure of an expert's s preferences implicit observed actions is closely linked to the expert's model of the environment dynamics (i.e. the "world" model). Thus, inaccurate models the world obtained from finite data with limited coverage could compound inaccuracy in estimated rewards. To address this issue, we propose bi-level optimization formulation maximization based upon conservative model of the expert's policy (lower level).
estimation task wherein the upper level is likelihood
The policy model is conservative in that maximizes reward subject to a penalty that is increasing in the uncertainty of the estimated model of the world. We propose a new algorithmic framework to solve the bi-level optimization problem formulation and provide statistical and computational guarantees of performance for the associated optimal reward estimator. Finally, we demonstrate that the proposed algorithm outperforms the state-of-the--e offline IRL and imitation learning benchmarks by large margin. over the continuous control tasks in MuJoCo and different datasets in the D4RL benchmark
Introduction
Reinforcement learning (RL) is powerful and promising approach for solving large-scale sequential decision- maasinn problems 1 3. However, RL struggles to scale to the real-world applications due two major limitations: 1) heavily relies the manually defined reward function [4], 2) it requires
the online interactions with the environment [5]. In many application scenarios such as dialogue system (6) and roboties (7), it difficult manually design appropriate reward for constructing the practical reinforcement learning system. Moreover, for some -enett-sensitive applications like
clinical decision making | and autonomous driving [10 online trials and errors are prohibited
"Equal Contribution.
"Our implementation available http:///gghaabacom////oo//2////f
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Transition Samples D s())
Maximum Likelhood Extimation
World Model
Reward Estimator r(s, a; 0)
P s))
Expert Trajectories D (F))
Illustration of the modular structure in algorithmic framework, Offline ML-IRL. In Offline ML-IRL, it first estimates world model from the dataset transition samples, and then implements an ML based offline IRL algorithm on the estimated world model to recover the ground truth reward function from the collected expert trajectories.
Figure 1:
due to the safety concern. Due to these limitations in the practical applications, a new paradigm learning from demonstrations, which relies on historical datasets of demonstrations to model the agent for solving sequential Decision-making problems becomes increasingly popular. In such a paradigm, it is important to understand the demonstrator and imitate the demonstrator's 's behavior by only utilizing the collected demonstration dataset itself, without further interactions with either the demonstrator or the environment.
In this context, offline inverse reinforcement learning (offline IRL) has become promising candidate to enable learning from demonstrations [12 13 14 95] Different from the setting of standard 20 21] which recovers the reward function and imitates expert behavior the expense of extensive interactions with the environment, offline IRL. is designed to get rid of the requirement in online environment interactions by only leveraging finite dataset of demonstrations. While offline IRL holds great promises in practical applications, its study is still in an early stage, and many key challenges and open research questions remain to be addressed. For example, one central challenge in offline IRL arises from the so-called distribution shift [22 15 situation where the recovered reward function and recovered policy cannot generalize well to new unseen states and actions in the real environment. Moreover, in the training process of the offline IRL, any inaccuracies in the sequential compound, leading to poor performance of the estimated reward function / policy in the real- world environment. This is due to the fact that offline IRL is trained upon fixed datasets, which only provide limited coverage the dynamics model of the real environment. Although there progress in offlin IRL. [14 9 15), how to alleviate distribution shift in offline IRL is still rarely studied and not clearly understood. Wimnessing recent advances in closely related area, the offline reinforcement learning. which incorporates conservative policy training to avoid overestimation of values in unseen states induced by the distribution shift 23 24 25 26 27,, in this work aim propose effective offline IRL method to alleviate distribution shift and recover high-quality reward function from collected demonstration datasets. Due the space limitations, refer readers to Appendix B for more related work.
IRL [6 17
18 19,
that is, the
eecision-making process induced by distribution shift will
some recent
Our Contributions. To alleviate distribution shift and recover high-quality reward function from fixed demonstration datasets, we propose to incorporate conservatism into a model-based setting and consider offline IRL as maximum likelihood estimation (MLE) problem. Overall, the goal is to recover a reward that generates an optimal policy to maximize the likelihood over observed expert demonstrations. Towards this end, we propose two-stage procedure (see Fig. for an overview). In the first stage, we estimate the dynamics model (the world model) from collected transition samples; by leveraging uncertainty estimation techniques quantify the model uncertainty, we are able to construct a conservative Markov decision process (conservative MDP) where the state-action pairs with high model uncertainty and low data coverage receive a high penalty value to avoid risky exploration in the unfamiliar region. In the second stage, we propose an IRL algorithm recover the reward function, whose corresponding optimal policy under the conservative MDP constructed in the first stage maximizes the likelihood of observed expert demonstrations. To the best the first time that ML-baaed formulation, as well as the associated statistical and computational guarantees for reward recovery, has been developed for offline IRL.
of our knowledge,
To summarize, our main contributions are listed as follows:
We consider a formulation of offline IRL. based on MIE over observed transition samples and expert trajectories. In the proposed formulation, we respectively model the transition dynamics and

---

the reward function as the maximum likelihood estimators to generate all observed transition samples and all collected expert demonstrations. We provide statistical guarantee to ensure that the optimal reward function of the proposed formulation could be recovered as long as the collected dataset of transition samples has sufficient coverage on the expert-visited state-action space.
We develop computationally efficient algorithm to solve the proposed formulation of offline IRL. To avoid repeatedly solving the policy optimization problem under each reward estimate, we propose an algorithm which alternates between one reward update step and one conservative policy improvement step. Under nonlinear parameterization for the reward function, we provide the theoretical analysis to show that the proposed algorithm converges to an approximate stationary point in finite time. Moreover, when the reward linearly parameterized and there is sufficient coverage on the expert-visited state-action space to construct the estimated world model, we further show that the proposed algorithm approximately finds the optimal reward estimator of the MLE formulation.
We conduct extensive experiments by using robotic control tasks in MuloCo and collected datasts in D4RL benchmark. We show that the proposed algorithm outperforms the state-of-the-an offline IRL such as [14 [5] and imitation learning methods such as (28). especially when the number of observed expert demonstrations is limited. Moreover, we transfer the recovered reward across different datasets to show that the proposed method can recover high-quality reward function from the expert demonstrations.
2 Preliminaries and problem formulation
Markov decision process (MDP) is defined by the tuple (S.AA which consists of the x state space S, the action space A. the transition dynamics P: x [0, 1]. the initial state distribution 77(), the reward function r : -S x A - IR and the discounted factor E (0. 1). Under a transition dynamics model P and a policy æ, we able further define the state-action visitation measure as (s, == (1 ( E 7 pT (st s|5o ~ 11) for any state-action pair (s, a).
Maximum entropy inverse reinforcement learning (MaxEnt-IRL) is specific IRL formulation which aims to recover the ground-truth reward function and imitate the expert's 's policy from expert's Let 7 {st, a.) ) denotes the expert trajectory sampled from the
demonstrations 18 29
20
expert policy x E. let T denote the trajectory generated by the RL agent with policy æ. Then the
MaxEnt-IRL. is formulated
max min EFENTE Σ r r ai)
E ETANT Σ r (rr((())))) H(m}
(I)
where H(T)= == E Ei=o - log π [ denotes the causal entropy of the policy x. The MaxEnt-IRI. formulation aims to recover the ground-truth reward function which assigns high rewards to the expert policy while assigning low rewards to any other policies. Although MaxEnt-IRL. has been well-studied theoretically (29 30 21 31 and has been applied to several practical applications
it needs to repeatedly solve policy optimization problems under each reward function and online interactions with the environment is inevitable. Such repeated policy optimization subroutine requires extensive online trials and errors in the environment and thus makes MaxEnt-IRA quite limited in practical applications.
33 33
Problem formulation Let us now consider an MI formulation of offline IRI.. Given the transition dataset D == {(x, a, 8))), we train an estimated world model P(s'|s, a) for any s', 8 E S and a E A (to be discussed in detail in Sec. 3 The constructed world model P will be utilized as an estimate of the ground-truth dynamics model P. Based on the estimated world model P, we propose a bodel-based offline approach for IRL from the ML perspective, given below:
max ((0)= := EE,E
Σ
log To(ayls))
(2a)
s.t. = arg max E (MA
((() ar 0) Usz, az) +H((((ss)))
((s
(2b)
1=0
where [((x(-1)) ==
EaEa - क (als) log x (als) denotes the entropy of the distribution x (-1s); U..-) is penalty function to quantify the uncertainty of the estimated world model P(-Is, under any
3

---

state-action pair (s, In practice, the penalty function is constructed based on uncertainty heuristics over an ensemble of estimated dynamics models 35
25 24 36]. A comprehensive study of the choices of the penalty function can be found in 37 Next, let us make few remarks about the above formulation, which we name Offline MI.-IRI.
First, the problem takes a bi-level optimization form, where the lower-level problem 2b assumes that the parameterized reward function -(::;0) 0) is fixed, and it deseribes the optimal policy To as a unique solution to solve the conservative MDP; On the other hand, the upper-level problem (2a) optimizes the reward function -(-.;;) 8) so that its corresponding optimal policy we maximizes the log- oogliikeihoodd L(Ø) over observed expert trajectories.
Second, formulating the objective as likelihood function is reasonable since it searches for an optimal reward function to explain the observed expert behavior within limited knowledge about the world (the world model P is constructed based on a finite and diverse dataset D == {(s, a, s'))).
Third, the lower-level problem 2b
corresponds to model-based offline RL problem under the current reward estimate. The policy obtained is conservative in that state-action pairs that are not well covered by the dataset are penalized with measure of uncertainty in the estimated world model. The penalty function (8, a) is used to quantify the model uncertainty and regularize the reward estimator. Therefore, the optimal policy No under the conservative MDP will not take risky exploration those uncertain region the state-action space where the transition dataset does not have sufficient coverage, and the constructed world model has high prediction uncertainty.
3 The world model and statistical guarantee
In this section, we construct the world model f from the transition dataset == {(s, a, 8'}} and solve a certain approximated version of the formulation (2). Then we further show high-quality reward estimator can be obtained with statistical guarantee.
Before proceeding, let emphasize that major challenge in solving (2) comes from the dynamics model mismatch between (2a) and (2b), which arises because the expert trajectory - is generated from the ground-truth transition dynamics P, while the agent samples its trajectory 7 through interacting with the estimated world model P. To better understand the above challenge, next will explicitly analyze the likelihood objective in (2) and understand the mismatch error. Towards this end, let us introduce below the notions of the soft Q-function and the soft value function of the conservative MDP in (2) (defined for any reward parameter & and the optimal policy π&):
Vo(5) == E E
Σ
> (r(s,, a;;0) + UU(, at) ) H(TT 30 = =]]
(3a)
1=
Q0(s, a) = r a; + U(s a) 7E (( [V)(s')
(3b)
According to B8 39 40). the optimal policy Toe and the optimal soft value function Vo have the following closed-for forsed-frmmm-o expressions under any state-action pair (s, a):
Te(a = exp Qa(s,
Ve(s) = log
Σ exp Qo(s aEA
Eaea exp @)(
Under the ground-truth dynamics model P and the initial distribution n7.), we further define the visitation measure ds, a) under the expert policy E as below:
d
= (1 (ss,a)=(1--)))
Σ
Etppt pm (st s/80 17)
By plugging the closed-form solution of the optimal policy no into the objective function (2a).
can decompose the dynamics-mode mismatch error from the likelihood function L(0) in 2
Lemma 1. Under any reward parameter 9, the objective L()) in 2a) can be decomposed as below:
L(Ø) = [())) + 7 --7
((era))d
Vo(s') Vo('))(P(s'lss,)) P(s'(s.a))
(6)
FES

---

where L(0) is surrogate objective defined as:
I(()= == (n R
x (r(s), a1;0) Use,
at)]] USS,,a))) E Bsom)()
Vo(so)
(7)
The detailed proof is included in Appendix D In Lemma 1 we have shown that the likelihood function in decomposes into two parts: surrogate objective [(0) and an error term dependent on the dynamics model mismatch between P and P. As remark, in the surrogate objective 7.()), separate the two dynamics models (P and P) into two relatively independent components. Therefore, optimizing the surrogate objective is computationally tractable and we will propose an efficient algorithm to recover the reward parameter from the next section.
To further elaborate the connection between the likelihood objective L(0) and the surrogate objective ((). we first introduce the following assumption: Assumption 1. For any reward parameter 0 and state-action pair (s, following conditions hold: [r(s, oz 0)) S Cr, [U(8,a) S Cv.
(8)
where Cp and Cv are positive constants.
As a remark, the assumption of the bounded reward is common in the literature of inverse reinforcement learning and imitation learning [4] 22 42 43 Moreover, the assumption of the bounded penalty function holds true for common choices of the uncertainty heuristics (37), such as the max aleatoric penalty and the ensemble variance penalty. Then we can show the following results. Lemma 2. Suppose Assumption 2 holds, then we obtain (where C2 is a positive constant):
(0)) - [(0)] S yCe
(((so))
[IIP())s.)) (1)p))
((-Is,a)]l1)
Please see Appendix E for the detailed proof. The above lemma suggests that the gap between the likelihood function and its surrogate version is bounded by the model mismatch error ((a.))-d((.)) [[|s(sss a) P(-|s, a)]l1]- The fact that the objective approximation error ( [+) depends the model mismatch error evaluated in the expert- state-action distribution d-
crucial to the construction of the world model P. Based on Lemma [ we understand that full data coverage on the joint state-action space S x A is not necessary. Instead, as long as the collected transition dataset D = {(s, a, 8')} provides sufficient coverage on the expert-visited state-action space Ω == ((s, a) (s, a) > 0). then the surrogate objective [.()) will be an accurate approximation the likelihood objective L(0).
Intuitively, considering the goal to recover reward function to model expert behaviors which only lie in a quite limited region of the whole state-action space, data collection with full coverage can be redundant. This result very useful in practice, since it serves to greatly reduce the efforts on data collection for constructing the world model. Morcover, it also matches recent theoretical understanding on offline reinforcement learning 44 45 46) and offline imitation learning [22], which show that is enough to learn good policy from offline data with partial coverage.
To analyze the sample complexity in the construction of the world model P. quantitatively analyze the approximation error between T.(0) and T.(0). In discrete MDPs, the cardinalities of both state space and action space are finite (SS < oc and |A| < co). Therefore, based on collected transition dataset D = {(8, s')), }. we will use the empirical estimate to construct the world model P. Also recall that Ω = {(s, a) a) (s. a) > 0} denotes the set of expert-visited atate-action pairs. Define S : {s) Eaea (s, a) 0} C S as the set of expert-visited states. Using these definitions, have the following result. The detailed proof is in Appendix H
Proposition 1. For any E 20. suppose there are more than N data points on each state-action puir (8, a) E St and the total number of the collected transition samples satisfies:
²² -112
[2
#transition samples 2 |02
2² In
where is a constant dependent on 5. With probability greater than the following results hold:
7C,
E ((s.a)=d (1)((ss)))))) P(-ss, 1) Se, (L(0) I((0 < 1-7
(10)
5

---

The above result estimates the total number of samples needed to construct the estimated world model P. so that the surrogate objective (()) can accurately approximate L(0). A direct implication is that, the reward parameter obtained by solving [-)) also guarantees strong performance. To be more specific, define the optimal reward parameters associated with L(-) and L(-) as below, respectively:
6* E arg max I(0); i E arg max Í(0)).
The next result characterizes the performance gap between the reward parameters and 0*. The detailed proof included in Appendix |
Theorem 1. For any 8 E (0, 1 suppose there are more than data points on each state-action pair (8, a) E Ω and the number of transition dulaset DD satisfies:
#transition samples 122 47² C. c² |22| |SE
[2
In
(1 (1-7)2E2
8
where è is a constant dependent on 8. With probability greater than 1 6, the following result holds:
I(O0) I(0)) Se. e.
(11)
4 Algorithm design
In the following sections, we will design a computationally efficient algorithm optimize Σ(-), and obtain its corresponding optimal reward parameter Ó.
From the definition (7), is clear that L(-) depends on the optimal soft value function V-(+) in (3a), which in turn depends on the optimal policy ли as defined in (26). Therefore, the surrogate objective maximization problem can be formulated as a bi-level optimization problem, expressed below, where the upper-level problem optimizes L(-) search for good reward estimate 0, while the lower-level problem solves the optimal policy my a conservative MDP under the current reward estimate:
max I(0),
S.C.
Taa = max E (
(r(st, U(I, [(r (((()))))))()()()))))))))))))))) H(T((ss)))) (12)
In order to avoid the computational burden from repeatedly solving the optimal policy To under cach reward estimate 0, we aim design algorithm which alternates between policy optimization step and a reward update step. That is, at each iteration k; based on the current policy estimate Ty and the reward parameter 0z. two steps will be performed consecutively: (/) the algorithm generates updated policy Th+1 through performing conservative policy improvement step under the estimated world model P. and (2) it obtains an updated reward parameter 0x+1 through taking reward update step. Next, we describe the proposed algorithm in detail.
Policy Improvement Step. Under the reward parameter Os. we consider generating new policy TK+1 towards approaching the optimal policy Te, as detined in (12). Similar to the definitions of Ve and Qo in (3a) (bb). under the current policy estimate Th. the reward estimate r(. r 0x) and the estimated world model P. define the corresponding soft value function V-(() and the soft Q-function as Q.)-.)) Please see (28a)
28b) in Appendix for the precise definitions.
In order to perform policy improvement step, we first approximate the soft Q-function by using an estimate Q (8, a), which satisfies the following: (where aapp > 0 is an approximation error)
1 Qulloo == max 1Q+(s, a) (s, aapp-
(13)
sES,
With the approximator Q,, updated policy Th41 can be generated by soft policy iteration:
T x exp as) E A.
(14)
As remark, in practice, one can follow the popular reinforcement learning algorithms such soft Q-learning (38) and soft Actor-Critic (SAC) (39 to obtain accurately approximated soft Q-function
with low approximation error Eapp (as outlined in [3 so to achieve stable updates for the soft

---

Algorithm A Model-based Approach for Offfine Maximum Likelihood IRL (Offline ML-IRL)
Input: Initialize reward parameter 8g and policy No. Set the reward parameter's is stepsize as a. Train the world model P on the transition dataset D. Specify the penalty function (.,-) based on P. for k 0, I....,KKK K - do
Policy Evaluation: Approximate the soft Q-function Qu(-,)) by Qu(s-) Policy Improvement: T (-|s) a exp (Qx(s,))) ()),,,)). Vs E
Data Sampling I: Sample an expert trajectory To := {st, av 1120
Data Sampling Sample Tद == {se. ae3 from mx+1 and P
Estimating Gradient: ga h(Ou: Tx) h(Ou: T where h(0:7) = Eizo I (oVorss,
Reward Parameter Update: 02+1 == O2 + agg.
end for
policy iteration in 14 In the literature of the model-based offline reinforcement learning, the state-of-the-art methods 25
37 47 also build their framework upon the implementation of SAC.
Reward Optimization Step. At each iteration k; given the current reward parameter 0g and the updated policy M we can update the reward parameter to 0k+1. First, let us compute the gradient of the surrogate objective ((²)).. Please see Appendix for the detailed proof.
Lemma 3. The gradient of the surrogate objective (() defined in 7 can be expressed as:
Vī(0) E, E,
Vor(ss B (n
Voorrss (15)
1=0
In practice, do not have access the optimal policy my. This is due the fact that repeatedly solving the underlying offline policy optimization problem under each reward parameter computationally intractable. Therefore, at each iteration construct an estimator of the exact gradient based on the current policy estimate T
To be more specific, take two approximation steps develop stochastic gradient estimator of vī.( 1) choose observed expert trajectory B: 2) sample trajectory Th from the current policy estimate Ts+1 in the estimated world model P. Following these two approximation steps, the stochastic estimator Gs which approximates the exact gradient (L(O)) in (15) is defined as follows:
Ok = h(00:: T - ((00: (():TT))
(16)
where h(B: = Ee 'Vrr(s (st, a,; 0) denotes the cumulative reward gradient under a trajectory. Then we can update reward parameter according to the following update rule:
0 02 + agk
(17)
In Alg. summarize the proposed algorithm (named the Offline ML-IRL).
Convergence analysis
In this section, we present theoretical analysis to show the finite-time convergence of Alg
Before starting the analysis, let us point out the key challenges in analyzing the Alg. Note that the algorithm relies on the updated policy Ti++1 to approximate the optimal policy Te, at each iteration
This coarse approximation can potentially lead to the distribution mismatch between the gradient
estimator 9k in (16) and the exact gradient vī((k) in (15). To maintain the stability of the proposed algorithm, we can use a relatively small stepsize a to ensure the policy estimates are updated in a
faster time-scale compared with the reward parameter 0. This allows the policy estimates ( 1 ):2 to closely track the optimal solutions {To, Jazo in the long run.
To proceed, 1et us introduce few assumptions.
7

---

Assumption 2 (Ergodic Dynamics), Given any policy the Markov chain under the estimated world model P is irreducible and aperiodic. There exist constants K. > 0 and p (0,1) to ensure:
[|P(st E -150 =, 11 kpp', 4120
max
SES
where || 1lrv denotes the total variation (TV) norm; P the stationary distribution of visited states under the policy π and the world model P.
The assumption about the ergodic dynamics is common in the literature of reinforcement learning
48
49
21
31 50), which ensures the Markov chain mixes at geometric rate.
Assumption 3 (Lipsehitz Reward). Under any reward parameter 0, the following conditions hold for any and E A::
[|Vorss, a; S L., [|Vor(s,a;0)) oor (2) Ly
02ll.
(18)
where L, and Lg positive constants.
According Assumption 3
the parameterized reward has bounded gradient and is Lipschitz smooth.
This assumption is common for min-max / bi-level optimization problems [4] 51
52 31 50).
Based Assumptions 2 show that certain Lipschitz properties of the optimal soft Q-function
and the surrogate objective hold. Please see the detailed proof in Appendix G
Lemma 4. Suppose Assumptions 2 hold. Under any reward parameter 01 and 02 the optimal soft O-function and the surrogate objective satisfy the following Lipschitz properties:
1Qe, (s, a) Qe, a)) ] & Lgll 01 - 0211. Vs E S
(19a)
(i)) - ((2))) S Lcl101 02l
(19b)
where bg and Le are positive constants.
Our main convergence result is summarized in the following theorem, which characterizes the convergence speed of the estimates (π=+1)*20 and (0x k220 The detailed proof is in Appendix
Theorem 2 (Convergence Analysis). Suppose Assumptions 2 hold. Let K denote the total number
of tterations to be run in Alg. 4 Setting the stepsize as
K where og 0, we obtain the
following convergence results:
log M4+1 - llo 1l0] = -OKK (Espp)
(20a)
K
E [V OKK Ocapp)
(20b)
K
where have defined log Th+ log Te Eapp is the approximation error defined in
1o max log (als) - log Te, (als) and
In Theorem 2 we demonstrated that Alg identifies the approximate stationary solution of the surrogate problem (12) in finite time. Next, show that when the reward is parameterized linearly, an improved result can be obtained, where the stationary solutions of problem (12) can be connected
the optimal solutions of the offline-IRL, problem (12. See Appendix K for detailed proof.
Theorem 3 (Optimality Guarantee). Assume that the reward function is linearly parameterized, i.e., r(s, a; 0) := ő(s, a) T where o(s, a) is the feature vector of the state-action pair (s, a). Then any stationary point of the surrogate problem (12) is a global optimum. Furthermore, for any e E (0, 4,*C2) suppose there are more than N data points on each state-action pair (s, a) and the number of transition dataset D satisfies:
#transition samples 19|-N2 4-2.C. .2. [2] 1
12
In
where is constant dependent on 8. With probability greater than 1 5, any stationary point ē of the surrogate objective L(.) is an apsilon-optimal solution to the maximum likelihood problem 2
L(0") L(Ø) <
(21)
where 9 is defined the optimal reward parameter of the log-likelihood objective L(-).
8

---

As remark, the epsilon-optimal solution on the MLE problem implies:
1
To- (al TT(als)
L( - L(0) -
1-y "_.Eaa Bandd()) log
S
When the expert trajectories are consistent with the optimal policy under a ground truth reward parameter 8", we have T zg- Due to this property, we can obtain that
L(07)- L(0) = E ().an log
1
(als)) = 1-7 Re [Dx (1 (+1))llmm())))) s To(a|s)
1
also provides a formal guarantee that the recovered policy zg is e-close to the expert policy BB measured by the KL. divergence.
Hence, Theorem 3
We remark that even under the linear parameterization assumption, showing the optimality of the stationary solution for the surrogate problem (12) is still non-trivial, since the problem still not a concave problem with respect to 0. In our analysis, we translate this problem to a certain saddle point problem. Under linear reward parameterization, we show that any stationary solution 0 of the surrogate problem (12) together with the corresponding optimal policy T& consist of a saddle point to the saddle point problem. By further leveraging the property of the saddle point, we show the optimality of the stationary solution for the surrogate problem. Finally, by utilizing the statistical guarantee in Theorem we obtain the performance guarantee for any stationary point 0 in (21).
Numerical results
In this section, present numerical results for the proposed algorithm. More specifically, we intend to address the following questions: 1) How does the proposed algorithm compare with other state-of-the-ard methods? 2) Whether offline ML-IRL can recover high-quality reward estimator of the expert, in the sense that the reward can be used across different datasets or environment?
We compare the proposed method with several benchmarks. Two classes of the existing algorithms are considered as baselines: 1) state-of-the-art offline IRL algorithms which are designed to recover the ground-trun truth reward and expert policy from demonstrations, including model-based approach CLARE [15] and a model-free approach IQ-Learn [14] 2) imitation learning algorithms which only learn policy to mimic the expert beahviors, including behavior cloning (BC) and ValueDICE (28
We test the performance of the proposed Offline MI.IRL on a diverse collection of robotics training tasks in MuloCo simulator [53], as well as datasets in D4RL benchmark (54). which include three environments (halfcheetah, hopper and walker2d) and three dataset types (medium-replay medium, and medium-expert In each experiment set, both environment interactions and the ground-tre reward are not accessible. Morcover, we train the algorithm until convergence and record the average reward of the episodes over 6 random seeds.
In Offline ML-IRL, we estimate the dynamics model by neural networks which model the location of the next state by Gaussian distributions. Here, we independently train an ensemble of N estimated world model { 0..pp (5t+1 [tt, ax) (Hi, (st, ax), (s, (e,,))))) a.)) 1 via likelihood maximization over transition samples. Then we can quantify the model uncertainty and construct the penalty function. For example, in (25. the aleatoric uncertainty is considered and the penalty function constructed as Uss,a)
i......l....... |r (s, al)]... For the offline RL subrouinte in 13 14). we follow the
implementation of MOPO 25 We follow the same setup in (37 to select the key hyperparameters for MOPO. We parameterize the reward function by a three-layer neural network. The reward parameter and the policy are updated alternatingly, where the former is updated accodring to (I7). while the latter is optimized according to MOPO. More experiment details are in Appendix A
In Figure 2 and Table I we show the performance comparison between the proposed algorithm and benchmarks. The results of IQ-Learn is not presented since it suffers from unstable performance. To obtain better performance in BC and ValueDICE, we only use the expert demonstrations to train their agents. From the results, it is clear that the proposed Offline ML-IRL outperforms the benchmark algorithms by a large margin most cases. In the experiment sets where Offline ML-IRL does not obtain the best performance, we notice the performance gap is small compared with the leading benchmark algorithm. Moreover, among the three dataset types (medium-replay. medium and medium--epper), medium-expen dataset has the most complete coverage on the expert-visited

---

Hopper (5,000 Expert Deroonstrations
Walker2d 15, 900 Expert Demonstrations)
HallCheetah 15, 0,000 Expert Demonstrations)
0890
37:07-
Nvw Avesuge Reavers
20000
3080
1000
1000
-300
Teper
Offinne Episodes
Offine Episodes
Offine Episades
Figure 2: The performance of Offline ML-IRL given 5,000 expert demonstrations.
Dalasel lype
Envinonment
Ofline ML-IRL
BC
DICE
CLARE
Expent Performance
mediuum
hopper
2453.25
711.30
2801.
330.8
3073.16
538.67
3015.37
474.38
3112.08 + 21.63
medium
halfcheetah
7640.73
195.00
4471.72 72 2835.55
1125. 17 959.45
841.46
344.06
12174.61
91.45
medium
walker2d
3989.20
487.82
2328.75
906.96
3191.47
1887.90
237.49 +: 160-62
5383.96
1: 52.15
medium-replay
hogper
3046.36
429.25
1901.11-10
330.88
3073.16 1 538.67
2888.04 1: 844-48
3512.09 1: 21.65
medium-replay
halfcheetah
9236.84
309.10
4471.72 + 2835.55
1125. 17 959.45
437.18 + 182.99
12174.61
91.45
medium-replay
walker2d
3995.32
487.82
2328.73 75 906.96
3191.47 4: 1887.90
291.71
5383.98
52.15
medium-esp
hopper
3347.11 238.18
2801
330.88
3073. 16 538.67
3350.47
245.78
3512.01
21.65
medium-cxp
halfcheetah
11231-40 + 585.21
4471.72 + 2835.55
1125.
959.45
622.79 1 50.46
12171 + 91.45
medium-cxp
walker2d
4201.-10 = 637.99
2328. 75 + 906.96
3191.47 = 1887.90
9G9.50 = 470.64
5383 98 + 02.15
Table 1: MuJoCo Results. The performance versus different datasets and 000 expert demonstrations. The
bolded numbers are the best ones for each data set, among Offline ML-IRL, BC, ValueDICE, and CLARE.
state-action space. As result, Offline ML-IRL can nearly match the expert performance when the world model is trained from the medium-expert transition dataset. This numerical result matches our theoretical understanding in Theorem [ As a remark, in most cases, we notice that Offline ML-IRL outperforms CLARE by large margin which is also a model- offline IRL algorithm. As we discussed in the related work (Appendix [B). this is due to the fact that when the expert demonstrations are limited and diverse transition samples take large portion of all collected data, CLARE learns policy to match the joint visitation measure of all collected data, thus fails to imitate expert behaviors.
In Appendix A we further present experiment showcasing the quality of the reward recovered by the Offline MI-IRL algorithm. Towards this end, we use the recovered reward from the medium-expert dataset estimate the reward value for all transition samples (s, a, s') in the medium-replay dataset. With the recovered reward function r(,;;;)) and given those transition samples with estimated reward labels {(s, a, r(s, o; 0), s')}, we run MOPO to solve Offline RL tasks given these transition samples with estiamted rewards. In Fig. we infer that the Offline ML-IRL can recover high-qualin reward functions, since the recovered reward can label transition samples for solving offline RL tasks. Comparing with the numerical results of directly doing offline-IRL on the respective datasets in Table [
we show that solving Offline RL by using transferred reward function and unlabelled transition datasts can achieve similar performance in Fig.
7 Conclusion
In this paper, we model the offline Inverse Reinforcement Learning (IRL) problem from a maximum likelihood estimation perspective. Through constructing a generative world model to estimate the environment dynamies, we solve the offline IRL problem through a model-based approach. We develop a computationally-effiiiee algorithm that effectively recovers the underlying reward function and its associated optimal policy. We have also established statistical and computational guarantees for the performance of the recovered reward estimator. Through extensive experiments, we demonstrate that our algorithm outperforms existing benchmarks for offline IRL and Imitation Learning, especially on high- -igh-dimnssional robotics control tasks. One limitation of our method is that we focus solely on aligning with expert demonstrations during the reward learning process. In an ideal scenario, reward learning should incorporate diverse metrics and data sources, such as expert demonstrations and preferences gathered through human feedback. One direction for future work is to broaden our algorithm framework and theoretical analysis for a wider scope in reward learning.
10

---

Acknowledgments
M. Hong and Zeng are supported by NSF grant CIF-1910385.
References
[1] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, "Playing atari with deep reinforcement learning," arXiv preprint arXiv:1312.500,, 2013.
[2] D. Silver, T. Hubert, J. Schrittwieser, I. Antonogiou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel al., "A general reinforcement learning algorithm that masters chess, shogi, and go through self-play," Science, vol. 362, no. 6419, pp. 1140-1 144, 2018.
[3] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, Т. Ewalds, P. Georgiev et al., "Grandmaster level in starcraft ii using multi-agent reinforcement learning," Nature, vol. 575, no. 7782, pp. 350-354, 2019.
[4] D. Silver, Singh, D. Precup, and S. Sutton, "Reward is enough," Artificial Intelligence, vol.
299, p. 103535, 2021.
[5] Levine, "Understanding the world through action," in Conference on Robot Learning. PMLR, 2022, pp. 1752-1757.
[6] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray al., "Training language models to follow instructions with human feedback," arXiv preprint arXiv:2203.022 22212223.,,,, 2022.
[7] Kober, J. A. Bagnell, and Peters, "Reinforcement learning in robotics: A survey," The International Journal of Robotics Research, vol. 32, no. 11, pp. 1238-1274, 2013.
[8] S. Liu, K. C. See, K. Y. Ngiam, L. A. Celi, X. Sun, M. Feng et al., "Reinforcement learning for clinical decision support in critical care: comprehensive review," Journal of medical Internet research, vol. 22, no. 7, p. e 14477 2020.
[9] A. J. Chan and M. van der Schaar, "Scalable bayesian inverse reinforcement learning," arXiv preprint 2XXi:22122.06483,, 2021.
[10] B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Pérez, "Deep reinforcement learning for autonomous driving: A survey." IEEE Transactions on Intelligent Transportation Systems, 2021.
[11] R. Wei, A. Garcia, A. McDonald, G. Markkula, J. Engström, Supcene, and M. O'Kelly, "World model learning from demonstrations with active inference: Application to driving behavior," in Forthcoming, 3rd International Workshop on Active Inference, Grenoble, France, 2022.
[12] E. Klein, M. Geist, and O. Pietquin, "Batch, off-policy and model-free apprenticeship learning." in European Workshop on Reinforcement Learning. Springer, 2011 1, pp. 285-296.
[13] M. Herman, T. Gindele, Wagner, Schmitt, and W. Burgard, "Inverse reinforcement learning with simultaneous estimation of rewards and dynamics," in Artificial Intelligence and Statistics. PMLR, 2016, pp. 102-110.
[14] D. Garg. S. Chakraborty, C. Cundy, J. Song, and S. Ermon, "Iq-learn: Inverse soft-q learning for imitation," Advances in Neural Information Processing Systems, vol. 34, pp. 4028-4039, 2021.
[15] S. Yue, G. Wang. W. Shao, 7.. Zhang, Conservative model-based reward learning for offline inverse reinforcement learning," in International Conference on Learning Representations, 2023. [Online]. Available: http://opennevie..ee//omuuu www.://eeffoummmddSsSaaaaaaoaao
Lin, Ren, and J. Zhang, "CLARE:
[16] A. Y. Ng, S. Russell et al., "Algorithms for inverse reinforcement learning." in Icml, vol. 1, 2000, p. 2.
[17] P. Abbeel and A. Y. Ng, "Apprenticeship learning via inverse reinforeement learning." in Proceedings of the twenty-first international conference on Machine learning, 2004, p. 1.
[18) B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey al., "Maximum entropy inverse reinforcement learning." in AAAI, vol. 8. Chicago, IL, USA, 2008, pp. 1433-1438.
[19] M. Wulfmeier, P. Ondruska, and Posner, "Maximum entropy deep inverse reinforcement learning," arXiv preprint (107.::5507.8888800 2015.

---

[20] J. Fu, K. Luo, and S. Levine, "Learning robust rewards with adversarial inverse reinforcement learning," " arXiv preprint arXiv:1710.1122 2017.
[21] S. Zeng, C. Li, A. Garcia, and M. Hong, "Maximum likelihood inverse reinforcement learning with finite-time guarantees." Advances in Neural Information Processing Systems, 2022.
[22] J. Chang, M. Uehara, D. Sreenivas, R. Kidambi, and W. Sun, "Mitigating covariate shift in imitation learning via offline data with partial coverage," Advances in Neural Information Processing Systems, vol. 34, pp. 965-979, 2021.
[23] A. Kumar, A. Zhou, G. Tucker, and S. Levine, "Conservative q-learning for offline reinforcement learning." Advances in Neural Information Processing Systems, vol. 33, pp. 1179-1191, 2020.
[24] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims, "Morel: Model-based offline reinforcement learning," Advances in neural information processing systems, vol. 33, pp. 21 810-21 823, 2020.
[25] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma, "Mopo: Model-baset offline policy optimization." Advances in Neural Information Processing Systems, vol. 33, pp. 14 149-14 142, 2020.
[26] T. Yu, A. Kumar, R. Rafailov, A. Rajeswaran, S. Levine, and C. Finn, "Combo: Conservative offline model-based policy optimization." Advances in neural information processing systems, vol. 34, pp. 28 954-28 967, 2021.
[27] G. Tennenholtz and S. Mannor, "Uncertainty estimation using riemannian model dynamics for offline reinforcement learning," in Advances in Neural Information Processing Systems, A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, Eds., 2022. [Online]. Available: https:/openneviee.etfffuu Sartummi-d-on lid=pGL.
(28) I. Kostrikov, O. Nachum, and J. Tompson, "Imitation learning via off-policy distribution matching," arXiv preprint arXiv: /9/::05022..2 2019.
[29] B. D. Ziebart, J. A. Bagnell, and A. K. Dey, "The principle of maximum causal entropy for estimating interacting processes." IEEE Transactions Information Theory, vol. 59, no. 4, pp. 1966-1980, 2013.
[30] Z. Zhou, M. Bloem, and N. Bambos, "Infinite time horizon maximum causal entropy inverse reinforcement learning," JEEE Transactions on Automatic Control, vol. 63, no. 9, pp. 2787-2802, 2017.
[31] S. Zeng, M. Hong, and A. Garcia, "Structural estimation of markov decision processes in high-dimensional state space with finite-time guarantees," arXiv preprint 10.00222...01282. 2022.
[32] J. Fu, A. Korattikara, S. Levine, and S. Guadarrama, "From language to goals: Inverse reinforcement learning for vision-based instruction following," in International Conference on Learning Representations, 2018
[33] Z. Wu, L. Sun, W. Zhan, C. Yang, and M. Tomizuka, "Efficient sampling- based maximum entropy inverse reinforcement learning with application autonomous driving," " IEEE Robotics and Automation Letters, vol. 5, no. 4, pp.
2020.
[34] L. Zhou and K. Small, "Inverse reinforcement learning with natural language goals." in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 12, 2021, pp. 11 116-11 11 124.
[35] B. Lakshminarayanan, A. Pritzel, and C. Blundell, "Simple and scalable predictive uncertainty estimation using deep ensembles," Advances neural information processing systems, vol. 30, 2017.
[36] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn, "Offline reinforcement learning from images with latent space models," in Learning for Dynamics and Control. PMLR, 2021, pp. 1 1154-1163
[37] C. Lu, P. Ball, Parker-Holder, M. Osborne, and S. J. Roberts, "Revisiting design choices in offline model based reinforcement learning." in International Conference on Learning Representations, 2021.
[38] Haarnoja, H. Tang, P. Abbeel, and S. Levine, "Reinforcement learnng with deep energy-based policies," in International conference machine learning. PMLR, 2017, pp. 1352-1361.
12

---

[39] Т. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, "Soft actor-critic: Off- Off-policy maximum entropy deep reinforcement learning with stochastic actor," in International conference on machine learning. PMLR, 2018, pp. 1861-1870.
[40] S. Cen, C. Cheng, Y. Chen, Y. Wei, and Y. Chi. "Fast global convergence of natural policy gradient methods with entropy regularization," Operations Research, 2021.
[41] M. Chen, Y. Wang, T. Liu, Z. Yang, X. Li, Z. Wang, and T. Zhao, "On computation and generalization of generative adversarial imitation learning," International Conference on Learning Representations, 2019.
[42] L. Shani, T. Zahavy, and S. Mannor, "Online apprenticeship learning." in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 8, 2022, pp. 8240-8248.
[43] B. Zhu, J. Jiao, and M. I. Jordan, "Principled reinforcement learning with human feedback from pairwise or k-wise comparisons," arXiv preprint arXiv:2301. / 2023.
[44] Y. Liu, A. Swaminathan, A. Agarwal, and E. Brunskill, "Provably good batch off-policy reinforcement learning without great exploration," Advances in neural information processing systems, vol. 33, pp. 1264-1274, 2020.
[45] M. Uehara and W. Sun, "Pessimistic model-based offline reinforcement learning under partial coverage," " in International Conference on Learning Representations, 2021.
[46] A. Ozdaglar, S. Pattathil, J. Zhang, and K. Zhang, "Revisiting the linear-programming framework for offline rl with general function approximation." arXiv preprint arXiv:2212. 138661 2022.
[47] M. Rigter, B. Lacerda, and N. Hawes, "Rambo-rl: Robust adversarial model-based offline reinforcement learning," arXiv preprint arXiv:2204. 0222044.2581, /2581, 2022.
[48] Bhandari, D. Russo, and R. Singal, "A finite time analysis of temporal difference learning with linear function approximation," in Conference on learning theory. PMLR, 2018, pp.
1691-1692.
[49] Y. F. Wu, W. Zhang, P. Xu, and Q. Gu, "A finite-time analysis of two time-scale actor-critic methods," " Advances in Neural Information Processing Systems, vol. 33, pp- 17 17617-17 628, 2020.
[50] M. Hong, H.-T. Wai, Z. Wang, and Z. Yang, "A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critie," SIAM Journal on Optimization, vol. 33, no. 1, pp. 147-180, 2023.
[51] C. Jin, P. Netrapalli, and M. Jordan, "What is local optimality in minimax optimization?" in International conference machine learning. PMLR, 2020, pp. 4880-4889.
[52] Z. Guan, T. Xu, and Y. Liang, "When will generative adversarial imitation learning algorithms attain global convergence," in International Conference on Artificial Intelligence and Statistics. PMIR,, 2021, pp. 1117-1125.
[53] E. Todorov. T. Erez, and Y. Tassa, "Mujoco; A physics engine for model-based control," in 2012 IEEE/RSJ international conference on intelligent robots and systems. IEEE, 2012, pp. 5026-5033.
[54] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine, "D4rl: Datasets for deep data-driver reinforcement learning." " arXiv preprint urXiv:2004.072/99 2020.
[55] H. Cao, S. Cohen, and L. Szpruch, "Identifiability in inverse reinforcement learning." Advances in Neural Information Processing Systems, vol. 34, pp. 12362-12373 2021.
[56] Rolland, L. Viano, N. Schürhoff, Nikolov, and V. Cevher, "Identifiability and generalizability from multiple experts in inverse reinforcement learning," arXiv preprint arXiv:2209. 10974, 2022.
[57] C. Finn, S. Levine, and P. Abbeel, "Guided cost learning: Deep inverse optimal control via policy optimization," in International conference on machine learning. PMLR, 2016, pp. 49-58.
[58] S. Liu and M. Zhu, "Distributed inverse constrained reinforcement learning for multi-agent systems," Advances in Neural Information Processing Systems, vol. 35, pp. 444-33 456, 2022.
13

---

[59] E. Klein, M. Geist, B. Piot, and O. Pietquin. "Inverse reinforcement learning through structured classification," Advances neural information processing systems, vol. 25, 2012.
[60] D. Lee, S. Srinivasan, and F. Dosei,Velez, "Truly batch apprenticeship learning with deep successor features," arXiv preprint arXiv: /00::/00030007., 2019.
[61] K. Zolna, A. Novikov, K. Konyushkova, C. Gulcehre, Wang. Y. Aytar, M. Denil, N. de Freitas, and S. Reed, "Offline learning from demonstrations and unlabeled experience," arXiv preprint arXiv:2011.13888 2020.
[62] M. Abdulhai, N. Jaques, and S. Levine, "Basis for intentions: Efficient inverse reinforcement learning using past experience," arXiv preprint (oXiv:2208.049)) 2022.
[63] T. Xu, Z. Li, and Y. Yu, "Error bounds of imitating policies and environments," Advances in Neural Information Processing Systems, vol. 33, pp. 15737-15 749, 2020.
[64] D. Jarrett, I. Bica, and M. van der Schaar, "Strictly batch imitation learning by energy-based distribution matching," Advances in Neural Information Processing Systems, vol. 33, pp. 7354-7365, 2020.
[65] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn, "Visual adversarial imitation learning using variational models," Advances Neural Information Processing Systems, vol. 34, pp. 3016-3028, 2021.
[66] Y. Liu, A. Swaminathan, A. Agarwal, and E. Brunskill, "Off-policy policy gradient with state distribution correction," arXiv preprint arXiv:1904.08477 2019.
[67] O. Nachum, B. Dai, I. Kostrikov, Y. Chow, L. Li, and D. Schuurmans, "Algaedice: Policy gradient from arbitrary experience," arXiv preprint 19/2v: 2019.
[68] C.-A. Cheng, T. Xie, N. Jiang, and A. Agarwal, "Adversarially trained actor critic for offline reinforcement learning," in International Conference on Machine Learning. PMLR, 2022, pp. 3852-3878.
[69] A. Agarwal, N. Jiang, S. M. Kakade, and W. Sun, "Reinforcement learning: Theory and algorithms," CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, pp. 10-4, 2019.
[70] T. Xu, Z. Wang, and Y. Liang, "Improving sample complexity bounds for (natural) actor-critic algorithms," " Advances in Neural Information Processing Systems, vol. 33, pp. 1358-43661 2020.
14

---

Hopper (1.000 Expert Demonstrratoos!
Walker2d 11,000 Expert Demonstrational
HallCheetah 11, 3,000 Expert Demonstrations
0020
23000
Aeclog: Rohardd
2000
2000
Teper
Offinne Episodes
Offine Episodes
Offine Episades
Figure 3: The performance of Offline ML-IRL in different environments given 1, 1,000 expert demonstrations
Dataset rype medium
Favironment
Oftine MI.IRI
ValueDICE
CLARE
Expert Performance
hopper
1750.59
507.06
F43.59 + 500.8
2417.83 + 1258.30
1550.00 + 324.10
3533.30
medium
halfchectah
7690.30 + 119.17
2790.10 = 1046 00
-175,87 313.57
240 50 143.25
12107.50 55
medium
walker2d
+ 291.86
1248.13
1794.97 1400.70
320.51 93.07
5284133
medium-replay
hopper
2395.03 = 593.
843.59 = 503.83
2417.83 + 1258.30
2360.79 + 201.23
3533 0.3
medium-replay
halfcheetah
9313.29
261.94
2799. 1046.00
-175.87 313.57
343.37 108.00
12107.5
medium replay
walker2d
4100.99 +: 293.88
1248.13 4 72.89
1794,97 +: 1400.70
440.78 68.06
5264.33
medium-cxpeut
hopper
3366. +: 229.56
8 = 503.83
2117 83 = 1258.30
3071 31 = 186.37
3533.90
medium expert
halfcheetah
10812.15 551.38
1046.00
-175.87 343.57
202.05
12107.51
madium expert
walker2d
4049.43 + 1046.61
1248.13 + 72.89
1794.97 + 1400.70
$48.59 146.09
5284.33
Table 2: MuJoCo Results. The performance versus differrnt datasets and 1, 000 expert demonstrations. Here.
the expert demonstration dataset only includes a single expert trajectory (1, 000 expert transition samples) and the value of expert performance corresponds to the cumulative reward value of the provided expert trajectory.
Appendix
Limitations and broader impacts
Offline inverse reinforcement learning is a method designed to recover the reward function and the associated optimal policy from an observed expert dataset. However, potential negative societal impacts may arise if the demonstration dataset incorporates low-quality data. For safety-critical applications, such as autonomous driving and clinical decision support, we need exercise particular caution to avoid the introduction of detrimental biases from the demonstration dataset. Ensuring safe adaptation is vital for real-world applications.
One limitation of our method is that we exclusively use expert demonstration in the process of reward learning. Ideally, reward learning should incorporate varied data sources, like expert demonstrations and preferences (pairwise comparisions). A possible direction for future work is to broaden our algorithm and theoretical analysis for wider scope in reward learning. This would allow to develop a more robust reward model through the integration of diverse data inputs.
A Experiment details
In this section, we provide implementation details and additional numerical results.
A.1 Detailed experiment setting
In our experiment, we use two kinds of datasets: 1) transition dataset D = {(s, a, s')} which includes diverse transition samples and is downloaded from D4RL V2; 2) expert demonstration dataset DE = (r which consists of several expert trajectories and the expert trajectories are collected from an expert-level policy. All evaluations are based single NVIDIA GeForce RTX 2080 Ti.
The transition dataset. The transition datasets are downloaded from D4RL V2 and the transition datasets for each specific task have three different type: medium, medium-replay and medium-expert. In D4RL V2, the datasets are generated as follows: medium: use SAC to train a policy with medium-level performance, then use it to collected 1 million transition samples; medium-replay: use SAC to train a policy until an environment-specifie performance is obtained, then save all transition samples in the replay buffer; medium- expert: combine million transition samples collected from
15

---

Hopper 120,000 Expert Demonstrations)
Walkkrr (10,000 Expert Demonstrations
HaifCheetah (20.000 Expert Demonstratore)
$1.00
Newnage Seaoord
+400-
Tonvarr
2014
2009
1300
Capert
-260
Tacoe cead
Offline Episodes
Offine Episadies
Offline Episodes
Figure 4: The performance of Offline ML-IRL in different environments given 10, 000 expert demonstrations
Dataset type
Envirnemntt
Offline MI.IRII.
BC
ValueDICE
CLARE
Expert Performance
medium
hopper
2461.45 1 705.
3236, 70 70 05.97
3442.82 + 199.21
3317.52
270.45
3512.64 =: 17.10
medium
halfcheetah
7706.43 + 159.39
4935.60
2635.55
3248.60 2001.05
841.46
344.06
12156616
88.01
mediom
walker2d
4195.36 1: 352.86
2822.56 + 978.97
3046.76 + 2001.28
825.15 + 738 33
5365.62 + 55.79
medium replay
hopper
2889.73
542.65
3236. 70 45.97
3442.82 10::::1990 199.21
3139.98
478.75
3512.64
17.10
medium-replay
halfchesrah
9383.34
1: 358.67
4955.60
2835.55
3248.60
2001.05
437.18
182.99
12156616
88.01
medium-replay
walker2d
1092.58
+ 308.71
2822.56 + 978.97
3016.76
2001.28
869.07 + 612.56 56
5365.66 = 55.79
madium expert
hopper
33350.79
264.96
3236. 70 45.91
3442.82
199.21
3106.68 51.2.7
3512.64
17:10
medium-expert
halfcheerah
11276.09
551.94
4935. 60 + 2835.55
3248 60 2001.05
2020..5 + 520.46
121.56 16 88.01
medium expert
walker24
4363.54 + 729.60
2822.56 + 978.97
3046.76 2001.28
3245.79 + 1911.56
5365.66 55.79
Table 3: MuJoCo Results. The performance versus different datasets and 10, 000 expert demonstrations.
medium-level policy with another million transition samples collected from an expert-level policy. As a remark, since we are considering the setting of offline IRL where the ground-truth reward not accessible, we hide the reward information of those downloaded transition datasets from D4RL V2.
The expert demonstration dataset. The expert demonstration dataset includes the transition samples in several collected expert trajectories. Here, we first train a reinforcement learning agent by SAC under the ground-truth reward function to achieve expert-level performance. Then we save the well-train expert-level policy to collect expert trajectories. For each trajectory, it includes 1, 000 consecutive transition samples (s, a, s') in one episode.
In the model-based algorithms like Offline ML-IRL and CLARE, the estimated dynamic models are trained using the transition dataset. After the estimated dynamics model is constructed, the corresponding algorithms (Offline ML-IRL and CLARE) will further utilize the expert trajectories in the expert demonstration dataset DE to recover the ground-truth reward function and imitate the expert behaviors.
For model-free offline imitation learning algorithms like BC and ValueDICE, they directly learn policy to imitate the expert behaviors. Hence, those model-free offline imitation learning algorithms (BC and Valu)DICE) will only utilize the expert demonstration dataset D Due to the fact that BC and ValueDICE do not use the transition dataset D, their recorded performance in Table related to the type of the transition dataset D.
is not
In our implementation of Offline ML-IRL, we parameterize the reward network by a (256, 256) MLP with ReLU activation function. The input of the reward network is the state-action pair (s, a) and the output is the estimated reward value r(s, a; 0). Moreover, we use Adam as the optimizer and the stepsize to update the reward network is set be x 10-4 For the policy optimization subroutine (3) (4), we consider it as a model-based offline RI. subtask. Since it is under an entropy- reguaae-e framework. SAC-based algorithm is used as the corresponding RL solver. More specifically, we use model-based offline policy optimization (MOPO) (25) in the RL subroutine (13)- (14). For the implementation of MOPO and the corresponding hyperparameterrs, we follow the setup provided in [37 which guarantees strong performance through fine-tuning the key hyperparameters of MOPO, including the number of estimated world models, the choice of the penalty function U(..)) the penalty coefficient and the imaginary rollout length in the estimated world model. The official code base of (37 available in
During the training of Offline MI-IRL, each time we implement policy improvement subroutine under the current reward estimator, we update the agent by running MOPO for 20 offline episodes in the estimated world model and under the current reward estimator. After that, sample expert trajectory
sett/forumiid=zz20e
16

---

Hopper
Walker2d
HarChestah
-090
Baxe Itowarr
1400
Txpon
Tepos
126
Offline Episades
Offline Episodes
Offline Episades
Figure 5: Reward Transfer. The recovered reward by Offline ML-IRL in the medium-expert dataset transferred to the medium-replay datasets for solving offline RL tasks.
from the expert demonstration dataset DE = [r and sample agent trajectory from the estimated world model P to construct the stochastic reward gradient estimator following the expressing given in (16). Then are able to update the reward network by a stochastic gradient step.
For the benchmark algorithms, the official code base of CLARE is provided in https:// openreview htt.://ooormm.mddaaaaaaaaaaaaaa Moreover, the official implementation of VICE is provided in htttps://githut com google research coogoll researchh/rree/masterr value_dice
A.2 Additional numerical results
In Fig. we show the convergence curves of Offline ML-IRL when expert demonstration datasets D include 1, 1,000 and 10, 000 expert demonstrations (1 and 10 expert trajectories) respectively. Moreover, we compare the performance with benchmark algorithms in Table 2- According the numerical results, we show that our proposed method Offline ML-IRL is insensitive to the number of expert demonstrations, Even when only 1, 000 expert demonstrations are provided, Offline ML-IRL can achieve strong performance (close to the expert-level performance with the medium-expert dataset).
In Fig. [] we show the numerical results of reward transfer experiment. Here, we first use the mediim-expertt dataset and expert demonstration dataset generate reward estimator by running Offline ML-IRL. Then we transfer the recovered reward function r(:::3) 0) to the medium-replay dataset for labelling all transition samples with the estimated reward value. With the transferred reward function -(,-;;0 0) to label the transition dataset as {(*,, r(8, a; 8), we can treat the problem an Offline RL task and run MOPO to solve it. The numerical result shown in Fig. [ Comparing Fig. with the results in Fig which are obtained by directly running Offline ML-IRL with expert demonstrations/ trajectories, similar performance can be obtained by run Offline RL on the transition datset D with the transferred reward estimator. These results suggest that Offline ML-IRL can recover high-quality reward estimator, which can be transferred across different datasets to label those unlabeled data with the estimated reward value.
B Related work
Inverse reinforcement learning (IRL) consists of estimating the reward function and the optimal
policy that best fits the expert demonstrations [16, [7] 18]
20 31 In the seminal work [18)
formulation for IRL is proposed based on the principle of maximum entropy which enjoys the
theoretical guarantees to recover the reward function 229 30 55 56). Furthermore, in [57] 20 21). series of sample-efficient algorithms are proposed to solve the the maximum entropy IRL formulation. Moreover, there still exists one major limitation in IRL, due to the fact that most existing IRL methods require extensive online trials and errors which could be impractical in real-life applications.
To side-step the online interactions with the environment, offline IRL is considered to recover the reward function from fixed datasets of expert demonstrations. 13 by taking into account the bias of expert demonstrations, the authors propose gradient-based IRL. methods to jointly estimate the reward function and the transition dynamics. In [1], a model-free algorithm called IQ-L is proposed by implicitly representing the reward function and the policy from soft Q-function.
17

---

Although avoiding the online interactions with the environment, IQ-Learn sacrifices the accuracy of the estimated reward function, since its recovered reward is highly dependent on the environment dynamics. In [9] the authors propose a variational Bayesian framework to estimate the approximate posterior distribution of the reward function from a collected demonstration dataset. Recently, in [15), a model-based offline IRI. approach (called CLARE) is proposed, which implements IRL. algorithm in an estimated dynamics model to learn the reward. To avoid distribution shift, CLARE incorporates conservatism into its estimated reward to ensure the corresponding policy generates a state-action visitation measure to match the joint data distribution of collected transition samples and expert demonstrations. However, when the number of expert demonstrations is limited and/or most of the transition samples about the state-actions action-next state observations are collected from a low-quality behavior policy, matching the empirical state-action visitation measure of all collected data will force the recovered reward/ / policy to mimic the low-quality behavior policy, which is not enough to guarantee accurate model of the expert. Moreover, matching the visitation measure is sensitive to the quality of the estimated dynamics model. Matching the visitation measure in an estimated dynamics model with poor prediction quality cannot guarantee high-quality recovered reward.
Here, we discuss the literature in several closely related areas as below.
Online IRL. [8] is a seminal work in which the expert policy is formulated as the model that maximizes entropy subject to a constraint requiring that the expected features under such policy match the empirical averages in the expert's 's observation dataset. [29] [9] propose algorithms with nested loop structure solve such maximum entropy estimation problem. Those algorithms with a nested loop structure can suffer from computational burden, because they need to alternate between an outer loop with a reward update step and an inner loop that calculates the explicit policy estimates. In [58], the authors propose a method to estimate both reward functions and constraints from a group of experts. In [21. the authors propose an online IRL formulation based on maximum likelihood estimation perspective, which needs extensive interaction with the environment in order to learn the reward function. When the reward function is linearly parameterized, it is shown that the maximum likelihood IRL formulation is the dual problem to the classic maximum entropy IRL problem. Moreover, a theoretical analysis is provided to ensure the reward parameter can converge to a stationary solution in finite time. Despite the computational efficiency obtained in (21), it is not clear whether an optimal reward estimator can be recovered from the maximum likelihood formulation. As remark, compared with the online IRL work [1], our paper extends the maximum likelihood formulation to the setting of offline IRL. We further provide a statistical guarantee to show that when the transition dataset has sufficient coverage over the expert-visited state-action space, an estimated world model can be constructed so that the optimal reward estimator can be recovered. Moreover, we provide a computational guarantee show that the proposed algorithm can identify a stationary point in finite time. When the reward function is linearly parameterized, we show the optimal reward estimator of the maximum likelihood estimation formulation can be recovered by the proposed algorithm.
Offline IRL. In 59, the authors propose method to perform IRL through constructing a linearly parameterized score function- based multi-class classification algorithm. With an estimated of the feature expectation of the expert, the proposed algorithm is able to avoid direct RL subroutine. Given appropriate heuristic and expert trajectories, the proposed algorithm could recover the underlying reward function without any online environment interactions. In (60), the authors propose a model-free method to construct the Deep Successor Feature Networks (DSFN) to estimate the feature expectations in an off-policy setting. By parameterizing the reward linear function and estimating the feature expectation, the offline IRL. can be solved computation-frienddt with limited expert demonstrations. In [61), a modular algorithm called Offline Reinforcement Imitation Learning (ORIL) is proposed. In ORIL, reward function is first constructed by constrasting expert demonstrations with other transition samples {(s. a, s')} which are sampled from a behavior policy with unknown quality. With the constructed reward function, the quality of all unlabeled data can be evaluated and then an agent trained via offline reinforcement learning. In [62). the authors proposed an IRL method which leverages multi-task RL pre-training and successor features to train IRL from past experience. When a set of related tasks can be provided for RL pre-training and the feature expectation is estimated accurately, the proposed method could benefit from the multi-task pre-training. However, in practice, is difficult to obtain full knowledge of the feature structure the ground-truth reward function.
Offline Imitation Learning. Different from offline IRL, offline imitation learning aims to directly imitate the expert behavior by learning a policy in the offline setting. In (28). an algorithm called
18

---

ValueDICE is proposed to leverage off off-policy data to learn an imitation policy with the use of any explicit reward functions. Moreover, the numerical results of ValueDICE show that it could be easily implemented in the offline regime, where additional interactions with the environment is not allowed. In [63], the authors propose method to learn both policies and environment and analyze the corresponding error bounds. In [64], a model-free offline imitation learning algorithm is proposed by energy-based distribution matching (EDM). EDM provides an effective way to minimize the divergence between the state-action visitation measure of the demonstrator and the imitation policy, In [65], the authors develop a variational model-based adversarial imitation learning (V-MAIL) algorithm for learning from visual demonstrations. By constructing variational latent- space dynamics model, V-MAIL is able to solve sigh-dimensional visual tasks without any additional environment interactions. In 22] . the authors introduce Model-based Imitation Learning from Offline data (MILO), which extends model-based offline reinforcement learning to imitation learning. The theoretical analysis of MILO shows that full coverage of the offline data is not necessary. When the offiin dataset provides sufficient coverage to cover the expert-visited state-actions, MILO can provably avoid distribution shift in offline imitation learning by leveraging constructed dynamics model.
Offline RL. Offline RL considers the problem of learning policy from fixed datasets where the reward value is provided for each collected transition samples. For model-free offline RL algorithms, world model is not estimated and the algorithms directly learn policy from the collected dataset. In
In [66 67]. model-free offline RL algorithms are proposed to solve the importance sampling problem. 23 688. conservatism is incorporated into the value function avoid overestimation in the
offline RI. setting. For the model-based offline RI. algorithms, (24] constructs the estimated world model and sets hard threshold the model uncertainty for constructing terminating states to avoid dangerous explorations. In [25], the authors proposes a model-based offline policy optimization algorithm (MOPO) which utilizes uncertainty estimation techniques to construct a penalty function to regularize the reward function. Therefore, MOPO can learn a conservative policy which stays in the low-uncertainty region to aviod the distribution shift issue. As follow-up work, 37 revisits the design choices of several key hyperparameters in MOPO and fine-tune the corresponding hyperparameters in MOPO to guanrantee strong performance. In (26), the authors propose modellbased offline RL algorithm called COMBO which does not rely on explicit uncertainty estimation. By regularizing the value function on -iitt-idittiitttion state-action pairs generated in the estimated world model, COMBO can benefit from the conservatism without requiring explicit uncertainty estimation techiques. As a remark, the algorithms proposed in 25 24
37 26
all perform
conservative policy optimization in a well-constructed dynamics model and the estimated dynamics model keeps fixed during the training of the RL agent. Different from those algorithms mentioned above, 45 47 incorporate conservatism into the constructed dynamics model. By adversarially modifying the estimated dynamics model to minimize the value function under the current policy, the proposed methods can learn robust policy with respect to the environment dynamics and can obtain probably approximately correct (PAC) performance guarantee.
C Auxiliary lemmas
Before we introduce the auxiliary lemmas, we re-write Assumptions 0- here for convenience.
Assumption 1 For any reward parameter 0 and any state-action pair (s, a). the following conditions
hold:
|r(s, ())) S Cr, (U(s S C.
where C,- and C. are positive constants.
Assumption Given any policy ग, the Markov chain under the estimated world model Pis irreducible and p E (0, ensure the following condition holds:
and aperiodic. There exist constants
|| e -sso = s. (-)l & kpp, Vt
max SES
where || l/rv denotes the total variation (TV) norm; nz is the stationary distribution of visited states
under the policy π and the world model P.
19

---

Assumption 3 Under any reward parameter 8, the following conditions hold for any s E S and E A:
Ver(s, a; 0)1 S Ly, Vor(s, a; 01) Vor(s, B2)]l $ Lgll01 02l1.
where L, and bg are positive constants.
Then we introduce the auxiliary lemmas as below.
Lemma 5. (Proposition A.8. in (69]) Let be discrete random variable that takes values in
{)...., d}, distributed according to q. We write q as a vector where q = [Pr(z = =]]l Assume
there are N i.i.d. samples, and that the empirical estimate of q is [ū]; Ein 1[ = j]. where is a d-dimensional vector.
N
Then for any > O, the following result holds:
Pr l1ğ
1 VN
+e) SE-N
(23)
which implies that:
Pr 11 - 711 vd(
I Se-Ne. )c)===
(24)
VN
Lemma 6. (Lemma' 3 in (Z01) Under any initial distribution 17(-) and any transition dynamics P(.(s,a), a), we denote d= as the visitation measure of the visited state-action pair (s, a) with a softmax policy parameterized by parameter w. Suppose Assumption 2 holds, then for all policy
parameter w and w', we have
[|dmn(:s)) - (.: S Callu w'll
(25)
where C. a positive constant.
Lemma 7. (Lemmna 5 in (311) Suppose Assumption 3 holds. Under the soft policy iteration defined in 313 T4p. we denote the soft Q-function under reward parameter 02 and policy Tx+1 as Q 3
Moreover, recall that Q&+1 has been defined as the soft Q-function under the reward parameter 0x+1 and policy Tk+1. Then for any E S, A and k. 2 O, the following inequality holds:
14++ (s, a) - (k+1(s,a) I Lgl10, 0x+1ll
(26)
Lr where Lg 1- 0 and L, is the positive constant defined in Assumption
Lemma 8. (Lemma 6 in (37)) Following the soft policy iteration defined in 13 holds for any iteration k 2
14 the following
2yEapp
ax(s, S 1.+2k(( 1-7¹
Vs S
(27a)
2T<pp
11Qu, -u+ S s11Q0 Qullo+
(27b)
1-7
where (x- denotes the soft Q-function under reward parameter 0g and updated policy Th+
and ()) denotes the soft Q-function under reward parameter 0g and corresponding optimal
policy The- Moreover, we denote |.Qu Qu+ g max,es max (Qo, (s, a) Q (s, a)).
Remark. In Lemma 7 and Lemma 8 the definitions of the soft Q-function Qk and the soft value function VA under a conservative MDP are given below:
(s a) = r(s, a; 0x) + U's, a) - [s')]
(28a)
xc
Vi((s) := BB
Σ
* (r(s), 0.) ) U(s,, at) + H((T = ]
(28b)
t=0
where U(s, a) is the penalty function for the state-action pair (s, a), which is used to quantify the uncertainty in the estimated world model P. If we rewrite the above defined soft Q-function Q2 as:
Qu(s, a) F(s, a; 0x) + - (s())
where f(s, 0z 0g) == r8,, a: 0c) + Us, a), then we can directly follow the proof steps in 31 to
prove Lemma 7 and Lemma
20

---

D Proof of Lemma I
Proof. According to the closed-form expressions of the optimal policy xy and the optimal soft value function Vo in (4, let us decompose the objective L(0) defined in 22) as below:
L( E, E.
Y' log (a(ls))))
exp oe)
E. (.
>' log Eeed exp Qs(
Exesexp
E, E, P)
-Vo(ss))))V(ss)))
(o(sn. ai) - - Buy Volot)
E ((
a(s); U(s, as) + T M (o(Sun))) -
Bn Vo(o.)
tm0
[m)
'B ((Sh Us) Eegonnt- Vo(60)
++1 E Pa [Vo(set)]
/ B Barir [Vo(set1)]
B (ooe r(a,, ar: U(sr at)
B2o-n()) [o(80)]
1=
surrogate objective
8
( Phanaaattt [Vo(s41)]
E
[Vo(sc11)]
T2: error term due to transition probability mismatch
(29)
where (i) and (ii) follows the closed-form expression of the optimal policy my and the optimal soft value function in ( Moreover, (iii) follows the definition of the soft Q-function in (b).
From the T2 term in (29). we observe that the error term comes from the transition dynamics mismatch between the estimated world model P and the ground truth dynamics model P. Then we analyze the transition dynamics mismatch:
T2 =
++1 E
(enaa)) Plater) (Va(st)) [o(++)))] -
Σ ++1 E (t
Pheeerr [o(se+1)]
0m0
++1 ((oe
B Peen~P(1enaa) [o((++1)] - Passs
++ E (e
Σ ( ai) P(s442ls),a)))
S++1ES
Recall that d..... is defined in where ( a) == F Eio p (st s/so ~ m)
we obtain the following result:
T2
Blo
Σ #11ES
Vo(se41 sc) - P( ozl)
) P(s s/50 7) E (ot = ale: =)) Σ
= 8,00 a) p( =0))
t= sES
-
- (s)ss
(30)
21

---

Then we further denote the surrogate objective E(-) as below:
[())
=
Y ((n
P) ((st, 01; 0) + Use, an) E
s
vo(so)
(31)
1=0
By plugging below:
30 and (11 into 29 we obtain the decomposition of the likelihood objective L(0) as
L = L()) + ?
1-7 ((s.a))dE(..))
Σ
Vo(s') (P(sss(( -
(32)
s'ES
The lemma is proved.
E Proof of Lemma 2
Proof. According
32 we have the following series of relations:
[L(0) i()) = 1 7
?
E ((.a))aP
Vols') (() -P())s.
e'ES
7
B(s,a)nd"(s))
Vo(s') |P(8's, P((11sa))
1-7
e'ES
< 7
mmx [V((3] RES
E Bss
Σ -
P(s [|)
1-?
EES
7
R(0) [a.a) (1.)
P( (|ls,a))l) |l1
(33)
1-7
where
follows the definition R(O) :- max.es Vo(s)
and the definition of the visitation measure According to the definition of Vo(-) in (3a) we have
5
R(0) max .ES
[V((s]
(0) max SES
E (1, P)
(r(ss a;; + (st, H((o(((ss)))) 0o =
max
E (n
Σ
71(r(s,, oz 0)1 + 1U ar) + 80
#E8
(6)
Σ max sES
(n
HC, + Cv + [ so = s)
(182)
max
Br P)
(', C, + log (4)) =
SES
C, + C. + log |A|
where (i) follows the definition of V-(+) in Ba) and (ii) follows 8 Moreover, (iii) follows the fact that information entropy is non-negative and the maximum entropy is obtained under uniform
distribution where we have ( (-I)) = (((T(()1)) < E _ log [ log [A].
Denote C, = C.+C LA| result into
we obtain the property: R(0)= = max,es Vo(o) S C.. Plugging this we obtain the following result to finish the proof:
I-
33
7C [L(0)) [(0) < 1-7 E( (1)) [|| P(1)s a) P(-|s, a)11]
The lemma is proved.
22

---

Proof of Lemma 3
Proof: To start the analysis, we first take gradient of the surrogate objective [(()) (as defined in w.r.t. 0:
vi(0) (2)
E En
(Vor (Vor(s ax:0) VoU(st, ax)] -E Bsson()))
Vo VoVo(so)
(it) E (1,
Vorrss, 0)
-Boom())
Vo Volog
Σ
exp Q0(00, (8)
1=
aEA
=B,en(n
8
Vor(st, Baoom()))
exp a)
VoQo(so.a))
ue exp exp Q0(
aEA
E E (i
/ Vor(s: 0r:0)
-Bson(()
To (al (ololss)VQ
aEA
E,Ew(n
(oor(
VoQo(su.
(34)
where (i) follows the definition the surrogate objective I()) in (7) and (ii) follows the closed-form expression of the optimal soft value function Ve in (4) and the penalty function (s, a) is independent of the reward parameter 0. Moreover, (iii) follows the closed-form expression of the optimal policy To in (4) To further analyze the gradient expression in (34), must derive the gradient of the optimal soft Q-function Qo. Recall that Qo is the soft Q-function under the optimal policy Te. the
penalty function and the estimated world model P. Therefore, we have the following derivations:
VaQa ao)
V r(so, uo; 0) + U80, un) + 7IE, P [Vo(si)]
62)
ver(so, a0: 0) + VoU(so, ao) + 7B P Volog Σ exp CexpQo(8pp (o((0.0a)
ded
exp a
Ver(so, 0) 71E 51 P(-
Ened exp Q0(81, VoQo(s1.a)
(181
Vor(su, ao: 0) + +B 7 s
Tolalsz)
(osaa)
aEd
Vo 0() U +nE szn (() [V((2))) ]
V Vor(so: a0: + yIE sy (10
BB
where (i) and (iv) follows the definition of the optimal soft Q-function in 3b); (ii) follows the closed-form expression of Ve in (4); (iii) is from (4). By recursively applying the equalities (i) and (iv), we obtain the following result:
VoQo(so, ao)
Vo ((s0, (oooaa;:0 ag;00
+ (so, ao) 7B ao)
Vo ((()))))))) U(1,, + 71E 71B - p [V((2)) }
Vor(so, api 71E (100)
-EAAN(TT.P)
Ver (1) 50,
(35)
1=0
23

---

Finally, by plugging
35 into 34
we obtain the gradient expression of the surrogate objective L())
below:
vī(0) = B
V Vor ac:
-B (110))
VoQo(so.a) ao)
V (5t, (5r 0)
-Bsonn())
1 *AN(Ta.
/' Vor(s,:
so. ao
1=
Vor 8cs ar: 0)
E -B
Σ
7 Ver(st,ari0)
(36)
1=
The lemma is proved.
G Proof of Lemma
Suppose Assumptions hold, we prove the Lipschitz continuous property of the optimal soft Q-function Qo in 11a and the Lipschitz smooth property of the surrogate objective L(0) in (99)) respectively.
G.1 Proof of Inequality (9a))
Proof. In order show that the optimal soft Q-function (s a) is Lipschitz continuous w.r.t. the reward parameter 9. for any state-action pair (s, as. we take two steps to finish the proof. First, we show that Qo has bounded gradient under any reward parameter 0. Then we could use the mean value theorem to complete the proof.
According to the gradient expression of the optimal soft Q-function Qo in (35). the following result holds:
[V
E
V Vor(se; 8, =
1=0
E B .)
V Vor oz: ();
800 8,40
t=
(18)
S E
A P)
J br 80 = 0, = a
1=0
Tor
(37)
1 1-7
where (i) follows the Jensen's 's inequality and (ii) follows from the assumption that the reward gradient
bounded; see (18) in Assumption 3 Denote Lg {em we are able show the Lipschitz
continuous property the optimal soft Q-function Qo below:
[Qo, (s, a) (x)
( (0) 02, VoQg(s,a)) 00 02ll [Voog(s, < L1 02ll
where Ø a convex combination between 0 and 0z. Moreover, (i) from the mean value theorem and (ii) follows (37
24

---

G.2 Proof of Inequality (9b)
Proof. In Lemma 3 we have shown the expression of the gradient of the surrogate objective T.(0).
Then for any reward parameters 01 and 02, we able to obtain the following result:
[[] vi(02)l
E,
y Vor(s), (x:11)
B, A P)
(Beendnat m))
V (or((ss
BAAN(O P)
Vor((t ar: 02) 1)
<
E ( P))
>' Vorrss - -B
V orr Vor(s 02)
1=0
1=0
+
B P)
y Vor(s,, (or(s,, (1:b1)
E B P)
y (5r
(38)
where (i) follows the gradient expression in 15 For the first term in relations holds:
38 the following series of
E (NE
Vor(s ar; 02)
1=
1=
E,E~(neBB
X' (Vor(sn Vor(s,, (2)
(
E, (n.
Vor(st, V Vor 4es 0z; 02)
(0)
7 7bo 02 0,
1=0
Lg
1 0n - -211
(39)
where (i) follows Jensen's 's inequality and (ii) follows (8) from the Assumption have
For the second
term in 38
DO
E (q,
BAA P.)
Vor(s, an;))
EF E p.)
v Vor
B
BRAN A,
Vor (1:b))
EAAn(y.TO...))
Vor(st,
1=0
t=0
T1: enror term due to policy mismatch
R
Vor (st
E B.AA(M
Vor(st
(40)
An P1
T2: error term due to reward parameter mismatch
In 40 we decompose the difference between reward gradient trajectories into two error terms. The first error term is due to the policy mismatch between тe, and meg- The second error term is due to
25

---

the reward parameter mismatch between 01 and Bz. Here, we first bound the error term T1 in (40)
(1.
P)
Vor 0::01)
E B
P)
Vorr(s,
az;
(a To
[or(s,
1-7 E
(saa)dy xx (1))
[or(s,
Vor(s,a;01)- (d (s,a) dx (s,
sES,aEA
(or(s,a;01)] 10 (s. a) - d (s.a)
sEB,aEA
(1) 2Lr Thi (3. - -x (3.)llVV
-
(10)
2I,
$
1 y 1Q0, - -Qall
(1)
2L, Cav |8| [A|
10e, Q0
7
II Cay () A
1|0
02ll
(41)
1-7
where (i) is from the definition of the visitation measure and (ii) follows the bounded gradient
of reward function in 18) in Assumption 3 Moreover, (iii) is from (25) in Lemma 6 and (v)
follows the Lipschitz property (19a) in Lemma 4 For the inequality (iv), it holds due to the fact
1Q,, (s,a) Qo, (s,a)l S liQe, Q0 for any fate-action pair (s, a) and thus |[Q,, Qa s
VISI |
||Qa, Q 11,0
Next, let us bound the second error term in (40):
80
E
BA M)
Vorrsse, ari)))
E E, P)
Σ
r > Vor ari 02)
B B.
(Vor(st, Vor(si, ar: 02))
1=n
(8)
1 A~ (m. 3)
Vor(st: 02) (ar(s,,a;:01)
a,; 02)
1=0
(11)
o L9 821l
Bysw(nnToy P]
(=0
Lg 1|0 Azll
(42)
where (i) follows Jensen's inequality and (ii) follows the Lipschitz property in (88 from Assumption Plugging 41) and 42 into (40). could show the following result:
2L, L, L.
|S|
|A|
1AI++ Lg
B (o,
Ver(ss (c;)) E, EAA
Vor(st,
110
02l1
1-7
(43)
We denote the constant L. 2I, |S|-|||+ By plugging 39 and
48 into 38
arrive
at the desired Lipschitz property of the surrogate objective Ī(0), shown below:
vi( vī(0.)) < Lol10 - 8211-
We completed the proof of the Inequality (9b..
26

---

H Proof of Proposition
Proof. Before starting the proof, first recall that have defined the expert-visited state-action space
Q= {(s,a) ds, a) > 0} and the expert-visited state space S = {s EOEA de (s, a) > 0}.
Then have the following analysis.
For all state-action pairs (s, a) E ft there are at most | active states the distribution P(-ss, a) such that P(s'|s,
[s, a) > 0. This is due to the fact that the next state s' of an expert-visited state-action pair (s, a) must belong to the expert-visited state space S Then suppose there are N i.idd samples each state-action pair (s, a) S2 we can obtain empirical estimate of the transition probability as P(s'|s a) (ssa.a²) for any E S where N(s,, a, s') is the number of observed transition N samples (s, a, s'). For any (s, a) E Ω and s' & SF, we know P(5'(s,)) N(sa..)) 0, since the
expert only visit states in S..
Therefore, any state-action pair (s, a) E Ω and by following (24) in Lemma 5 holds with probability greater than eeNe:;
the following result
|P(()s, a) P(-ls,))ll S V VISEI.
1
to)
(44)
VN
Let us denote δ = Ne² then we have = N
In By plugging into
44
we can show that
the following result holds with probability greater than 1 5- 6:
P(-ss, s Visei-
( + V
SE
1
= 9
In
(45)
N
where we define = 1+ V
as positive constant dependent the probability 8. Then we can
further show that
SE
1
P max (s.a)nn
(P({s,a) - P(-1s, 2c
In
N
Σ p ((1P((ss0)- P(-1s, 2 ev
|S
1 3
In
N
(8
(41)
[2] -8
where (i) follows the union bound and (ii) follows 45 1 [2] 5, we have
Therefore, with probability greater than
SE
max [(((ss, (s.0)ER
-P S
N In
(46)
Denoting 8 : [Ω] 5. then we have ő- -. [ following result holds
Therefore, with probability greater than 1 - 6, the
S
|| - -(-ss, Sey (s.a)nn
max
In
(47)
N
8
In order to control the estimation error max(s.))en) (P(.ss samples N at each state-action pair (s, a) E Q should satisfy:
P(l|s, a) & e, the number of
SE
e² SE
<e
N
Suppose we uniformly sample each expert-visited state-action pair (s, E Ω and the total number of samples in the transition dataset D {(s, a, s')} satisfies:
|2|
Ω
#transition samples > |62| N
In
(48)
22
8
27

---

Then with probability greater than ó- the generalization enror between P and P could be controlled:
a (()) P((ss,)) P(.ls. a))la < max [P(+)s, P(-|s, a)ll S e.
(49)
(s,a)En
The theorem proved.
I Proof of Theorem
Proof. First, we have the following decomposition of the error between the loss function evaluated 8" and 6, respectively:
L(0") L(Ø)
(L(0") - ()) + (i(0) Ê(O)) + (i(0) L(Ō))
(9 30.
((aa) P({)..) p()s,)))l,) (I(0") I(0))
1-7 (
JC, (.. P()s,a))1,]
1-7
2yCr.
1-7 B( P(-ss a) P(+s, a(Hss + (L(0") - L(())
(50)
9 Since we have defined 0 as the optimal solution to Σ(.), we know that T((0) (0) < 0 for any 0. Plugging this result into 50 the following result holds:
where (i) follows
2yCr
L(00)= (()) -
(a
(..) [|Is P(-ss, a)].] + (ī(or) I(0))
1-7
<
(o, (.-) (I|ssssss P(-ls.))l)
Based on (48 and (49). assume that each expert- visited acttin pair is uniformly sampled
and the total number of transition samples satisfies:
2² | |8
12
412 C c². [ 10] (55)
[22
#transition samples 2
In
(1-)))
(1 =g
2yC
Then with probability greater than 8, the generalization error between transition dynamics and the optimality gap between reward estimates could be controlled:
(1-7)E
(( (()s(
(1) P(-|s, 111 <
2yC
L(O") L(0) ~ 27C,
- B(aaa)a [1] P(-1s,a)llil Se.
The theorem proved.
Proof of Theorem 2
Proof. In this section, we prove the convergence results 20a)
20b
respectively.
J.1 Proof convergence the policy estimates
20a
In this section, we show the convergence result of the policy estimates {Tk+1 hzzu, which track the optimal solutions [ Ja20- Recall that each policy T generated from the soft policy iteration in (14), then we track the approximation error between mh+1 and To, below:
log T - log Te (als)
i log exp (s) a)
ΣREA exp ()) ) log
exp Qu (s,a) Enec exp exp )1
[(Qu(s,a) log Σ exp (x, (Qo, (s, log
Σ exp Qo (s,
REA
aeA
< (u((sa) - Qo. (s,a
log Σ exp @xp - -log Σ
exp Qo. (s, (51)
aed
aEA
28

---

where (i) follows 14 and (4) In order to amalyze the second enror tern in (5T). we first denote two
-ilmmnennio vectors d' [a1, 02:-11
:ar::::aaa -qail and b' = [b,, bb, b2, b J. Then we could obtain
the following result:
log (Il (xp(d))bb - log (Il expp b) ]11))
b, V log (1 exp( ())(()))1)
bx V log (1l cxp exx(())
1xc
(52)
where (i) follows the mean value theorem and 8 is a convex combination between vectors 7 and b.
Moreover, (ii) is due to the equality that
exp(u:)
< log (Il exp (I
V log (Il exp ()))( 11 11,
exp 1
Based on the property we show in
52,
we could further analyze 51
as below:
log ( - log Toe (als) -
Qu(s, a) (s, a)
+ log
exp (s. - -log
exp (s, (s,
aE.A
aeA
(s, -o, + max
(s( a) Qo (s. a)}-
(53)
acA
If we take the maximum over all state-action pairs the both sides of 53 and denote log क max log π (a s)], then we obtain the following property:
|| log Th+1 - log To, To I 2 - Qoal
(54)
Recall that QN is approximation the soft Q-function Qx where Eapp is the approximation error, then we have:
log T - log moull 22 Qaull = =2 Qx + Qx Qox S 2eapD + +2|1 - Qa. lo
(55)
In order track the approximation error log Tk+1 log Te, Il we could analyze the convergence between the Qk and Qe, according to (55). Here, we define an auxiliary sequence (Qx+ y J220 in the conservative MDP where Q is the soft Q-function under the reward parameter Og and the policy T defined in (14. Then we have the following analysis:
||Qx Qo a0
1 Qo,
Q0n-1 Qu.,, Qh Qu-2lla
< 10u Qu. .1lsc + +1|Q0, - Q bl1o + +1|Qx Qx-
|1 Qx 111 + 22gll(0x - 0k-z/l
(0)
27Eapp
S ?1|Q0-1 Qk- + 2T, - 0 +
(56)
1-
where (i) follows 19a in Lemma 4 and (66 in Lemma Moreover, the inequality (ii) follows (27b) in Lemma 8 Recall that the update rule of the reward parameter @ defined in (7). then the following result holds:
(4) 2
||0, 02 || = allg--11l all
1: a)ll s <a (1l1((00 )) a (1 Oll
2
1 1-7
where (i) follows the definition of the reward gradient estimator defined in (66). The incquality (ii) follows the definition of h(0;7) in (16) and the bound gradient property
V Ver(s, a; 0)] < L, in
(18) from Assumption Recall that we have defined the constant Lg :=
L. then we obtain the
1-
following result:
|0+ 0x-1l1 S 22Lg.
(57)
Plugging 57 into (56). we can show that
2yc
||Qx Qo. Ilso - Q0n 11.0 4aI2
(58)
29

---

Summing 58
from k = 1 to h = K and dividing K on both sides, then we have the following result:
K
K-
Qx - Qu 1l.00 S
7
Qxx - Qo, oo + +4x11 + 1-9
2yppp
(59)
K 1=1
K
k=
By rearranging
559
obtain the following inequality:
2 (1100 - Qoollc 11Q - lox 4al's + 27Eap 11 Qoalla - K k=1
Σ
(60)
K
1 1-7
Assuming the initial error 11Qu 100000011 alc is bounded by positive constant Ag where 1Qu-Q0olle Au and dividing 1 & on the both sides of (60), then the following inequality holds:
-
1
K
4aL2 7
2-/eap
Σ
Qx Qo
K(1 7) Do
(61)
K
71-
Then we subtract following result:
Q0x 1lo0 and add : 10 Qualoo on both sides on 61 we obtain the * Qk
K-1
Q2x
Q0el100 -
7
Do
+
1
Ao 1Qx Qax 10.0 + 1-7 +
4a Capp
K
K(1 - 7)
K
21-
-
0=
1
4a 2
2yFpp 71-
K(1 -1 An
+
(62)
7
Plugging 62
into (5). then we obtain the convergence result the policy estimates:
1 K-1
K-1
log M++1 log Tox
$
2capp +
2
Σ
1
10x
K
K
(1))
47
2Ao
8aL2
Capp +
(11-)) +
(63)
7)2
K(1
where (i) follows 55 and (ii) is from 662 Recall that the stepsize is defined to be op K-# then we could obtain the convergence rate of the policy estimates as below:
log Th41 log ne Il -E(ap)) + +O(K +).
(64)
A=
The relation (20a) has been proven.
J.2 Proof of the convergence of reward parameter
20b
Proof. In the section, we prove the convergence of the reward parameters [0x] Recall that have shown the Lipschitz property of the surrogate objective [()) in (19b) of Lemma Based on the Lipschitz smooth property, are able have following analysis:
Ie
ī(0k+1) 2 [((+) (vi())), 0x41 0k)
11
12
2
(6) [((b)) - 9x)
L.a²
11
+ =[
+ a:l ( V
2 |1
(65)
where (i) due the Lipschitz smooth property in (99b) and (ii) follows the update rule of the
reward parameter in (17). Recall that the gradient estimator Sh is defined in (16). Due to the bound gradient of any reward parameter, could obtain the following bound:
(1)
2T,
||gx|] |l [1(( + h(Ox: A
[A(0k); F
[1((k: T
S
(66)
1 1-7
30

---

where (i) is from (6)) and (ii) follows the bounded gradient of reward parameter in (T8) of Assumption
Recall that we have defined the constant Lg == I-y then the following result holds;
[(0++1) 2 [(0)) + -x VZ((0) + +Z -2
(67)
After taking expectation on both sides of (67 have
E [[((0]+))]
2 R. [T((0))]] + aB -V + 22
= =E[[(0.] + aB (VL E(gx
+ (eE - -2L, LL.
0
@ ((a) + +IF (< (Vi,(O:), BAANEnn P)
"Vor(s), 0;; 0x)] F. A
Vor
1=0
t=0
+aB[| 12.
(1)
% [[(()=)] 2 | BAA
V Var(st, ar 0w)] - (1.
Vor ar: 0k)
t=0
t=0
T1: error term due to policy mismatch
aE [[ -
(68)
where (i) is from the definitions of the reward gradient estimator in (T6) and the reward gradient expression in (15) of Lemma 3 (ii) follows [|V] < 2h, which could be proved according to (66). Then we analyze the error term due policy mismatch as below:
20 Vor(se -E Bra Vor(s 0b))
1
E
(v)) Vor(s,a;0x)) 1 (s0) k+ (.))
E
Vor a; 0w)
(s,a)ad
Σ
Vor(s,a;(b) (dp² (s,a) - of (s.
1
SES
Vor(s,a;00) 0x) -1 (s) det (s,
SES,
(0)
L
1d (s,a) - oz (s,
sES,aEA
(181) 2
[|055 (1.)) - df (.))lT))]
E
(69)
Jex (:,-) and d (ii) follows the bound where (i) follows the definition of the visitation measures d gradient of reward parameter in (18) of Assumption 3 and (iii) follows the definition of the total variation norm. Recall that we have defined the constant I. = I Due to the fact that тe, and
Th+1 are softmax policies parameterized by Qe, and Qx. then obtain the following result based
31

---

on Lemma 6
2
m0 P)
?
Vor(st, 0r:
B Bo
V
For(s at 0k)
= 22, || p (.,
(s [1v
- 2L C.B 1100 - -x1l]
2L,
Cav
81 A B |100 [0
-2,
A E
11Qu, Qx + +Q Qx
S2L,CaV 8
18
11Qe, 1x11 + Capp
(70)
where (i) follows 25
Lemma and (ii) follows the conversion between Frobenius norm and the infinity norm. Plugging (70) into (68). have
E [[[(0x+1)] 2 [[ L A
1100.
-Q + Eapp + al - - 2L.L,a²
(71)
Rearranging
7 and dividing its both sides by a, could obtain the following result:
B[IVI(0+) - +
-
/ |
S|
|A Qu1lo0 +<app II
(72)
Let us denote the constant C. == 4C S dividing K on the both sides, then we obtain
x
[A]. Summing 72
from 0 k: K and
K
B[](Ok) [(00)]
Co
K
E [[I k=0
&
Q0, Qull + <app) + 2aLL, I
K
aK
K
k=
(73)
Recall that bound the gap between the likelihood objective and the surrogate objective in (9). Then we have
7C.. (L(0) I(0) I- ((.aa) [[|(()),,)) P(-ls, a)lh]
27C
(74)
I 1-7
(s, a) S 2 holds for any state-action pair (s, a). Since L( is the log-likelihood function which is always negative, we can show that the surrogate objective L(0) is upper bounded under any reward parameter 0. Denote positive constant C 2yCo have
where the last inequality is due the fact that -
i(0)= < L(0) + 27C,
2.C. Ct. 1-7
(75)
Furthermore, considering Ż(0b) the initial value of the surrogate objective, we can simply denote C2 = L(Ob). After plugging 622 into 73 we obtain the following result:
C1 C2 + Coeapp + K(1 7)
Codo
2 4aCo
[[IVi()) < aK
+ 2aLob?
1 1-7
(1 - 7)
k=0
Recall that the stepsize a is defined as a an K- t, then we can show the order of the convergence error as
= o(K - ((app))
E
K
This completes the entire proof.
32

---

K Proof of Theorem
Proof. In this section, prove the optimality guarantee when the reward function is linearly parameterized as r(8, 0) == ((s, a) T where Φ(s, a) is the feature vector the state-action pair (s, a). Based on the definition of the surrogate objective in 7 and the definition of the soft value function in (3a) we can rewrite the formulation (12) as below:
max [(0) == E, E,E. (n(nE 8
((()) a; US5. ai)]
L8 -
(r(se: uri
0r 0) Us), (+(o
(76a)
1=
8.b. To == arg max E SAA(T
7dth (((se, ar; + (st, ai) + 7 (8)
(76b)
As a remark, there is a common term L (yn Ei 76 ar; (); a.) H (F(+|s)))
in both (66) and (6b)). Then we can obtain the following formulation which is equivalent to (76)
max [(0, π) == En ( 0
ax; U(s.))))))))))
ai))
1=
? (r(se: (c: U(s, H(T((1s)))
(77)
Based on the equivalence between (76 and 777. we further show that any stationary point 0 in (76)
together with its corresponding optimal policy Ty consist of a saddle point (0, TT) to the problem (77) when the reward linearly parameterized. We summarize this statement in the following claim:
Claim 1: Assume the reward function is linearly parameterized, i.e., r(s,, a; 0) == Ø(s,a) a) where
ő(s, a) is the feature vector of the state-action pair (s, a). Any stationary point 0 in (76) together with its optimal policy Tg consist of saddle point (6, ng) to the problem (77)
Here, we show the proof to Claim below.
Under the linearly parameterized reward function r(s, a; 0) == 0(8, a) 0, we can further rewrite
C(0, π) as below:
DC
C(0, m)
0, O, P)
Σ r (s ae) t=
E EEAN(n
> (os an)
Term linear function of the reward parameter
(s at) - E (y, P)
(U(s). + H((T(-18))))
E .P)
t=
Term B: independent of the reward parameter 0
(78)
Note that the max-min objective L(..)) linear (thus concave) in the reward parameter given any fixed policy т. We can utilize this property to prove the statement in Claim 1.
Recall that a tuple (0, ng) is called a saddle point of C(.,-) -) the following condition holds:
C(O, RT) S C(0, mq) < C(0, T)
(79)
for any other reward parameter 0 and policy To show that (0, mg) satisfies the condition 79 prove the following conditions respectively:
i arg max L(O, ((0,mg),
(80a)
Ti E arg min L(,, T).
(80b)
33

---

Here, we first show that any stationary point of the surrogate objective [(-) satisfies the optimality condition (80a). Recall the gradient expression of L(.) in [5]. any stationary point & of the surrogate
objective (()) satisfies the following first-order condition:
Vi((0) B,
Vor(st, oe: 0)] B (nag P)
V Vor (0r ax;
1=0
1=0
Moreover, when the reward function is linearly parameterized r(s,a;0) Φ(5, (.a,). 0. obtain the following condition for any stationary point A:
Vi = E,
X' (5 an) B (n
Σ
y o(1 az) = 0
(81)
1=
1=0
Then we can go back to the formulation .):,)) in (8)). Given any fixed policy т, we can show the gradient of [(0, π) w.r.t. the reward parameter 0 below:
v voL(0, = E,
( an) - E (T
(
(82)
t=0
t=
Due to the first-order condition we show we obtain the following result:
VoL 0 = TT) 0.
(83)
Recall that have shown -) is concave in terms of the reward parameter 0 given any fixed policy и. Due to the concavity as shown in (78) and the condition in 83 we have completed the proof of the optimality condition (80a).
Then we prove the optimality condition 80b). Recall that Ty is the optimal policy defined (7b under the reward parameter 0. By observing the objective :) we obtain the following result:
C(A, T) = E, (n.
rtt (r(s),
ac; a1;
((s a:)
Term Iı: independent of
E BpAw(1n (r(s), + U(s, + H(m()1ss)))
(84)
Term Ig function of the policy
Here, we can observe that the second term in (84) is exactly the objective in (76b). Moreover, since Tg is the optimal policy defined in 76b) under reward parameter 8, we obtain the result that
Tg arg min C(O,T)
which completes the proof to show the opitmality condition 80b Hence, we obtain that any stationary point á of the surrogate objective Σ(.,.) together with its optimal policy ga is saddle point of :.
Therefore, we complete the proof of Claim 1.
Next, show that for any saddle point of (77), its reward parameter is globally optimal solution to (76). We summarize this statement in the following claim:
Claim 2: For any saddle point (0, ma) the saddle point problem 77). the reward parameter Ø is
globally optimal solution to the surrogate objective î.()) in (76).
Here, start show the proof to Claim below.
Given saddle point (0, πT) of C(.--) defined in 777 we have the following property that
min max L(B, T) < max L(e, TT)
C(. TT)
min C(8, ④) < max min L(e, T) (85)
34

---

where (i) follows the optimality condition (0a)) and (ii) follows the optimality condition (80b).
According to the minimax inequality, we always have the following condition that
max min [(0, T) I min max ((0,
(86)
Through combining the saddle point inequality and the minimax inequality (86). we obtain the following equality:
min max L(0, xT) max C(0, mT) [(0, m²) min [(0, max min [(0, m))
(87)
Therefore, for any saddle point (0, ma), obtain the following property of the reward parameter 0 and the corresponding policy Tg-
arg max min C(0, x),
(88a)
TØ E arg min max L(O, n) 9
(88b)
Due the expression of the surrogate objective Z(-) in 76) and the objective [(;,-) in we have the following equality holds for any reward parameter 6:
?(0)= min [(0, m))
(89)
Combining (88a) and 199. obtain the following result:
arg max min L(O, x) arg max T.(0).
(90)
According to the property we shown 90). we obtain that for any saddle point (0, ma) of C(,,
:), the
reward parameter is globally optimal solution of the surrogate objective L(-) in (76)
Therefore, complete the entire proof of Claim 2.
Through combining Claim Claim 2, we obtain that when the reward function is linearly parameterized, any stationary point of the surrogate problem (76) is a global optimum.
Let's denote the optimal reward parameters associated with L(-) and î(-) as below, respectively:
l E arg max L(0); i E arg max (()).
(91)
0
Recall that in Theorem I. we have shown for any E (0, suppose there are more than N data and the number of transition dataset 'D satisfies:
points on each state-action pair (s,
472 C c² |1| S
[2]]
# transition samples 2 |2| N
In
(92)
2g2
where e is a constant dependent on With probability greater than 8, 8, the following result holds:
L(O") - L(0) Se.
Through combining Claim 1 Claim 2, we have already shown that any stationary point Ø is a global optimum of the surrogate problem (76) when the reward function is linearly parameterized. Therefore, we obtain that when the number of transition samples satisfies
(92). any stationary point Ó of the surrogate problem (76) an epsilon-optimal solution to the maximum likelihood estimation problem
(2). For any stationary point of the surrogate problem (76), with probability greater than holds that
L(0 - L(Ø) S E.
This completes the entire proof of Theorem g
35


Reviews:
[{"review_id": "5QmKlfnOBK", "reviewer": "Reviewer_5szs", "summary": "The paper \"Understanding Expertise through Demonstrations: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning\" proposes an innovative approach of offline inverse reinforcement learning. After a deep theoretical analysis of the inter-dependence between errors arising from dynamics modeling from limited offline data and performance gaps of resulting policies, authors propose an efficient algorithm to practically exploit conclusions for reward/policy learning. A small experimental section validates the approach.    ", "questions": "- Could authors give more insights about U, both theoretically if possible, and by some experimental results that show its impact ? ", "limitations": ". ", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Very well written with every notation well defined and every choice well justified\n- Very interesting problem and strong theoretical analysis\n- A practical algorithm that looks easy to reproduce\n- Good results ", "weaknesses": "- My main concern is that there is very few discussion about model uncertainty U in the paper, and particularly in section 3. I am surprised to not see it involved in the derivations and bounds, with no assumptions about it (except that it is bounded). No real meaning is given to it and it seems that it could be removed without changing anything in the theoretical conclusions. Is it true ? If yes, why introducing it in that part ? Also its impact is therefore not well understood from the theoretical analysis, which is a little be limitative to me (as it looks to have importance). \n- Still on U, I feel that experimental results on the choice of U would have been very useful.      ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 5szs for your positive comments and recognizing the importance of this work. Below, we address the reviewer's comments in a point-by-point manner.\n\n**Our Response to Weakness 1:** We appreciate the reviewer for raising this insightful question. In our practical implementation, we utilize a set of ensemble models to construct the penalty function. It's important to note that the penalty function is based on heuristic design and relies solely on the estimated dynamics model. Consequently, once the construction of the estimated dynamics model is complete, the associated penalty $U(s,a)$ for each state-action pair $(s,a)$ becomes a fixed constant when using the proposed alternating algorithm for reward/policy update steps. It is essential to differentiate our approach from model-based offline reinforcement learning, where the reward function remains fixed and the penalty function plays a crucial role in alleviating distribution shifts. In contrast, in IRL, the parameterized reward function is kept adjusted and optimized to align with expert demonstrations. Thus, the penalty function $U(s,a)$, being a fixed regularization term added to the parameterized reward function, does not significantly impact the theoretical analysis of our offline IRL method. This characteristic makes model-based offline IRL less sensitive to the construction of the penalty function U, as compared to model-based offline reinforcement learning. \n\nEmpirically, we've observed that the uncertainty-based penalty function effectively mitigates distribution shift effects during policy optimization in the estimated dynamics model, consistent with [R1]. In offline IRL settings, where a suitable initialization for the reward function may be lacking, the agent might suffer from distribution shifts in the initial training stages. Including the penalty function to regularize the agent's behavior and guide it to remain in the low-certainty region can facilitate the imitation of expert demonstrations. Furthermore, since the policy optimization subroutine in our proposed algorithm utilizes the practical implementations of model-based RL methods that explicitly incorporates the penalty function, it is natural to include the penalty function in our approach and theoretical analysis. \n\n[R1] Yu, Tianhe, et al. \"Mopo: Model-based offline policy optimization.\" Advances in Neural Information Processing Systems 33 (2020): 14129-14142.\n\n**Our Response to Weakness 2:** We appreciate the reviewer for raising this insightful question. To address the reviewer's question, we provide a supplementary experiment on three different choices of U. Below, we will elaborate the experiment details.\n\nTo estimate the model uncertainty by ensembles models, we have independently trained an ensemble of $N$ estimated dynamics models $\\\\{ P\\_{\\phi,\\varphi}^i(s\\_{t+1}|s_t,a_t)=\\mathcal{N}(\\mu^i\\_{\\phi}(s_t,a_t), \\Sigma^i\\_{\\varphi}(s_t,a_t)) \\\\}\\_{i=1}^N$ via likelihood maximization over transition samples. Here, each model estimates the location of the next state by gaussian distributions. Three common choices of the penalty function have been considered: 1) Max Aleatoric: $U(s,a)=-\\max\\_{i = 1,\\cdots,N} || \\Sigma^i\\_{\\varphi}(s,a) ||_F$, 2) Ensemble Variance: $U(s,a) = -(\\frac{1}{N} \\sum\\_{i=1}^N (\\Sigma^i\\_{\\varphi}(s,a))^2 + \\frac{1}{N} \\sum\\_{i=1}^N (\\mu^i\\_{\\phi}(s,a))^2 - (\\bar{\\mu}(s,a))^2)$ where $\\bar{\\mu}(s,a) = \\frac{1}{N}\\sum\\_{i=1}^N \\mu^i\\_{\\phi}(s,a)$, 3) Ensemble Standard Deviation: $U(s,a) = -\\sqrt{\\frac{1}{N} \\sum\\_{i=1}^N (\\Sigma^i\\_{\\varphi}(s,a))^2 + \\frac{1}{N} \\sum\\_{i=1}^N (\\mu^i\\_{\\phi}(s,a))^2 - (\\bar{\\mu}(s,a))^2}.$ We evaluate the effect of the three penalty functions in our proposed algorithm on HalfCheetah. We provide 10 expert trajectories and utilize three transition datasets (medium-expert, medium, medium-play) from D4RL. We show the numerical results in the following table:\n\n| HalfCheetah | Ensemble Standard Deviation | Ensemble Variance | Max Aleatoric |  \n|----------|----------|----------|----------|  \n| Medium-Expert | $11137.21 \\pm 872.47$ | $10941.80 \\pm 878.90$ | $10752.50 \\pm 655.96$ |  \n| Medium-Replay | $8214.46 \\pm 491.64$ | $8142.77 \\pm 621.12$ | $8612.29 \\pm 108.25$ |  \n| Medium | $6324.95 \\pm 556.42$ | $6454.43 \\pm 664.95$ | $7973.86 \\pm 108.86$ |\n\nAccording to our supplementary experiments, we observe that these three choices of penalty functions lead to similar final performance in our offline IRL method.\n\n**Our Response to Question 1:** We thank the reviewer for raising this insightful question. As we clarified in the response to Weakness 1, the penalty $U(s,a)$ is a fixed regularization term added on the reward function $r(s,a;\\theta)$ and the reward function $r(s,a;\\theta)$ will be updated every reward optimization step. Given that the reward function will be updated to align with expert demonstrations, the theoretical analysis will not heavily rely on the (fixed) penalty function. \n\nTo better understand the effect of the penalty function, we provide a supplementary experiment on three different choices of U. We observe the final performance of the proposed offline IRL algorithm with different choices of penalty function will not deviate from each other by too much, since the penalty function represents a constant regularization term while the magnitude of the parameterized reward function will continuously update during the reward learning process.. \n\nWe also note that the uncertainty estimation is still an active research problem and the theoretical understanding is limited, especially in the context of neural networks. Hence, we utilize common heuristics design of the penalty function from the literature in model-based offline RL to develop our algorithm. We would like to thank the reviewer’s insightful comments again and we will consider developin theoretical understanding on the impact of penalty function in model-based offline IRL as one of the directions for future work. \n"}, {"review_id": "m3KTF6sHbe", "reviewer": "Reviewer_VBZK", "summary": "In this paper the authors present a two level maximum likelihood based framework for offline inverse reinforcement learning, where both a world model and a reward model are learnt from expert demonstrations. In this two level algorithm the outer level or loop involves estimating the reward function, while the inner loop estimates the optimal policy for the chosen reward function in a conservative MDP setting, where a penalty is added which is loosely proportional to the uncertainty in the learnt model. The authors provide theoretical guarantees for the performance of their algorithm under fairly standard technical assumptions. They also show the numerical performance of their algorithm on 3 MuJoCo environments, comparing them with other state of the art offline RL algorithms.", "questions": "1. How is the state action visitation measure defined in Eq (5)? Specifically, I did not understand the term $P(S_t = s|s_0 \\sim \\eta)$. Does this imply that the transitions are action independent?\n2. In the proof of Lemmma 1, how is the equation below the one labelled (iii) obtained?\n3. How is the last step in Eq(30) in Proof of Lemma 1 obtained?\n4. [Typo] Line 806 \"bonded\" -> \"bounded\".\n5. I have understood most of the proofs as they are well written with adequate justification, but since I am not sure about the definition of the state-action visitation measure, I was unable to verify some proofs in the supplementary material.", "limitations": "The authors address the limitations of this work and also some suggestions to overcome some of these limitations.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is novel, clearly written and is easy to comprehend.\n2. The authors have stated their results formally in the form of Lemmas and Theorems and have proved them in the supplementary material. This analysis proves the validity and utility of their proposed approach.\n3. While model based offline inverse RL has been studied, I think the theoretical guarantees from this paper are novel and important.", "weaknesses": "1. The authors have demonstrated performance on only 3 environments, in which in one of the cases, their proposed algorithm is not the best.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review of the paper and the valuable feedback. Below, we address the reviewer's questions in a point-by-point manner.\n\n**Response to Question 1:** We appreciate this question raised by the reviewer. This question indeed helps us find a typo in the term $P(s_t=s|s_0\\sim\\eta)$, which should be corrected as $P^\\pi(s_t=s|s_0\\sim\\eta)$. \n\nUnder any fixed policy $\\pi(a|s)$ and transition function $P(s^\\prime | s,a)$, the term $P^\\pi(s_t=s|s_0\\sim\\eta)$ denotes the probability that $s_t = s$ at time step t when $s_0$ is sampled from the initial state distribution $\\eta(\\cdot)$ and the actions $\\{a_0 , a_1, …, a_{t-1}\\}$ in the Markov decision process are sampled from the policy $\\pi$. \n\nTo avoid potential confusion, under any policy $\\pi$ and transition function $P$, let us correct the typo and rewrite the definition of the corresponding state-action visitation measure $d^\\pi\\_{P}(s,a)$ as below: $d^\\pi\\_{P}(s,a) := (1 - \\gamma) \\pi(a|s) \\sum\\_{t=0}^{\\infty} \\gamma^t P^\\pi(s_t = s | s_0 \\sim \\eta)$.\n\nHence, in Eq (5), the state-action visitation measure under the expert policy $\\pi^E$ is defined as below:\n$d^E(s,a) := (1 - \\gamma)  \\pi(a|s) \\sum\\_{t=0}^{\\infty} \\gamma^t P^{\\pi^E}(s_t = s | s_0 \\sim \\eta  )$.\n\n**Response to Question 2:** In (iii), we have shown that the objective function $L(\\theta)$ can be expressed as $L(\\theta)=\\sum\\_{t=0}^{\\infty} \\gamma^t \\mathbb{E}_{(s_t,a_t)\\sim(\\eta,\\pi^E,P)}[ r(s_t,a_t;\\theta) + U(s_t,a_t) + \\gamma \\mathbb{E}\\_{s\\_{t+1} \\sim \\hat{P}(\\cdot|s_t,a_t) }[V\\_{\\theta}(s\\_{t+1})] ] - \\sum\\_{t=0}^{\\infty}\\gamma^t \\mathbb{E}\\_{s_t \\sim (\\eta,\\pi^E,P)}[V\\_{\\theta}(s_t)].$   \n\nBased on the equation in (iii), we can further write down the following equality: $L(\\theta)=(\\sum\\_{t=0}^{\\infty} \\gamma^t \\mathbb{E}_{(s_t,a_t)\\sim(\\eta,\\pi^E,P)}[ r(s_t,a_t;\\theta) + U(s_t,a_t) ]  + \\sum\\_{t=0}^{\\infty} \\gamma^{t+1} \\mathbb{E}\\_{(s_t,a_t)\\sim(\\eta,\\pi^E,P),s\\_{t+1} \\sim \\hat{P}(\\cdot|s_t,a_t)}[V\\_{\\theta}(s\\_{t+1})] ) - (\\mathbb{E}\\_{s_0 \\sim \\eta(\\cdot)}[V\\_{\\theta}(s_0)] + \\sum\\_{t=0}^{\\infty} \\gamma^{t+1} \\mathbb{E}\\_{s\\_{t+1} \\sim (\\eta,\\pi^E,P)}[V\\_{\\theta}(s\\_{t+1})] ).$\n\nThen it leads to the equality below the one labeled (iii) by interchanging terms.\n\n**Response to Question 3:**  In the first equation of eq. (30), we have obtained the following equation of the term T2:\n$ T_2 =  \\sum\\_{t=0}^{\\infty} \\gamma^{t+1} \\mathbb{E}_{(s_t,a_t)\\sim(\\eta,\\pi^E,P)}[\\sum\\_{s\\_{t+1} \\sim \\mathcal{S}} V\\_{\\theta}(s\\_{t+1}) (\\hat{P}(s\\_{t+1} | s_t, a_t) - P(s\\_{t+1} | s_t, a_t))]$\n\nHere, recall that we have defined $P^{\\pi^E}(s_t=s|s_0 \\sim \\eta)$ as the probability that $s_t=s$ when $s_0$ is sampled from the initial state distribution $\\eta(\\cdot)$ and the actions in the MDP are sampled from the expert policy $\\pi^E$. Hence, we can obtain the second equality in (30): $T_2 = \\gamma \\sum\\_{t=0}^{\\infty} \\sum\\_{s \\in \\mathcal{S}, a\\in \\mathcal{A}} \\gamma^t P^{\\pi^E}(s_t = s|s_0\\sim \\eta) \\pi^E(a_t = a|s_t = s)( \\sum\\_{s^\\prime \\in \\mathcal{S}} V\\_{\\theta}(s^{\\prime}) (\\hat{P}(s^{\\prime} | s_t=s, a_t=a) - P(s^{\\prime} | s_t=s, a_t=a)) ).$\n\nThen we can further express the term $T_2$ as below:\n$T_2 = \\gamma \\sum\\_{s \\in \\mathcal{S}, a\\in \\mathcal{A}} ( \\pi^E(a|s) \\sum\\_{t=0}^{\\infty} \\gamma^t P^{\\pi^E}(s_t = s|s_0 \\sim \\eta)) \\cdot ( \\sum\\_{s^\\prime \\in \\mathcal{S}} V\\_{\\theta}(s^{\\prime}) (\\hat{P}(s^{\\prime} | s, a) - P(s^{\\prime} | s, a)) )$.\n\nGiven the expert policy $\\pi^E$, recall that we have defined the corresponding state-action visitation measure as $d^E(s,a) :=(1-\\gamma)\\pi^E(a|s) \\sum\\_{t=0}^{\\infty} \\gamma^t P^{\\pi^E}(s_t =s|s_0 \\sim \\eta).$\nThen we obtain the following expression of the term $T_2$ as below:\n$T_2 = \\frac{\\gamma}{1 - \\gamma} \\sum\\_{s \\in \\mathcal{S}, a\\in \\mathcal{A}} d^E(s,a) \\cdot ( \\sum\\_{s^\\prime \\in \\mathcal{S}} V\\_{\\theta}(s^{\\prime}) (\\hat{P}(s^{\\prime} | s, a) - P(s^{\\prime} | s, a)) ).$\n\nFinally, we can show the last equality in eq (30):\n$T_2 = \\frac{\\gamma}{1 - \\gamma} \\mathbb{E}\\_{(s,a)\\sim d^E(\\cdot, \\cdot)}[ \\sum\\_{s^\\prime \\in \\mathcal{S}} V\\_{\\theta}(s^{\\prime}) (\\hat{P}(s^{\\prime} | s, a) - P(s^{\\prime} | s, a))  ].$\n\n**Response to Question 4:** We thank the reviewer for pointing out this typo. We will correct it in our paper.\n\n**Response to Question 5:** We thank the reviewer for raising the question on the definition of the state-action visitation measure. The question indeed helps us find a typo in the definition of the state-action visitation measure. As we clarified in the response to Question 1, under any policy $\\pi$ and transition function $P$, the corresponding state-action visitation measure $d^{\\pi}\\_{P}(s,a)$ is defined as below: $d^{\\pi}\\_{P}(s,a) := (1 - \\gamma) \\pi(a|s) \\sum\\_{t=0}^{\\infty} P^\\pi(s_t = s | s_0 \\sim \\eta)$.\nHere, $P^\\pi(s_t=s|s_0\\sim\\eta)$ denotes the probability that $s_t = s$ when the actions in the MDP are sampled from the policy $\\pi$ and the initial state $s_0$ is sampled from the initial state distribution $\\eta(\\cdot)$.\n\nWe hope our correction of the typo and the corrected definition of the state-action visitation measure can address the reviewer’s question, and can improve the readability of our paper."}, {"review_id": "vRwVqoq1cK", "reviewer": "Reviewer_2Qwv", "summary": "Offline inverse reinforcement learning (IRL) is a method for finding an unknown reward function optimized by an agent from demonstrations using only a finite dataset. The most common framework is maximum entropy (MaxEnt) IRL, which attempts to find a reward function that induces a policy which achieves the same expected reward as the trajectories from the expert demonstrations while maximizing its entropy. Prior work has shown that this is equivalent to finding a policy which maximizes the likelihood of the demonstrations under the constraint that this policy comes from solving a MaxEnt RL problem. This formulation as a bi-level optimization problem reduces the computational burden that results from alternating between finding the policy and updating the reward. However, it requires access to the true dynamics of the environment, which is incompatible with the offline IRL setup. Instead, this work proposes to learn the dynamics model in an uncertainty-aware fashion and incorporate a measure of this uncertainty in the learned reward function. This results in a two-stage procedure: 1) fitting a dynamics model from transition samples in the dataset and 2) recovering the reward function using the maximum likelihood (ML) formulation of the IRL problem. To perform the second step, the authors propose a novel decomposition of the upper-level objective, which consists of a surrogate objective that is more computationally tractable to optimize. The authors provide statistical guarantees about the optimality of the recovered reward function in terms of dataset coverage, a concept common in offline RL. Importantly, their bounds depend on dataset coverage on expert-visited state-action pairs, not the full joint space.", "questions": "- Is there a difference in the alternating optimization scheme presented here and the one in [21]?\n- How does Theorem 2 in this text relate to Theorem 5.4 in [21]?\n- Am I correct in assuming that the three different datasets are used to evaluate how the coverage of the dataset around expert state-action pairs affects the algorithm's performance?", "limitations": "There is no discussion of limitations in the paper. The paper would be made a lot stronger if this was discussed in a conclusion section.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Offline inverse RL is an important area for tackling challenging sequential decision making problems in potentially safety-critical applications. \n- The paper is well organized and clearly written. It does a good job explaining the novelty and results and provides enough information to support its claims.\n- The paper presents an extensive experimental evaluation on several benchmarks, comparing to both model-based and model-free offline IRL algorithms and existing imitation learning approaches. Their algorithm outperforms these baselines in most cases.\n- The authors provide a nice analysis of surrogate objective and its relation to the true upper-level objective. This provides a nice motivation for optimizing the surrogate instead, which is more computationally tractable.\n- The authors present a nice optimality guarantee of the stationary point of their algorithm in terms of the surrogate objective in the case where the reward function is linear in a feature vector of states and actions. They also relate this stationary point to the true optimal solution of the original problem.\n- The additional reward transfer experiments indicate that the learned reward function may transfer to dynamics models trained on different state-action distributions. This appears to hold even if the state-action coverage used to train the reward function is close to expert-visited states.", "weaknesses": "- The alternating optimization scheme discussed in this work appears to be identical to that presented in [21]. If true, that is fine, as the main contribution of the work lies in modeling the conservative MDP and providing novel bounds in the offline setting. However, it should be made explicit in the paper and mentioned in the contributions.\n- Theorem 2 seems very similar to Theorem 5.4 in [21], except that the Q function approximation error is considered explicitly. If this is true, it should be discussed that this is the novelty in the text.\n- Section 6 should discuss the differences in the three dataset types used, as the current text does not explain what they entail. This makes it difficult to understand the performance of the proposed algorithm in each setting without carefully looking at the Appendix. It should also talk about the purpose of using these different datasets. From the Appendix, it appears that they are only used to train the dynamics model. Thus, they are evaluating the effect of dataset coverage around the expert on performance. This should be explicitly discussed in the paper. I know space is limited, but these are important details that should be in the main text.\n- A minor comment is that it would be nice for the main paper to end with a conclusion section rather than ending abruptly. And this conclusion should mention limitations of the current method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review of the paper and the valuable feedback. Below, we address the reviewer's comments in a point-by-point manner.\n\n**Response to Weakness 1&Question 1:** We appreciate the reviewer’s comments. However, we do not agree that the proposed alternating optimization algorithm in this work is identical to the one presented in [21]. In this response, we would like to highlight the key difference between our proposed algorithm and the one presented in [21]. \n\nCompared with the online IRL algorithm presented in [21], one key difference is that our proposed offline IRL algorithm solves a **much harder** IRL problem (2a) - (2b) than the one in [21]. This is due to the fact that our outer objective $L(\\theta)$ in eq (2a) is defined on the *ground-truth* dynamics model P while the inner problem in eq (2b) considers policy optimization in the *estimated* dynamics model $\\hat{P}$.  Due to the existence of the dynamics model mismatch, we turn to optimize a novel surrogate objective by developing a *new* algorithm. \n\nTo be more specific, compared with the existing algorithm in [21], there are a few major differences in our proposed algorithm. Indeed, as reviewer has mentioned, we adopt a generic “alternating optimization” scheme, which alternates between one conservative policy improvement step and one reward optimization step. However, the algorithm under the hood of “alternating optimization” is very different. \n* First, compared with the existing algorithm in [21], the proposed algorithm in paper aims to solve a completely different problem – offline IRL. Due to the difference in problem settings and formulations, we need to analyze the gradient expression of the surrogate objective function w.r.t. the reward parameter, which leads to a different reward update step compared to the one in [21]. \n\n* Second, for the policy optimization under each reward estimator, we consider updating the policy under the estimated dynamics model with a regularization term (penalty function) based on the model uncertainty. In contrast, the existing algorithm in [21] only solves the online IRL problem where there is no estimated dynamics model and uncertainty estimation. This difference in algorithm design also leads to new algorithm implementations where we solve the underlying optimal policy in the estimated dynamics model through taking advantage of the recent advances in model-based offline policy optimization and uncertainty estimation. \n\n* Third, in the policy optimization subroutine (13) - (14), we consider a more realistic scheme compared with [21], where we first approximate the current soft Q function $Q_k$ by $\\widehat{Q}_k$ in (13) and then update the policy by an approximate soft policy iteration in (14). Compared with the existing algorithm in [21] which simply assumes the soft Q-function can be accurately estimated without any approximation error, the proposed algorithm in our paper is more practical since the approximation error has been explicitly taken into account. \n\n**Response to Weakness 2&Question 2:** We appreciate the reviewer’s comments. In this response, we would like to highlight the key differences between Theorem 2 in this work and Theorem 5.4 in [21]. \n\nIn the convergence analysis of our proposed algorithm, the previous analysis in [21] does not hold anymore. This is due to the fact that we are optimizing a *surrogate objective function* which is different from [21]. Moreover, as opposed to the analysis considered in [21], the analysis of our proposed algorithm involves two dynamics models: the ground-truth dynamics model $P$ and the estimated dynamics model $\\hat{P}$. Due to the existence of the dynamic model mismatch and regularized penalty function in the setting of offline IRL, it is not clear whether previous properties, such as the Lipschitz continuity proved in [21] can still hold here. Furthermore, as shown in eq (13) - (14) which is different from [21], we do not assume that the soft Q function $Q_k$ can be accurately approximated at each step. In the convergence analysis of [21], the accurate approximation of the soft Q function at each step can guarantee a monotonic improvement for each soft policy iteration step. However, in our work, the approximation error between $\\widehat{Q}_k$ and $Q_k$ in (13) - (14) also makes it more challenging to analyze the stability of our proposed alternating algorithm. To tackle this problem, we develop new proof techniques in the proof of Theorem 2 to guarantee the finite-time convergence analysis of our proposed algorithm.\n\nWe appreciate the reviewer’s comments, which encourages us to discuss the key difference in proof techniques compared with [21]. We will include the discussion above into our paper.\n\n**Response to Weakness 3:** We appreciate the reviewer’s comments. These transition datasets are used to train the dynamics model and further evaluate the effect of the dataset’s coverage over the expert-visited state-action space on the performance of benchmark algorithms. We will explicitly include these details and move the discussion in Appendix into the main paper.\n\n**Response to Weakness 4:** We appreciate the reviewer’s suggestion. We present a conclusion and discuss the limitations in the global response and we kindly refer the reviewer to check it (see https://openreview.net/forum?id=oML3v2cFg2&noteId=oCZvqhc8PI).\n\n**Response to Question 3:** Yes. Indeed, the three types of datasets are gathered from policies with varying performance levels and exhibit different coverage across the expert-visited state-action space. We leverage these three dataset types to assess how their coverage of the expert-visited state-action space influences the algorithm's performance.\n\n**Response to Limitations:** We would like to kindly remind the reviewer that the limitations of this work have been discussed in the section of Limitations and broader impacts in Appendix. We will discuss it in the conclusion section."}, {"review_id": "pBciLwhSKl", "reviewer": "Reviewer_1VXD", "summary": "This paper addressed the issue of covariate shift in offline imitation learning. The authors extended the uncertainty-regularized model-based offline RL to the imitation learning setting. The key idea is to first learn transition dynamics from samples, and then solve an optimization problem that jointly seeks a policy such that it optimizes the learned transition dynamics accompanied by a reward model and maximizes the log-likelihood of actions in data. The authors provide theoretical guarantees for the maximization of action log-likelihood and empirical results for the learned policy.", "questions": "1. Why do you include discounting in the log-likelihood term?\n2. Why do you include the entropy term in eq. 3a?\n3. The theoretical analysis relies on the relation between optimal policy and optimal soft Q-function (eq.4 and eq. 14). But as mentioned in line 226--230, practical implementations utilize an actor-critic framework to approximate the optimal policy. How does this approximation affect the analysis?\n4. Eq. 15 shows a clear resemblance with the reward maximization part in max-entropy IRL (eq. (1)). It suggests that the proposed algorithm seems to replace the online interaction of max-entropy IRL with sample generation using the learned dynamics. Then, what is the motivation to maximize the log-likelihood of expert actions?\n\nA suggestion.\nOne concern of this approach is that to estimate the transition dynamics we need a sufficient amount of data, but the covariate shift issue occurs when we only have a small amount of data. I would suggest the authors include results on a few expert demonstrations (e.g. 10~100) to verify their claim.", "limitations": "No. there is no discussion on limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The issue of covariate shift is indeed important for offline IRL.\n2. The efficacy of the algorithm is partially supported by empirical results.\n3. Analysis is provided for the model-based part of this algorithm.", "weaknesses": "1. The paper is somewhat hard to follow. See questions below.\n2. The effect of overcoming the distributional shift is not emphasized in the experiment section. None of the experiments was carried out on small datasets where the coverage of state–action space is limited. In fact, the medium datasets in D4RL contain 1M transitions, and the medium-expert versions contain 2M transitions. The datasets for results in Figure 2 contain 5000 expert demonstrations, which might correspond to 5M transitions if each expert trajectory contains 1000 transitions.\n3. The paper does not have an informative conclusion part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review of the paper and the valuable feedback. Below, we address the reviewer's comments in a point-by-point manner.\n\n**Response to Weakness 1:** We will address your comments to improve the readability of the paper. \n\n**Response to Weakness 2:** We appreciate the reviewer’s comments. In our paper, in fact, we have utilized small datasets to evaluate the performance of our proposed algorithm when the coverage of the expert-visited state-action space is limited. To address some potential misunderstanding regarding our experiment, we would like to clarify that **in our work, one expert demonstration corresponds to one transition sample $(s, a, s^\\prime)$ from the expert trajectories.** We have explicitly mentioned this in Appendix A and the caption of Table 2.  In Fig. 2 -4, we respectively report the performance of our proposed algorithm when five trajectories (5,000 transition samples), one trajectory (1,000 transition samples) and ten trajectories (10,000 transition samples) are available.\n\n**Response to Weakness 3:** We appreciate the reviewer’s suggestion. We provide a conclusion section in the global response and we kindly refer the reviewer to check it (see https://openreview.net/forum?id=oML3v2cFg2&noteId=oCZvqhc8PI).\n\n**Response to Question 1:** We appreciate the reviewer for raising this insightful question. The general answer that the discounted log-likelihood objective is derived from the dual problem of the classic formulation maximum entropy (MaxEnt) IRL.\n\nIn the literature of MaxEnt IRL [1], the problem aims to learn a (linearly parameterized) reward that can induce a policy to achieve the same expected reward as the expert trajectories while maximizing its entropy. Under infinite horizon MDP, the existing results have shown that the dual problem of MaxEnt IRL is the maximization of the “discounted” log-likelihood over expert trajectories on an optimal policy which solves an underlying entropy-regularized MDP (see Theorem 1 in [2]). This fundamental result justifies the use of discounting in the log-likelihood term.\n\n**Response to Question 2:** The general answer is that the entropy term in our formulation eq. (2b) is translated from the maximum entropy objective of MaxEnt IRL.\n\nAs we discussed in the response to Question 1, in online IRL with linear parameterized reward, it has been shown that the dual problem of MaxEnt IRL is a maximum likelihood formulation of IRL where the optimal policy is modeled as a solution to a entropy-regularized MDP. Here, in the derivation of maximum likelihood formulation of IRL, the entropy term in the objective of MaxEnt IRL has been translated to the model of agent’s behavior. Hence, in this paper, when we propose the maximum likelihood formulation of IRL with nonlinear reward parameterization in the offline setting, we include the entropy term in the definition of the soft value function and soft Q-function in eq. 3a - 3b.\n\n**Response to Question 3:** We would like to clarify that our theoretical analysis aligns with the algorithmic framework of our practical implementations, since we have taken the approximation error into account in our theoretical analysis. \n\nMore specifically, we have considered the approximation error from the soft actor-critic (SAC) algorithm [3] into our policy optimization subroutine eq. 13 - 14. Our theoretical analysis does not require the policy or the soft Q-function to be optimal at each step, therefore, running a few steps of the soft actor-critic algorithm will be sufficient. At each step k, we first run policy evaluation steps (critic steps) to approximate the corresponding soft Q-function $Q_k$ of the current policy $\\pi_k$ by $\\hat{Q}\\_k$, as shown in eq. 13. Then we run the soft policy iteration (action steps) to obtain the updated policy $\\pi_{k+1}(a|s) \\propto \\exp \\hat{Q}_k(s,a)$, as shown in eq. 14. Note that our policy optimization subroutine matches the practical implementation of SAC. Moreover, the approximation error $||\\hat{Q}_k - Q_k ||\\_{\\infty}$ in estimating the soft Q-function has been explicitly considered in our theoretical analysis (see Theorem 2).\n\n**Response to Question 4:** The general answer is that the maximum likelihood formulation is a broader problem formulation compared with MaxEnt IRL. First of all, as we discussed in the response to Question 1, the MaxEnt IRL is based on the online IRL setting with linear reward. In this case, the maximum likelihood IRL is the dual problem to MaxEnt IRL. Hence, we can find resemblance between the maximum likelihood IRL and MaxEnt IRL. However, since MaxEnt IRL is limited to the online setting with linear reward, its formulation is incompatible with broader problems. Therefore, the maximum likelihood IRL formulation offers greater flexibility to model IRL problems in a broader scope. It can be applied effectively to scenarios involving either linear or nonlinear reward parameterization, as well as online or offline settings.\n\n\n**Response to Reviewer's Suggestion:** We appreciate the reviewer’s suggestion. As we clarified in our response to Weakness 2, one expert demonstration in our paper means one transition sample $(s, a, s^\\prime)$ collected from an expert trajectory. In our numerical results, we have considered the setting that only 1 / 5 / 10 expert trajectories are available. \n\n**Response to Limitations:** We have include them in the section of Limitations and broader impacts in Appendix.\n______\n[1] Ziebart, Brian D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.\n\n[2] Zeng, Siliang, et al. \"Maximum-likelihood inverse reinforcement learning with finite-time guarantees.\" Advances in Neural Information Processing Systems 35 (2022): 10122-10135.\n\n[3] Haarnoja, Tuomas, et al. \"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.\" International conference on machine learning. PMLR, 2018.\n"}, {"review_id": "MyeWRBtLwm", "reviewer": "Reviewer_VRGm", "summary": "This paper focuses on learning from demonstration via offline inverse reinforcement learning (IRL). Offline IRL suffers a similar problem as offline reinforcement learning (RL) and imitation learning (IL), where the policy cannot generalize well on unseen states and actions---this problem is known as distribution shift. To address this problem, this paper proposes to first learn a dynamic model, and formulates a maximum likelihood (ML) objective to simultaneously recovers both the reward function and the policy. Notably, the policy is optimized using a maximum entropy objective along with pessimism based on the uncertainty of the learned dynamic model. This paper provides PAC-style bounds to quantify the amount of samples required to achieve $\\varepsilon$-optimal solution to the MLE objective. The paper further provides an algorithm that obtains such a $\\varepsilon$-optimal solution under specific assumptions. Finally, this paper provides empirical evaluation on the D4RL benchmarks.\n\n## Contributions\n- A new MLE objective for recovering a policy that is close to the expert policy.\n- A theoretical analysis that describes the statistical guarantees of the objective\n- An algorithm that obtains to near-optimal solution under linear parameterization of the reward function\n- An empirical evaluation on standard benchmark, D4RL, that indicates statistically significant improvement over existing baselines in majority of the tasks.", "questions": "- On page 3, equation 2a: $\\theta$ parameterizes only the reward function, and $\\pi_\\theta$ corresponds to the policy obtained when $r$ is parameterized by $\\theta$, correct?\n- On page 5, proposition 1: Is there any result regarding larger $\\varepsilon$? It seems like we may want to sacrifice this approximation error.\n\n## Possible Typos\n- Page 3, line 93: gamma \\in (0, 1)?\n- Page 3, line 95: The $P$ should not be the same as the transition dynamics right? I feel we should be clear that we are either overloading notation or use another symbol.\n- Page 5, equation 11: What's $\\epsilon$? Is it the same as $\\varepsilon$?\n- Page 8, line 303: Missing space after \"2)\"", "limitations": "- The paper's proposed method requires uncertainty estimation which is still an active research problem, especially in the context of neural networks---as far as I understand the paper leverages existing work that lacks theoretical guarantee.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper is well written and easy to follow in general---I particularly appreciate the presentation on providing formal statements followed by the high-level intuitions.\n- The paper proposes a novel formulation for offline inverse reinforcement learning.\n- The paper provides numerous theoretical justifications and an algorithm that is inspired by said analyses.", "weaknesses": "- In practice, how do we ensure assumption 2 (ergodicity)? It seems like this assumption actually \"hides\" some part of the coverage requirement?\n- I am curious as to how the MLE objective connects to the policy error---I completely understand that if $\\pi_\\theta = \\pi^E$ then the policy error is zero. However, it does not seem to me that achieving $\\varepsilon$-error on the MLE (i.e. $L(\\theta^*) - L(\\hat \\theta) \\leq \\varepsilon$) does not directly tell us the policy error.\n- I think the training description for BC is somewhat vague---on page 8, line 314: what exactly does it mean by \"train the algorithm until convergence\"? Do we have some form of validation checking for BC? [1, 2, 3] have results regarding how BC would perform based on specific validation. Secondly, was there any hyperparameter search on BC, ValueDICE, and CLARE?\n- Regarding the experiment on recovered rewards, what is the performance if we were to fix the reward to 0? Isn't it better if we were to consider the correlation between the true reward function and the obtained reward function?\n\n## References\n[1]: Hussenot, L., Andrychowicz, M., Vincent, D., Dadashi, R., Raichuk, A., Ramos, S., ... & Pietquin, O. (2021, July). Hyperparameter selection for imitation learning. In International Conference on Machine Learning (pp. 4511-4522). PMLR.  \n[2]: Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., ... & Martín-Martín, R. (2021). What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03298.  \n[3]: Ablett, T., Chan, B., & Kelly, J. (2023). Learning from Guided Play: Improving Exploration for Adversarial Imitation Learning with Simple Auxiliary Tasks. IEEE Robotics and Automation Letters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review of the paper and the valuable feedback. Below, we address the reviewer's comments in a point-by-point manner.\n\n**Response to Weakness 1:** We thank the reviewer for this good question. As we discussed in Section. 6, we constructed an ensemble of estimated dynamics models as $ \\\\{ \\hat{P}^{i}\\_{\\phi,\\varphi}(\\cdot|s,a) = \\mathcal{N}( \\mu^{i}\\_{\\phi}(s,a), \\Sigma^{i}\\_{\\varphi}(s,a) ) \\\\}\\_{i=1}^N$ where each one models the location of the next state by Gaussian distribution. Under Gaussian distributions, it is achievable to eventually get from every state to every other state with positive probability. Hence, the ergodicity holds in our estimated dynamics model in practice.\n\n**Response to Weakness 2:** We appreciate the reviewer for raising this insightful question. The general answer is that $\\varepsilon$-error on the MLE implies that the recovered policy $\\pi_{\\hat{\\theta}}$ is $\\varepsilon$-close to the expert policy $\\pi^E$ measured by the KL divergence.\n\nIn our formulation eq (2a) - (2b), the MLE objective follows $L(\\theta) := \\mathbb{E}\\_{\\tau^E \\sim (\\eta,\\pi^E,P)}[ \\sum\\_{t=0}^{\\infty} \\gamma^t \\log \\pi\\_{\\theta}(a_t|s_t) ]$. According to the definition of the state visitation measure $d^E(s)$ under the expert policy $\\pi^E$ where $d^E(s) := (1-\\gamma) \\sum\\_{t=0}^{\\infty} \\gamma^t P^{\\pi^E}(s_t = s|s_0 \\sim \\eta)$, we can rewrite the MLE objective as below: $L(\\theta) := \\mathbb{E}\\_{\\tau^E \\sim (\\eta,\\pi^E,P)}[ \\sum\\_{t=0}^{\\infty} \\gamma^t \\log \\pi\\_{\\theta}(a_t|s_t) ] = \\frac{1}{1-\\gamma}\\mathbb{E}\\_{s \\sim d^E(\\cdot), a \\sim \\pi^E(\\cdot|s)}[ \\log \\pi\\_{\\theta}(a|s) ].$\n\nTherefore, the $\\varepsilon$-error on the MLE implies that $L(\\theta^*) - L(\\hat{\\theta}) = \\frac{1}{1-\\gamma}\\mathbb{E}\\_{s\\sim d^E(\\cdot), a \\sim \\pi^E(\\cdot|s)}[\\log \\pi\\_{\\theta^*}(a|s) - \\log \\pi\\_{\\hat{\\theta}}(a|s)] = \\frac{1}{1-\\gamma}\\mathbb{E}\\_{s\\sim d^E(\\cdot), a \\sim \\pi^E(\\cdot|s)}[\\log  \\frac{\\pi\\_{\\theta^*}(a|s)}{\\log \\pi\\_{\\hat{\\theta}}(a|s)} ] \\leq \\varepsilon.$ When the expert trajectories are consistent with the optimal policy under a ground truth reward parameter $\\theta^*$, we have $\\pi^E = \\pi\\_{\\theta^*}$. Due to this property, we can show $L(\\theta^*) - L(\\hat{\\theta}) = \\frac{1}{1-\\gamma}\\mathbb{E}\\_{s\\sim d^E(\\cdot), a \\sim \\pi^E(\\cdot|s)}[\\log  \\frac{\\pi\\_{\\theta^*}(a|s)}{\\log \\pi\\_{\\hat{\\theta}}(a|s)} ] = \\frac{1}{1-\\gamma}\\mathbb{E}\\_{s\\sim d^E(\\cdot), a \\sim \\pi^E(\\cdot|s)}[\\log  \\frac{\\pi^E(a|s)}{\\log \\pi\\_{\\hat{\\theta}}(a|s)} ] = \\frac{1}{1-\\gamma}\\mathbb{E}\\_{s\\sim d^E(\\cdot)} [ D\\_{KL}( \\pi^E(\\cdot|s) || \\pi\\_{\\hat{\\theta}}(\\cdot|s) ) ] \\leq \\varepsilon.$ Hence, we can show the $\\varepsilon$-error on the MLE implies that the recovered policy $\\pi\\_{\\hat{\\theta}}$ is $\\varepsilon$-close to the expert policy $\\pi^E$ measured by the KL divergence.\n\n**Response to Weakness 3:** We appreciate the reviewer’s comments. Regarding the training of BC, we assess the checkpoints generated throughout the training process and monitor the performance of the resulting policies at every few updates. Once the performance of BC stops to achieve further improvement (in terms of the average rewards in episodes) within 20 training epochs, we will use the updated policy to generate ten rollout episodes and then record the average reward of the rollout episodes as the final performance measure. \n\nMoreover, it is important to note that for benchmark algorithms BC, ValueDice, and CLARE, we have utilized their official implementations, incorporating their default hyper-parameters, which have been fine-tuned beforehand.\n\nBelow, we provide the sources for the official implementations of the benchmark algorithms, which we have mentioned in Appendix A (the section of experiment details):\n\nBC, ValueDice: https://github.com/google-research/google-research/tree/master/value_dice\n\nCLARE: https://openreview.net/forum?id=5aT4ganOd98\n\n**Response to Weakness 4:** We appreciate this suggestion. The numerical results are included in the PDF in global response (see https://openreview.net/forum?id=oML3v2cFg2&noteId=oCZvqhc8PI).\n\n**Response to Question 1:** Yes. When the reward function is $r(\\cdot,\\cdot;\\theta)$, we use $\\pi_{\\theta}$ to denote the optimal policy obtained in the estimated dynamics model.\n\n**Response to Question 2:** We appreciate the reviewer for this good question. The reason we define $\\varepsilon \\in (0,2)$ is because we use $\\varepsilon$ to bound $\\mathbb{E}\\_{(s,a)\\sim d^E(\\cdot,\\cdot)}[|| P(\\cdot|s,a) -  \\hat{P}(\\cdot|s,a)||_1]$ and the maximum dynamics mismatch error under L1 norm is bounded by 2.\n\nBased on the definition of the L1 distance, we have \n$|| P(\\cdot | s,a) - \\hat{P}(\\cdot | s,a) ||_1 = \\sum\\_{s^\\prime \\in \\mathcal{S}} | P(s^\\prime | s,a) - \\hat{P}(s^\\prime | s,a) | \\in [0,2]$. When two distributions perfectly match, their L1 distance is 0, while a complete mismatch between distributions results in an L1 distance of 2.\n\nHence, when analyzing sample complexity, we define $\\varepsilon \\in (0,2)$ to bound the error $\\mathbb{E}\\_{(s,a)\\sim d^E(\\cdot,\\cdot)}[|| P(\\cdot|s,a) -  \\hat{P}(\\cdot|s,a)||_1]$.\n\n**Response to Typo 1:** We will explicitly define $\\gamma \\in (0,1)$ in our paper.\n\n**Response to Typo 2:** We appreciate the comments. There is one typo in the term $P(s_t = s | s_0 \\sim \\eta)$, which should be corrected as $P^{\\pi}(s_t = s | s_0 \\sim \\eta ).$ Under any fixed policy , the term $P^{\\pi}(s_t = s | s_0 \\sim \\eta )$ denotes the probability that $s_t = s$ at time step t when $s_0$ is sampled from $\\eta(\\cdot)$ and the actions in the MDP are sampled from $\\pi$. \n\nHence, the state-action visitation measure $d^{\\pi}\\_{P}(s,a)$ in line 95 follows: $d^{\\pi}_{P}(s,a):= (1 - \\gamma) \\pi(a|s) \\sum\\_{t=0}^{\\infty} \\gamma^t P^{\\pi}(s_t = s | s_0 \\sim \\eta ).$\n\n**Response to Typo 3:** We appreciate the comments. $\\epsilon$ is a typo and we should correct it as $\\varepsilon$.\n\n**Response to Typo 4:** We will include the space after \"2)\"."}]

Author rebuttal:
We thank the detailed review and comments from all reviewers.

In the global response, we present a conclusion here and we will include it into our final version:

In this paper, we model the offline Inverse Reinforcement Learning (IRL) problem from a maximum likelihood estimation perspective. We develop a computationally-efficient algorithm that effectively recovers the underlying reward function and its associated optimal policy. We have also established statistical and computational guarantees for the performance of the recovered reward estimator. Through extensive experiments, we demonstrate that our algorithm outperforms existing benchmarks for offline IRL and Imitation Learning, especially on high-dimensional robotics control tasks. One limitation of our method is that we focus solely on aligning with expert demonstrations during the reward learning process. In an ideal scenario, reward learning should incorporate diverse metrics and data sources, such as expert demonstrations and preferences gathered through human feedback. One direction for future work is to broaden our algorithm framework and theoretical analysis for a wider scope in reward learning.



Meta-review: The submitted paper was reviewed by 5 knowledgeable reviewers, all of whom recommended acceptance of the paper. Several initial concerns/questions were answered convincingly by the authors in their rebuttal and discussions with the reviewers. Hence, in line with the reviewers' recommendations, I am recommending acceptance of the paper. The authors are encouraged to further improve their paper by incorporating the feedback of the reviewers and relevant parts of the discussions with them in the final version of their paper.
Decision: Accept (oral)