CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning
Authors: Sheng Yue
Guanbo Wang
Wei Shao
Zhaofeng Zhang
Sen Lin
Ju Ren
Junshan Zhang
Venue: ICLR 2023 poster (2023)
Keywords: offline inverse reinforcement learning
inverse reinforcement learning
offline reinforcement learning

Abstract: This work aims to tackle a major challenge in offline Inverse Reinforcement Learning (IRL), namely the reward extrapolation error, where the learned reward function may fail to explain the task correctly and misguide the agent in unseen environments due to the intrinsic covariate shift. Leveraging both expert data and lower-quality diverse data, we devise a principled algorithm (namely CLARE) that solves offline IRL efficiently via integrating "conservatism" into a learned reward function and utilizing an estimated dynamics model. Our theoretical analysis provides an upper bound on the return gap between the learned policy and the expert policy, based on which we characterize the impact of covariate shift by examining subtle two-tier tradeoffs between the exploitation (on both expert and diverse data) and exploration (on the estimated dynamics model). We show that CLARE can provably alleviate the reward extrapolation error by striking the right exploitation-exploration balance therein. Extensive experiments corroborate the significant performance gains of CLARE over existing state-of-the-art algorithms on MuJoCo continuous control tasks (especially with a small offline dataset), and the learned reward is highly instructive for further learning. 

Published as a conference paper at ICLR 2023
CLARE: CONSERVATIVE MODEL-BASED REWARD LEARNING FOR OFFLINE INVERSE REINFORCEMENT LEARNING
Sheng Yue', Guanbo Wang", Wei Shao?; /Zhoffeg Zhang', Sen Lin"; Ju Ren!.!
Junshan Zhang'
'Tsinghua University, 'Tongii University, Univeessity of California, Davis,
4 AArizona State University, 'Ohio State University, ZZhongguaneun Laboratory
ABSTRACT
This work aims to tackle major challenge in offline Inverse Reinforcement Learning (IRL), namely the reward extrapolation error, where the learned reward function may fail to explain the task correctly and misguide the agent in unseen environments due to the intrinsic covariate shift. Leveraging both expert data and lower-quality diverse data, we devise principled algorithm (namely CLARE) that solves offiin IRL efficiently via integrating "conservatism" into a learned reward function and utilizing an estimated dynamics model. Our theoretical anal- ysis provides an upper bound on the return gap between the learned policy and the expert policy, based on which we characterize the impact of covariate shift by ex- amining subtle two-tier tradeoff's between the "exploitation" (on both expert and diverse data) and "exploration" (on the estimated dynamies model). We show that CLARE can provably alleviate the reward extrapolation error by striking the right pepplottttion-epllraaatio
balance therein. Extensive experiments corroborate the significant performance gains of CLARE over existing sate-off-hh--a algorithms on MuJoCo continuous control tasks (especially with a small offline dataset), and the learned reward is highly instructive for further learning (source code).
INTRODUCTION
The primary objective of Inverse Reinforcement Learning (IRL. is to learn a reward function from demonstrations (Arora & Doshi, 2021; Russell, 1998). In general, conventional IRL methods rely on extensive online trials and errors that can be costly or require a fully known transition model (Abbeel & Ng, 2004; Ratiff al., 2006; Ziebart et al., 2008; Syed & Schapire, 2007; Boularias et al., 2011; Osa et al., 2018), struggling to scale in many real-world applications. To tackle this problem, this paper studies offline IRL, with focus on learning from previously collected dataset without online interaction with the environment. Offline IRL holds tremendous promise for safety- sensitive applications where manually identifying an appropriate reward is difficult but historical datasets of human demonstrations robotics, etc.). In particular, since the learned reward function is a succinct representation of an expert's 's intention, it is useful for policy learning (e.g., in offline Imitation Learning (IL. (Chan & van der Schaar, 2021)) as well as number of broader applications (e.g., task description (Ng et al., 2000) and transfer learning (Herman et al., 2016)).
readily available (c.g.. in healthcare, autonomous driving,
This work aims to address a major challenge in offline IRL, namely the reward extrapolation error, where the learned reward function may fail to correctly explain the task and misguide the agent in unseen environments. This issue results from the partial coverage of states in the restricted expert demonstrations (i.e., covariate shift) as well as the high-dimensional and expressive function ap- proximation for the reward. It is further exacerbated due to no reinforcement signal for supervision and the intrinsic reward ambiguity therein. In fact, similar challenges related to the extrapolation
* Parr of this work was done when Sheng Yue, Wei Shao, and Sen Lin worked at Arizona State University.
1 'Corresponding author: renju juu@ta itttinghuu inghua, edu.. edu.cn en
'The reward ambiguity refers the fact that same behavior can be optimal for many reward functions.

---

Published as a conference paper at ICLR 2023
error in the value function have been widely observed in offline (forward) RL, e.g., in Kumar et al. (2020); Yu et al. (2020; 2021). Unfortunately, to the best of our knowledge, this challenge remains not well understood in offline IRL, albeit there is some recent progress (Zolna et al., 2020; Garg et al., 2021; Chan & van der Schaar, 2021). Thus motivated, the key question this paper seeks to answer is: "How to devise offline IRI. algorithms that ameliorate the reward extrapolation error effectively?"
We answer this question by introducing principled offline IRL algorithm, named conservative model-based reward learning (CLARE), leveraging not only (limited) higher-quality expert data but also (potentially abundant) lower-quality diverse data to enhance the coverage of the state-action space for combating covariate shift. CLARE addresses the above-mentioned challenge by appro- priately integrating conservatism into the learned reward to alleviate the possible misguidance in out-of-distiibution states, and improves the reward generalization ability by utilizing a learned dy- namics model. More specifically, CLARE iterates between conservative reward updating and safe policy improvement, and the reward function is updated via improving its values weighted expert and diverse state-actions while in turn cautiously penalizing those generated from model rollouts. As result, it can encapsulate the expert intention while conservatively evaluating di-oo-diistibutionn state-actions, which in turn encourages the policy to visit dupa-supp states and follow expert behaviors and hence achieves safe policy search.
Tradeoff
Expert data
Diverse data
Model-based synthetic data
"Exploitation"
Tradeoff
"Exploration"
Figure I: An illustration of the two-tier tradeoffs in CLARE.
Technically, there are highly nontrivial two-tier tradeoffs that CLARE has delicately calibrate: "balanced exploitation" of the expert and diverse data, and "exploration" of the estimated model. As illustrated in Fig. 1, The first tradeoff arises because CLARE relies on both exploiting expert demonstrations to infer the reward and exploiting diverse data to handle the covariate shift caused by the insufficient state-action coverage of limited demonstration data. At a higher level, CLARE needs to judiciously explore the estimated model to escape the offline data manifold for better gen- eralization. To this end, we first introduce the new pointwise weight parameters for offline data points (state-action pairs) to capture the subtle two-tier exploitation-explee explorattin tradeoffs. Then, we rigorously quantify its impact on the performance by providing an upper bound on the return gap between the learned policy and the expert policy. Based on the theoretical quantification, we derive the optimal weight parameters whereby CLARE can strike the balance appropriately to minimize the return gap. Our findings reveal that the reward function obtained by CLARE can effectively capture the expert intention and provably ameliorate the extrapolation error in offline IRI.
Finally, extensive experiments are carred out to compare CLARE with state-of- oftt-oof-hhe-ar offline IRL and offline IL algorithms on MuloCo continuous control tasks. Our results demonstrate that even using small offline datasets, CLARE obtains significant performance gains over existing algorithms in continuous, high-dimensional environments. We also show that the learned reward function can explain the expert behaviors well and is highly instructive for further learning.
2 PRELIMINARIES
Markoy decision process (MDP) can be specified by tuple M = (S. u, 7), consist-
ing of state space S, action space A, transition function S x A - P(S), reward function R : S A + IR, initial state distribution / S + [0, 1]. and discount factor
(0, A
stationary stochastic policy maps states to distributions over actions as ा + P(A). We de- fine the normalized state-action occupancy measure (abbreviated occupancy measure) of policy
Thh exploration in the context of this manuscript refers to enhancing the generalization capability of the algorithm by escaping the offline data manifold via model rollout.
2

---

Published as a conference paper at ICLR 2023
under transition dynamics T as p (s. = (1- Ei= P ,.) The objec- tive of reinforcement learning (RL) can be expressed as maximizing expected cumulative rewards: maxxem J(T) = -a-pp= [R(s, a)], where II is the set of all stationary stochastic policies that take actions in A given states in S.²
Maximum entropy IRL (MaxEnt IRL) aims to learn the reward function from expert demonstra- tions and reason about the stochasticity therein (Ziebart et al., 2008; Ho & Ermon, 2016). Based on demonstrations sampled from expert policy π n", the MaxEnt IRL problem is given by
min EER
max ( aspp (((ssa)) E, Es (r(s,a)] ((r), #Ell
with H()) = IS pr (s, a) ((as) ds da being the 7-discounted causal entropy, R. a family of reward functions, a 2 0 the weight parameter, and th R -> R U {oo} convex reward regularizer Fu et al. (2018); Qureshi al. (2018). Problem (1) looks for a reward function assigning higher rewards to the expert policy and lower rewards to other policies, along with the best policy under the learned reward function. Although enjoying strong theoretical justification and achieving great performance in many applications, MaxEnt IRL has solve forward RL problemm in the inner loop that involves extensive online interactions with the environment.
Ofline IRL is the setting where the algorithm is neither allowed to interact with the environment nor provided reinforcement signals. only has access to static dataset D = DE U DD consisting
of expert dataset D {(ss, ag, s 1 and diverse dataset DB = {(s, a, st) Du collected by
expert policy = and behavior policy " respectively. The goal offline IRL is to infer a reward function capable of explaining the expert's preferences from the given dataset.
3 CLARE: CONSERVATIVE MODEL-BASED REWARD LEARNING
A naive solution for offline IRL is to retrofit MaxEnt IRL to the offline setting via estimating dynamics model using offline data (e.g., Tanwani Billard (2013); Herman et al. (2016). Unfor- tunately, it has been reported that this naive paradigm often suffers from unsatisfactory performance in high-dimensional and continuous environments Jarrett et al. (2020). The underlying reasons for this issue include: (1) the dependence on full knowledge of the reward feature function, and (2) the lack of effective mechanisms to tackle the reward extrapolation error caused by covariate shift (as stated in Section 1). Nevertheless, we believe that utilizing learned dynamics model is beneficial because is expected to provide broader generalization by learning on additional model-generated synthetic data (Yu et al., 2020; 2021; Lin al., 2021). With this insight, this work focuses the model-based offline IRL method that is robust to covariate shift while enjoying the model's gener- alization ability.
As illustrated in Fig. 1, there are two-tier subtle tradeoff's that need to be carefully balanced between exploiting the offline data and exploring model-based synthetic data. On one hand, the higher-quality expert demonstrations are exploited to infer the intention and abstract the reward function therein, while the lower-quaity diverse data exploited to enrich data support. On the other hand, it is essential to prudently explore the estimated dynamics model to improve the generalization capability while mitigating overfitting errors in inaccurate regions. To this end, we devise conservative model- based reward learning (CLARE) based on MaxEnt IRL, where the new pointwise weight parameters are introduced for each offline state-action pair to capture the tradeoffs subtly. We elaborate further in what follows.
As outlined below, CLARE iterates between (I) conservative reward updating and (II) safe policy improvement, under a dynamics model (denoted by 7) learned from offline dataset.
(I) Conservative reward updating. Given current policy R,, dynamics model î, and offline datasets
Dg and Dg. CLARE updates reward funtion r based on the following loss:
[(rlm)= a (r(s, a)]
E. Eganp [r(s,a) a)]
Es.o-po [8(s, (s)(s, a)) + (zrb(r),
(2)
penalized model rollous
incressed expert data
weighting expert and diverse data regalarizer
For convenience, we omit constant multiplier, 1/(1 7). in the objective for conciseness, i.c., the complete objective function given by max " [R(s, a)/1 7
3

---

Published as a conference paper at ICLR 2023
where (PD ())(ss))))/(D) + +D) is the empirical distribution of (s. a) in the union dataset D DE UDB and pt -D=sss,, is that for expert dataset De: p the occupancy measure when rolling out with dynamics model T; and de denotes a convex regularizer mentioned above. One key step to add an additional term weighting the reward of each offline state-action by [(s, a), which is "fine-grained control" for the exploitation of the offline data. For the data deserving more exploitation (e.g., expert behaviors with sufficient data support), we can set a relatively large (s, a); otherwise, we decrease its value. Besides, it can also control the exploration of the model subtly (consider that if we set all B(s, a) = 0, Eq. (2) reduces to MaxEnt IRL, enabling the agent to explore the model without restrictions). Here, ZB = 1 + Ey a ما>مم [3(s', ,a)] is normalization term. The new ingredients beyond MaxEnt IRI. are highlighted in blue.
Observe that in Eq. (2), by decreasing the reward loss, CLARE pushes up the reward good of- fline state-action that characterized by larger ((,, a). while pushing down the reward on potentially out-of-distribution ones that generated from model rollouts. This is similar COMBO (Yu al., 2021) in spirit, a state-of-the-ard offline forward RL algorithm, and results in a conservative re- ward function. can encourage the policy to cautiously exploring the state-actions beyond offline data manifold, thus capable of mitigating the misguidance issue and guiding safe policy search. In Section 4, we will derive closed-form optimal [(s, a) that enables CLARE achieve a proper exploration-exploiaaiio trade-off by minimizing return gap from the expert policy.
(II) Safe policy improvement. Given updated reward function r, the policy is improved by solving
max [(*rr)= am (r(s. a)] + (T)) xEll
where a 2 0 is weight parameter, and (T()) = - If pt (s, a) log (als) da da is the y-discounted causal entropy induced by the policy and learned dynamics model. Due to the embedded expert intention and conservatism in the reward function, the policy is updated safely by carrying out conservative model-based exploration. One can use any well-established MaxEnt RL approach to solve this problem by simulating with model T and reward function r. It is worth noting that for Problem (3) in this step, the practical implementation of CLARE works well with a small number of updates in each iteration (see Sections and 6).
THEORETICAL ANALYSIS OF CLARE
In this section, focus answering the following question: "How to set B(ss, a) for each offline state-action pair to strike the two-tier exploitation-exploriiiie first quantify the impact of the tradeoff's via bounding the return gap between the learned policy and expert policy. Then, we derive the optimal weight parameters to minimize this gap. All the detailed proofs can be found in Appendix B. Notably, this section works with finite state and action spaces, but our algorithms and experiments run in high-dimensional and continuous environments.
balance appropriately?" To this end,
4.1 CONVERGENCE ANALYSIS
We first characterize the policy learned by CLARE, in terms ((s, a) and empirical distributions j and 9P. Before proceeding, it is easy to see CLARE is iteratively solving the min-max problem:
min max o (T) + Z [r(s,a)] E5 [3(s, (s)s E3e [r(s,a)] + Zzw(r)-
(4)
rR nell-
=
For dynamics T, define the set of occupancy measures satisfying Bellman flow constraints as
{ P RIIAI 0 and Σ
= [()) + Σ T(sss ())((a)pssss)) a) v)
Cr =
(5)
We first provide the following results for switching between policies and occupancy measures, which allow us to use Tp to denote the unique policy for occupancy measure p.
Lemma 4.1 (Theorem 2 in Syed et al. (2008)). If p E Cr, then is the occupancy measure for sta- tionary policy ग T,(a (als) p(s,a)) a)/ Ex p(s, ).), and nT is the only stationary policy with occupancy measure p.
4

---

Published as a conference paper at ICLR 2023
Lemma 4.2 (Lemma 3.2 in Ho & Ermon (2016)). Denote H() = Es pls, a) log p(s.a)
Σ., p(s.ar)
Then, H is strictly concave, and for all E II and p E Cr, (T)) = H(p") and (p) = (xp)) hold true, where p s) p(s, u/) En p(s,a')
Based on Lemma 4.1 and Lemma 4.2, we have the follow results on the learned policy.
Theorem 4.1. Assume that 3(s, a) (s) (s,a) holds for (s, D. For Problem (4), the following relationship holds:
min max ((T,T) = max ( - ZzD' (p. 80 + ER well
(6)
pECT
ZB
with
(p= - P1), where x²* is the convex conjugate of vb.
Notably, by selecting appropriate forms of reward regularizers v. D& can belong to a wide-range of statistical distances. For example, if (b(r) ar² then (p P2) = क x P2): if tb restricts E [-R , Rmax], then D& (p1. P2) 2Rmaa Drv (P1. P2) (Garg et al., 2021). Theorem 4.1 implies that CLARE implicitly seeks a policy under T whose occupancy measure stays close to an interpolation of the empirical distributions of expert dataset DE and union offline dataset D. The interpolation reveals that CLARE is trying to trade off the exploration of the model and exploitation of offline data by selecting proper weight parameters (s, a). For example, if ((s. (s, a) )) E D, CLARE will completely follow the occupancy measure of the (empirical) expert policy by explore the model freely. In contrast, if [(s, a) increases with pD((s, a), the learned policy will look for richer data support.
for all
Remarks. Looking deeper into Eq. (6), the target occupancy measure can be expressed equiva- lently as (1+3Ds (Dops/DD)***
after rearranging terms in the above interpolation. As result, CLARE also subtly balances the exploitation between the expert and diverse datasets to extract potentially valuable information in the sub-optimal data.
4.2 STRIKING THE RIGIT
BALANCE
Next, we show how to set B(s, a) properly to achieve the right two-tier balance.
Recall that (TT) = Es [R(s, a)] is the return achieved by policy æ. The next result provides upper bound on the return gap between (TT) and (x which hinges on the intrinsic trade-offs.
Theorem 4.2. Suppose |R(s, I for any S,a A. For any stationary policy n, let pa denote the occupancy measure of x under estimated model T. We have that
J(TB) J(x) S C. Es,amip (Dv((T()ss,)), (()s,)))))))))) +2 (Div(0" ,p)) (p .P))
where = 27 and pm Fi is the occupancy measure of expert policy π. under true dynamics T.
Remarks. Theorem 4.2 indicates that good policy learned from the estimated model not only follows the expert behaviors but also keeps in the "safe region" the learned model, i.e., visiting the state-actions with less model estimation inaccuracy. Under the concentration assumption, the following holds with probability greater than d-d:
CCss
J - J(x) < E Ex,a~ir
+2 Drv(i",p") (.) +2 DTv(o".p")
VIDE (DD(s a) [DD(s,a)
(b)
(4)
D where D(s,a) a) = ((s',a') D = a}. It aligns well with the aforementioned exploration-exploiaaiio balance: 1) Term (a) captures the exploitation of offline data support; 2) Term (b) captures the exploitation of expert data and the exploration of the model (recall that nr is the occupancy measure of rolling out x with T); and 3) Term (c) captures the distributional shift in offline learning. Importantly, the result in Theorem 4.2 connects the true return of policy with its occupancy measure on the learned model. This gives us a criteria to evaluate the performance of a policy from offline. Define c(s, C Drv a), T(-|s, a)) and min = minga c(s, a). Subsequently, we derive the policy that minimizes the RHS of Eq. (7).
5

---

Published as a conference paper at ICLR 2023
Algorithm 1: Conservative model-based reward learning (CLARE)
expert data DE. diverse data DB. bar u, learning rate y, policy regularizer weight Learn dynamics model T represented by an ensemble of neural networks using all offline data; Set weight ((s, for each offline state-action tuple (s, a) De U DD by Eq. (11); Initialize the policy Ty and reward function rs parameterized by and & respectively;
Input:
while not done do
(Safe policy improvement) Run MaxEnt RL. algorithm for some steps with model 7 and current reward function To to update policy based on L( (r)) ADkI (="]] (Conservative reward updating) Update ra by Φ + Φ 7/ (ral(TTTT) for a few steps;
end
Theorem 4.3. Under the same conditions as in Theorem 4.2, the optimal occupancy measure mini- mizing the upper bound of Eq. (7) is given as follows:
3 (s. Ap.
ff(s,a) S emin
0'
if((s,a) > min + +2, otherwise.
(5,
where A, Ention Ha(s' a²) aa) omin >2 (s' and Nmin ((s,a) E :D & cmin).
[mmml
As shown in Theorem 4.3, the "optimal" policy leaned on model T conservatively explores the model by avoiding the visit of risky state-actions. Meantime, it cleverly exploits the accurate region, such that it does not deviate large from the expert. Now, of the weight parameters.
ready to derive the optimal values
Corollary 4.1. Suppose that when x (s, a) 0, c(s,a) Juin holds for each (8,a) ES Under the same condition as in Theorem 4.3, if B[, are set
is vc(s,a) cmin and 7P (8,0) > 0, if fc(s,a) min and p (s, 20, otherwise,
( (')a,)) (3,a))
(s) =
(9)
0.
then it follows that
min [(Trr) max aHT D rER wEll
(10)
Corollary 4.1 provides the value of B(s, a) for each (s. a) E D such that the learned reward function guide the policy minimize the return gap in Eq. (7). It indicates that the right exploitation- exploration trade-off can be provably balanced via setting the weight parameters properly. In par- ticular, 8 assigns positive weight to the offline state-action with accurate model estimation and negative weight to that with large model error. It enables CLARE to learn conservative reward function that pessimistically evaluates the -iits--distiotution states and actions, capable of amelio- rating the extrapolation eтror in unseen environments. However, the optimal weights require the model error, c(s, u). which is typically hard to obtain (especially in high-dimensional and contin- uous spaces). Section 5 will solve this problem by extending this result with the aid of the model ensembles and uncertainty quantification techniques.
PRACTICAL IMPLEMENTATION
Learning dynamics models. Following the state-of-the-arr model-based methods (Yu et al., 2020; 2021), model the transition dynamics by ensemble of neural networks, each of which outputs Gaussian distribution
next states, i.e., (s'(s')s, |s, a) = ((44ss a). E,(s,a)) N
Weights in continuous environments. The ideas of achieving CLARE in continuous environ- ments are 1) to approximately see the offline data sampled from large discrete space, and 2)

---

Published as a conference paper at ICLR 2023
to use uncertainty quantification technique for quantifying the model error. Specifically, because state-action pairs are basically different from each other in this setting, we let 50 (s, a) = 1/D and p= a) = 1/ and employ the uncertainty estimator, c(s, a) [ [[ a)ll r. pro- posed in Yu et al. (2020) for model error evaluation. Guided by the analytical results in Corollary 4.1, compute the weights for each (8,a) a) E DD via slight relaxation as follows:
N D
ff(ss, a) S u, if((s,a) > u, otherwise,
N'
B(s,a)
I((s,a) E DEl,
(11)
DE 0,
where N' = = [c((s,a) I uj and N" = 1 [c((s,)) > u/. Here, coefficient
user-chosen hyper-parameter for controlling the conservatism level of CLARE. If one wants the learned policy to be trained more conservatively on offline data support, u should be small; otherwise, can be chose to be large for better exploration.
Reward and policy regularizers. In the experiments, we use (r()) = 7² as the reward regularizer. Additionally, when updating the policy, we behavior policy xb induced by a subset of the offline dataset, D' C DD as follows:
KL divergence as regularizer with empirical
(KL(TT1/Z) EEED [Ba-s((1a] (als)] Bana [log [a
where =a(ass Did 1 1(===s,'=a))
Eid'a') [[s=ss] if (s. D', and (als)) = 0 otherwise. It can be im- plemented by adding - -Bs,a- [lgx<aaa)) (a|s)] to the actor loss. The intuition is to encourage the actor to perform in support of the real data for accelerating safe policy improvement. While this regular- ization lacks theoretical guarantees, we empirically find that it can indeed speed up the training.
Practical algorithm design. The pseudocode of CLARE depicted in Algorithm 1. The policy im- provement phase can be implemented by the standard implementation of SAC (Haarnoja al., 2018) with a change of the additional policy regularizer. We elaborate more details in the Appendix A.
6 EXPERIMENTS
Next, we use experimental studies to evaluate CLARE and answer the following key questions: (1) How does CLARE perform the standard offline RL benchmarks in comparison existing state- of-the-art algorithms? (2) How does CLARE perform given different dataset sizes?
(3) How does the "conservatism level", u, affect the performance? (4) How fast does CLARE converge? (5) Can the learned reward function effectively explain the expert intention?
To answer these questions, we compare CLARE with the following existing offline IRL methods on the D4RL benchmark (Fu et al., 2020): 1) IQ-LEARN (Garg et al., 2021), a state-of-the-ar model- free offline IRL algorithm; 2) AVRIL (Chan & van der Schaar, 2021), another recent model-free offline IRL method; 3) EDM (Jarett et al., 2020), a state-of state-of-the-ar art offline IL approach; and 4) Behavior Cloning (BC). To demonstrate the poor performance of the naive approach using a sim- ple combination of IRL with model-based offline forward RL (MORL) method, we also consider a baseline algorithm, namely MOMAX, by directly using COMBO (Yu et al., 2021) the inner loop of MaxEnt IRL. We present the results continuous control tasks (including Chef-,heetah, Walker2d, Hopper, and Ant) consisting of three data qualities (random, medium, and expert). Ex- perimental set-up and hyperparameters described in detailed in Appendix A.
Walker2d
Hopper
Ant
Halfcheetah
Average Reaum
8000
CLARE
Avernan Return
3.000
Averanee Reaum
4000
1occoon Return
CLAME
7500
2000
000
2000
5000
10 Leamn
E3M
B07
2500
EDN
Expas
10k
50k
150k
10k
50kk
200k
150k
10k
100k
150k
Number of Tuples
Number of Tuples
Number
Number Tuples
Figure 2: CLARE against other algorithms on all tasks over different dataset sizes consisting of expert and medium data equally.
7

---

Published as a conference paper at ICLR 2023
Table Results on D4RL datasets. For each task, the experiments are carried out with three different
data combinations: 1) 10k expert tuples, 2) 5k expert and 5k medium tuples, and 3) 5k expert and 5k
random tuples. The data scores below for 1). 2), and 3) correspond to expert, medium, and random data, respectively. We tune IQ-LEARN, EDM, and AVRIL based on their publicly available source code. Results are averaged over 7 random seeds. The highest score across all algorithms is bold.
Dataset type
Environment
Data score
CLARE
BC
IQ-LEARN
EDM
AVRIL
MOMAX
Walker2d
1.9
2873.8
17.8
256.9
165.5
100.9
-525,4
Exp. Rand. Hopper
18.4
1891.5
110.2
523.6
178.8
178.3
0.7
Ant
-64.4
1960.0
427.6
-247.2
-3000.9
1000.1
113.8
Half-Cheetah
-505.1
1113.7
-86.7
123.9
-346.7
-1093.5
-11.0
Walker2d
3496.3
3613,4
1674.2
1676.8
175.7
184.0
19.6
Hopper
1422.7
2135.0
947.0
2049.8
194.4
183.7
27.6
Exp. & Med.
Ant
3969.0
3879.4
2146.0
222.2
-3001.5
1001.0
-33.2
Half-Cheetah
4667.8
4888.6
2375.0
2957.7
-298.3
-1965.6
-0.2
Walker2d
5010.4
4990.5
1665.7
2445.4
189.7
194.1
23.2
Hopper
3603.2
2604.5
1436.1
2854.4
192.5
183.9
34,5
Exp.
Ant
5172.8
3940.3
1797.9
375.4
-3000.6
1000.2
48.1
Half- Chff-Chettah
10748.7
4975.1
242.4
3750.5
-299.5
-619.0
-0.4
Halfcheetah
Walker2d
Ant Iterations
Iterations
Iterations
Hopper
203
Average Return
Average Return
Average Return +
Average Returnn
2
2
2
2-
666 with the thel revori Sindinn undirdy ing recsee
CLARE
CLARE
Steps
Steps
Steps
Millian Steps
(a) Impact of u.
(b) Convergence speed.
(c) Convergence speed.
(d) Recovered reward.
Figure 3: Performance of CLARE. 1) Impact of u: Figure 3(a) shows the impact of user-chosen parameter the performance using 10k expert tuples. 2) Convergence speed: Figures 3(c) and 3(b) show the convergence CLARE using 10k expert and 10k medium tuples. In each iteration, CLARE carries out policy improvement by total 10k gradient updates (total 500 epochs with 20 gradient steps per epoch) for the actor and critic networks using SAC. 3) Recovered reward: Fig- 3(d) shows the result of training SAC via replacing the underlying reward by the one learned from CLARE.
Results MuJoCo control. To answer the first question and validate the effectiveness of the learned reward, we evaluate CLARE on different tasks using limited state-action tuples sampled from D4RL datasets. The ranges of standard deviations of the results in Exp. & Rand., Exp. & Med. and Exp. are 156.4-280.5, 15.7-127.8 and 42.4-89.5, respectively. As shown in Table 6, CLARE yields the best performance by significant margin on almost all datasets, especially with low- quality data thereof. It demonstrates that the reward function learned by CLARE can effectively guide offline policy search while exploiting the useful knowledge the diverse data.
Results under different dataset sizes. To answer the second question, we vary the total numbers of state-action tuples from 2k 100k and present the results on different tasks in Figure CLARE reaches expert performance on each task with sufficient data. Albeit with very limited data, CLARE also achieves strong performance over existing algorithms, revealing its great sample efficiency.
Results under different To answer the third question, we normalize the uncertainty measure to [0, and vary u from 0.1 to 1.0. Due to Eq. (11), smaller u corresponds to a more conservative CLARE. As illustrated in Figure 3(a), the performance becomes better with the decrease of u value. validates the importance of the embedded conservatism in alleviating the extrapolation error. We empirically find that the performance with respect to varies in different tasks. Thus, treat it as a hyper-paramete to tune In practice.
Convergence speed. To answer the fourth question, present the results the convergence speed of CLARE in Figure 3(b), revealing its great learning efficiency. showcases that CLARE converges in 5 iterations with totally less than 50k gradient steps.
8

---

Published as a conference paper at ICLR 2023
Recovered reward function. To answer the last question, we evaluate the learned reward function by transferring it to the real environment. As demonstrated in Figure 3(c), the reward function is highly instructive for online learning. It implies that it can effectively reduce the reward extrap- olation error and represent the task preferences well. Surprisingly, compared to the true reward function, the policy trained via the learned one performs more stably. The reason is that the learned one incorporates conservatism and thus is capable of penalizing risks and guide safe policy search.
7 RELATED WORK
Offline IRL. To side-step the expensive online environmental interactions in classic IRL, offline IRL aims to infer reward function and recover the expert policy only from a static dataset with no access to the environment. Klein et al. (2011) 1) extend the classic apprenticeship learning (i.e., Abbeel & Ng (2004)) to batch and off-policy cases by introducing a temporal difference method, namely LSTD-H, to compute the feature expectations therein. Klein et al. (2012) further introduce a linearly parameterized score canction-based multi-clase classification algorithm to output reward function based estimate of expert feature expectation. Herman et al. (2016) present gradient-basee based solution that simultaneously estimates the feature weights and parameters of the transition model by taking into account the bias of the demonstrations, Lee et al. (2019) propose Deep Successor Feature Networks (DSFN) that estimates feature expectations in an off-policy setting. However, the assumption of full knowledge of the reward feature functions in Klein et al. (2011); Herman et al. (2016); Lee et al. (2019); Jain et al. (2019); Pirotta & Restelli (2016); Ramponi et al. (2020) is often unrealistic, because the choice of features is problem-dependen and can become a very hard task for complex problems (Arora & Doshi, 2021; Piot et al., 2014). To address this problem, Piot et al. (2014) propose a non-parametric algorithm, called RCAL,, using boosting method to minimize di- rectly the criterion without the step of choosing features. Konyushkova et al. (2020) propose two semi-supervised learning algorithms that learn a reward function from limited human reward anno- tations. Zolna et al. (2020) further propose ORIL that can learn from both expert demonstrations and a large unlabeled set of experiences without human annotations. Chan & van der Schaar (2021) use variational method to jointly learn an approximate posterior distribution over the reward and policy. Garg et al. (2021) propose an off-policy IRL. approach, namely IQ-Learn, implicitly repre- senting both reward and policy via learned soft Q-function. Nevertheless, these methods primarily concentrate on offline policy learning with learning reward function being an intermediate step. Due to the intrinsic covariate shift, these methods may suffer from severe reward extrapolation error, leading to misguidance in unseen environments and low learning efficiency.
Offiine II. Akin to offline IRL, offline imitation learning (offline IL) deals with training an agent to directly mimic the actions of demonstrator in an entirely offline fashion. Behavioral cloning (BC (Ross & Bagnell, 2010)) is indeed an intrinsically offline solution, but fails to exploit precious dynamics information. To tackle this issue, several recent works propose dynamics-aware offline II. approaches, e.g., Kostrikov et al. (2019); Jarrett et al. (2020); Chang et al. (2021); ); Swamy et al. (2021). In contrast to directly mimicking the expert as done in offline IL, offline IRL explicitly learns the expert's 's reward function from offline datasets, which can take into account the temporal structure and inform what the expert wishes to achieve, rather than simply what they are reacting to. It enables agents to understand and generalize these "intentions" " when encountering similar environments and therefore makes offline IRL more robust (Lee et al., 2019). In addition, the learned reward function can succinetly explain the expert's 's objective, which is also useful in a number of broader applications (e.g., task description Ng et al. (2000) and transfer learning Herman et al. (2016)).
CONCLUSION
This paper introduces a new offline IRI. algorithm (namely CLARE) to approaching the reward ex- trapolation error (caused by covariate shift) via incorporating conservatism into a learned reward function and utilizing an estimated dynamics model. Our theoretical analysis characterizes the im- pact of covariate shift by quantifying subtle two-tier exploitation-exxllraiio tradeoff's and we show that CLARE can provably alleviate the reward extrapolation error by striking the right trade- offs therein. Extensive experiments corroborate that CLARE outperforms existing methods in con- tinuous, tigh-dimensional environments by a significant margin, and the learned reward function represents the task preferences well.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This research was supported in part by the National Natural Science Foundation of China under Grant No. 62122095, 62072472, and U19A2067, by NSF Grants CNS-2203239, CNS-2203412, and RINGS- 221425,, and by grant from the Guoqiang Institute, Tsinghua University.
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proc. ICML, pp. 2004.
Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, meth- ods and progress. Artif. Intell., 297:103500, 2021.
Abdeslam Boularias, Jens Kober, and Jan Peters. Relative entropy inverse reinforcement learning. In Proc. AISTATS, pp. 182-189, 2011.
Alex J Chan and M van der Schaar. Scalable bayesian inverse reinforcement learning. In Interna- fional Conference on Learning Representations, 2021.
Jonathan Chang, Masatoshi Uehara, Dhruv Sreenivas, Rahul Kidambi, and Wen Sun. Mitigating covariate shift in imitation learning via offline data with partial coverage. Proc. NeurIPS, 34: 965-979, 2021.
Ding-Zhu Du and Panos M Pardalos. Minimax and applications, volume 4. Springer Sei. Bus. Media, 2013.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse rein- forcement learning. In Proc. ICLR, 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song. and Stefano Ermon. Iq-learm: Inverse soft-q learning for imitation. Proc. NeurlPS, 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proc. ICML, volume 80, pp. 1861-1870. PMLR, 2018
Michael Herman, Tobias Gindele, Jörg Wagner, Felix Schmitt, and Wolfram Burgard. Inverse re- inforcement learning with simultaneous estimation of rewards and dynamics. In Proc. AISTATS, pp. 102-110. 10. PMLR, 2016.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Proc. NeurIPS, volume 29, pp. 4565-4573. Curran Associates, Inc., 2016.
Vinamra Jain, Prashant Doshi, and Bikramjit Banerjee. Model-free irl using maximum likelihood estimation. In Proc. AAAI, pp. 3951-3958, 2019.
Daniel Jarrett, Ioana Bica, and Mihaela van der Schaar. Strictly batch imitation learning by energy- based distribution matching. Proc. NeurlPS, 2020.
Edouard Klein, Matthieu Geist, and Olivier Pietquin. Batch, off-policy and model-free apprentice- ship learning. In Proc. EWRL, pp. 285-296. Springer, 2011.
Edouard Klein, Matthieu Geist, Bilal Piot, and Olivier Pietquin. Inverse reinforcement learning through structured classification. Proc. NewrlPS, 25, 2012.
Ksenia Konyushkova, Konrad Zolna, Yusuf Aytar, Alexander Novikov, Scott Reed, Serkan Cabi, and Nando Freitas. suni-supervieed reward learning for offline reinforcement learning. In Proc. Ne Workshop, 2020.
10

---

Published as a conference paper at ICLR 2023
Iyy Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off- off-policy distribution matching. In Proc. ICLR, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Proc. NeurIPS, pp. 1179-1191, 2020.
Donghun Lee, Srivatsan Srinivasan, and Finale Doshi-Velez. Truly batch apprenticeship learning with deep successor features. In Proc. IJCAI, 2019.
Sen Lin, Jialin Wan, Tengyu Xu, Yingbin Liang, and Junshan Zhang. Model-based offline meta- reinforcement learning with regularization. Proc. ICLR, 2021.
Andrew Ng. Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Proc. ICML, volume 1, pp. 2, 2000.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorithmic perspective on imitation learning. Found. Trends Rob., 2018.
Bilal Piot, Matthieu Geist, and Olivier Pietquin. Boosted and reward-regularized classification for apprenticeship learning. In Proc. AAMAS, pp. 1249- 1256, 2014.
Matteo Pirotta and Marcello Restelli. Inverse reinforcement learning through policy gradient mini- mization. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Ahmed H Qureshi, Byron Boots, and Michael C Yip. Adversarial imitation via variational inverse reinforcement learning. In Proc. ICLR, 2018
Giorgia Ramponi, Amarildo Likmeta, Alberto Maria Metelli, Andrea Tirinzoni, and Marcello Restelli. Truly batch model-free inverse reinforcement learning about multiple intentions. In International Conference on Artificial Intelligence and Statistics, pp. 2359-2369. PMLR, 2020.
Nathan D Ratliff, J Andrew Bagnell, and Martin A Zinkevich. Maximum margin planning. In Proc. ICML, pp. 729-736, 2006.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proc. AISTATS, pp- 661-668. JMLR Workshop and Conference Proceedings, 2010.
Stuart Russell. Learning agents for uncertain environments. In Proc. COLT, pp. 101-103, 1998.
Gokul Swamy, Sanjiban Choudhury, Andrew Bagnell, and Steven Wu. Of moments and matching: A game-theoretic framework for closing the imitation gap. Proc. ICML, pp. 10022-10032. PMLR, 2021.
Umar Syed and Robert E Schapire. A game-theoretic approach to apprenticeship learning. Proc. NeurlPS, 2007.
Umar Syed, Michael Bowling, and Robert E Schapire. Apprenticeship learning using linear pro- gramming. In Proc. ICML, pp. 1032-1039. ACM, 2008.
Ajay Kumar Tanwani and Aude Billard. Inverse reinforeement learning for compliant manipulation in letter handwriting. National Center of Competence in Robotics (NCCR), 2013.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In Proc. NeurIPS, volume 33, pp. 14129-14144 Curran Associates, Inc., 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline masel-based policy optimization. Proc. NeurIPS, 2021.
Brian Ziebart, Andrew I. Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Proc. AAAI, volume 8. AAAI Press, 2008.
Konrad Zolna, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcehre, Ziyu Wang, Yusuf Ay- tar, Misha Denil, Nando de Freitas, and Scott Reed. Offline learning from demonstrations and unlabeled experience. In Proc. NeurIPS Workshop, 2020.

---

Published as a conference paper at ICLR 2023
A EXPERIMENTAL DETAILS
In this section, we present necessary experimental details for reproducibility.
A. PRACTICAL IMPLEMENTATION DETAILS
In the experiment, our implementation is built upon the open source framework of offline RL al- gorithms, provided at: https: //////tttuut/ com com/pol Exir//offf ineRL, including data sam- pling, policy testing, dynamics model structure, etc. The implementation of SAC in the pol- icy improvement uses the open source code available at: https: : / //g thub. cww.ooo////oooo. hommppaaz24/ pytorch-soft aattr-ttoo------ sortrcc--oo-toooo--ooooooooooo-- critic (under the MIT License). Additionally, the expert and di- verse state-action pairs are sampled at random from the D4RL dataset provided at: https: (under the Apache License 2.0).
//github. com/ wttp:rraaabeeeeeeeeeeeeer
Model learning. Following the same line as in Yu et al. (2020; 2021), we model the transi- tion dynamics by an ensemble of probabilistic neural networks, each of which takes the current state and action input and outputs Gaussian distribution over next states, i.e., (Ti(s'),, (Mu((s, a). (s,s, a))};", Using offline state-action pairs, 7 models are trained independently via maximum likelihood, each of which is represented as by 4-layer feedforward neural network with 256 hidden units. The best 5 models are picked based on the validation prediction error on a held-out set. During model rollouts, one model will be selected randomly from the ensemble.
Policy improvement. We represent both critic and actor as 2-layer feedforward neural network with 256 hidden units and Swish activation functions. each iteration, we update the critic and actor networks using SAC (Haarnjja et al., 2018) for 500 epochs (each has 20 gradient updates). As described in Section 5, we use a KL divergence with the behavior policy to accelerate inner-loop policy search. It is implemented by adding -Es.awy [log [a]s)] (D' E DD to the actor loss. An instantiation of the policy improvement can be found in Algorithm 2.
Algorithm 2:
Safe policy improvement
Input: offline dataset D. policy regularizer weight A, rollout horizon H, rollout batchsize B, the number of epochs E, dynamics ensemble (T:) reward function rig policy π Initialize model buffer Dindel <- Ø; for epoch 1 to do for b = to B3 in parallel do
Sample state 81 from as the initial state of the rollout;
for / 1 to Hl do
Sample action αд ~ (-(9))));
Randomly pick dynamics T from (T and sample 8h+1 T( on);
Compute T4 + (o(Sh, an);
Add sample (sh, ah, Th, 8h+11 D Dmndel
end end
Sample batches from Dmodel and use SAC to update policy mo with -. (log (a(a)) (D' E 'D) added on the policy loss;
end
Reward updating. We represent the reward function as 4-layer feedforward neural network with 256 hidden units and Swish activate functions. In each iteration, the reward function is updated by 5 gradient steps with stepsize 5 10-5, based on the following practical reward loss:
[(ra) = Z peptaa [ro(s, a)] + Z,z a [r)(),a)"]
Eg (s, a)] E, Ex,aD [ ()(s,)) a)].
(12)
We use replay buffer Dreplay across iterations to save the simulated data for training stability. An instantiation of reward updating is shown in Algorithm 3.
Practical algorithm. Based on Algorithms 2 and 3, a detailed CLARE algorithm is outlined in Algorithm 4.
12

---

Published as a conference paper at ICLR 2023
Algorithm 3: Conservative reward updating
Input:
expert data Dg, diverse data DB, replay buffer Preplay model buffer moddel reward function re, learning rate 7) the number of steps 7
Update replay buffer repply t- Dreplay U model:
for t to I do Update the parameters reward function To by Φ o /
end
Algorithm 4: Conservative modeel-based reward learning (CLARE)
expert data Dr, diverse data DB, bar u, learning rate policy regularizer weight À Learn dynamics model T represented by an ensemble of neural networks using all offline data; Set weight (ss, a) for each offline state-action tuple (s, a) E Dg U DD by Eq. (11); Initialize the policy Ty and reward function rs parameterized by 0 and & respectively; Initialize replay buffer Dreplay +- Ø;
Input:
while not done do
(Safe policy improvement) Run Algorithm 2 to update policy vy and get model buffer
D model:
(Conservative reward updating) Run Algorithm 3 to update reward function Toi
end
A.2 HYPERPARAMETERS
We summarize the hyperparameters used in the evaluation follows.
Conservatism level For all tasks, we normalize the uncertainty measure to [0. 1] and test from set {0.4, 0.6, 0.8}. The result is shown in Table 3. In each experiment, we select the u value that achieves the maximum corresponding score.
Learning rates. For all experiments, the reward learning rate i) = 5x 10-5, Our empirical studies indicate that a relatively small reward learning rate leads to more stable training. Additionally, the learning rates for actor and critic are both x 10-4, and that for dynamics model is 10-3
Policy regularization. For all experiments, the policy regularization weight is À 0.25.
The additional hyperparameters are listed in Table A.2.
Table 2: Hyperparameters for CLARE. Instead of the hyperparameters used in the evaluation are identical across different tasks (Half-Cheetal) Walker2d, Hopper, and Ant).
Hyperparameter
Value
Reward learning rate (7)
10-5
Rollout batchsize (B)
5000
Rollout horizon (ff)
Policy regularization weight (A)
0.25
Discount factor (7)
0.99
steps per reward updating (T)
5
epochs (E)
500
steps per epoch
20
Actor learning rate
3 x 10
Critic learning rate
10
A.3 MORE EXPERIMENTAL RESULTS
We further evaluate CLARE by answering the following two questions: 1) Can CLARE exploit the useful information from diverse datasets? 2) How does CLARE perform compared the simple
13

---

Published as a conference paper at ICLR 2023
Table Performance under different values. We tune a from set {0.4, 0.6, 0.8}. For each MuJoCo
task, the experiments carried out with three data combinations: 1) 10k expert state-action tuples, 2) 5k expert and 5k medium state-action tuples, and 3) 5k expert and 5k random state-action tuples.
The highest score across different is bold.
Dataset type
Environment
0.4
0.6
t 0.8
Walker2d
2896.93
2989.79
1083.17
Hopper
1187.22
1841.15
1508.09
Exp. Rand.
Ant
2047.98
1496.09
1337.01
Half-Cheetah
453.03
1118.58
849.42
Walker2d
3334.55
3680.78
3275.21
Hopper
1722.44
2107.90
1963.59
Exp. & Med.
Ant
3568.49
3805.64
2635.30
Half-Cheetahh
4955.23
4349.8 .88
4000.17
Walker2d
4674.52
4958.04
4742.20
Hopper
1954.04
2605.82
2328.22
Exp.
Ant
2747.10
3925.90
3330.48
Half-Cheetah
5050.05
4942.20
4542.45
combination of MORL and (online) IRI. methods? 3) What is the impact of reward weighting? 4) What is the impact of expert sample sizes?
Exploitation on diverse data. Table shows the results under different data combinations. By using additional medium data, the performance can be improved over that only using 5k expert tuples. The underlying rationale is: 1) The diverse datasets contain some good state-actions; 2) the diverse data support enables CLARE to safely generalize to the states beyond expert data manifold.
Table 4: Impact of diverse data. For each MuJoCo task, the experiments are carried out with three data combinations: 1) 10k expert state-action tuples, 5k expert state-action tuples, 3) 5k expert and 5k medium state-action tuples, and 4) 5k expert and 5k random state-action tuples.
Task
Exp. (5k) & Rand. (5k) Exp. (5k) & Med. (5k)
Exp. (5k)
Exp. (10k)
Walker2d Hopper
2973.88
3613.49
2858.29
4990.57
1891.55
2135.07
1885.76
2604.59
Ant
1960.05
3879.48
1978.08
3940.30
Half-Cheetah
1113.75
4888. 64
1714.30
4975.17
sample sizes. Table shows the average returns (over 5 random sceds) under different expert
sample sizes with the fixed number of medium data (50k). It corroborate analytical results that with relatively sufficient data coverage of the empirical expert behaviors, the performance is dominated by the expert sample size (combining Theorem 4.2, Theorem 4.3 and Corollary 4.1).
Table 5: Results under different expert sample sizes.
Dataset
5k
10k
20k
50k
100k
Half-C -hfe-Chheah
4753.9
4978.2
5206.5
7865.5
10930.1
11121.9
Hopper
1989.9
2273.4
2507.8
2991.8
3571.1
3566.4
Walker
3439.9
3632.2
4753.3
4982.4
4977.8
4991.3
Ant
3375.4
3866.5
3968.9
4385.7
4797.5
4910.6
Comparison to MOMAX. To demonstrate the poor performance of the naive approach using a simple combination of IRL with model-based offline forward RL (MORL) method, design baseline directly using a state-of-the art MORL method, COMBO (Yu et al., 2021), 1), in the inner loop of MaxEnt IRI. (Eq. (1)), called MOMAX. As shown in Figure MOMAX does not work well in
14

---

Published as a conference paper at ICLR 2023
Walker2d Iterations
Hopper Iteration
Ant Iterations
Halfcheetah iterations
Average Return
Average Return
Average Return
Average Return
2
N
MUNNAA CLARE HOMAX
CLARE MOVAX
CLARE NCMAX
CLARE MC Kaxtrt
Steps
204
Steps
Steps
304
Steps
Figure 4: Comparison to MOMAX. Each experiment uses 10k expert and 10k medium state-actions.
Walker2d Iterations
Hopper Iteraaions
Ant Iterations
Ha Hflfhheetah Iterations
Average Return
Average Return
Average Return
Average Return
2
2
Hovy
With un givino Witeoul wuggiing
L
With weighting
With weeghin
WiTh suapptinn WiLT wed gfrci1
Withost waightng
W Whthost asgg
Steps
Steps
Steps
Steps
Figure 5: Ablation study reward weighting. Each experiment uses 5k expert and 50k medium
state-action samples from the D4RL benchmark.
these continuous control tasks. It reveals the challenges repurposing the online IRL methods in the offline IRL setting.
Ablation study of reward weighting, Fig. 5 shows the impact of reward weighting on performance. CLARR basically reduces to MaxEnt IRL. with reward weighting and thus not deal with the extrapolation error effectively in offline learning. Fig. 5 also demonstrates that the conservative reward function can stabilize the training.
A.4 COMPUTATIONAL COMPLEXITY
We implement the code in PyTorch 1.11.0 server with 32-Cores AMD Ryzen Threadripper PRO 3975WX and a Intel GeForch RTX 3090 Ti. For all tasks, CLARE converges in one hour (around 10 iterations with total 50k-100k gradient steps).
B PROOFS
In this section, we provide detailed proofs of main results Section 4.
B. PROOF THEOREM 4.1
This proof is built on that for Ho Ermon (2016, Proposition 3.1).
First, it follows from Eq. (4) that
[( a (T) + ZgE [r(s, a)] - -Bp (5s. - E&s [r(s, a)] + Zov(r)
= -af((m) + Σ
Lo0 (s,a) (s,0" -() 7 - r(s,a) a)+ Zzv(r)
8.a
r(s,a) + Zg(((r =a (T) + 4B Σ (3)(a))) (P(s,a)((s,a) + (p(8)))
Zg
15

---

Published as a conference paper at ICLR 2023
= aT) + Zo (Epr [r(s,a]] - Bpg [r(s,a)] + 2t(r))
(denoting p' (8, ((5
Kg
where the last equality holds due to
ZB = + E, Br [3(,a]]
(s p (s)..))((s)
2 O.
(from B(s, 2 (p((s,))/00 a) for (s,a) E D)
Thanks to Lemma 4.1, there exists a one-to-one correspondence between and Cc. Thus, we can rewrite
min max I( = min max a (TT) + Z3 [Bp[r(s, a B [r(s, + +())) rER pecr
(13)
PER xell
=L(rrr
It is easy to that R. is compact and convex. Besides, from the proof of Ho & Ermon (2016, Proposition 3.1), Cp is also a compact and convex set. Accordingly, based on the concavity of H (Lemma 4.2), the minimax theorem holds (Du & Pardalos, 2013), and hence have
min max L(p, r) max min L(ørr) PER PECy
PECT rER
max (p) + ZB
min
EA Ep[r(s, a)] - Ept [r(s.))] + ((r))
PECT
rER
max at) Zavi* (ø' - DECc
(from the definition of convex conjugate)
max ( + (p, DECT
(14)
Additionally, denote f" and fr as
r" E arg min max L(p, r), 0" E arggmas max afl 2z PER PECf
(15)
PEC,
Due to Eq. (14), (r", is easy to see that policy n" (that corresponds to (²) satisfies n" E arg max M(T, r"), therehy completing the proof.
"r",p"" (²)) is saddle point of L, and thus p E arg maxpee,,, L(p,r*) r²). By Lemma
B.2 PROOF OF THEOREM 4.2
We present the following two lemmas before our main result.
Lemma B.1. Denoting p1 (x,y) = qı (a))q1((ye and p2(œ.8) = ge(z)g2(y|z) as two joint dis- bound the total variation distance (TVD) between pi and pa
tributions over finite spaces,
as
DTv(PI,PH P2) S Ex)g((x) [Drv(a((()) + Drv(qi
(16)
Proof. The proof is straight-forward:
Drv P2)
Σ
(z v) p2(2,
2
2,9
Σ
( (2())))(((222
2
2,9
(2(2))2(112)
2 2.9
< Σ
a (x) (x)=) (<(1)=) + Σ (2(1) (x(=)) (2(x))
2,9
2.9
16

---

Published as a conference paper at ICLR 2023
Σ - + 1 Σ (z(x)) (2(x)) Σ (2(2)2)
2
Ea (x) (Pr((1(V)))) () (2 + D-v(Dr 42),
(17)
where the last equality is obtained due to E, ( =1.
Lemma B.2. Suppose that we have two Markov chain transition distributions (s'ssss and (s'ss),, and the initial state distributions are the same. Then, for each h .1.2....) the TVD of state marginals in time step h is bounded as
h-1
DTv(F1-P2)
S Σ
[Drv (2(1))))(((11)))
(18)
h'=0
where ps = Pr(sh = s 1.) for 1.2.
Proof. First, we have
ps())= p3(s)
Σ (isssss)pp (5') 1p (d)
Σ M A (s) (011
Ti(sss'pp (5) (s)) Ti
Σ (i(()ss) pi h- (5') 1/2 (s')
+ トリ - (s) (sls') - -
-T2(3)6))
Σ Tis p (5') (
F 1-1 - T2(616')
(19)
Thus, we can write
Drv(it (pf, pb)
2 Σ (s)) - p+()
2
Npp - 1 |
- T2(} | +
1 Σ Σ Ti(s)s') pT
p2 (ss)
(using Eq. (19))
Σ
E sippp
[Ti(s)s)- [2(3|31)|
Σ
p
(6') - -p Σ (s(s's)
2
2 5'
= -Egmpa-1
Σ [i((ss')) -T((o))))6(0066)))
+ Σ 1 (66) 1- (6')
2
(using Σ, (s(sss))) = 1)
-Eg
DrV (Ti(()s),, Ta(-ss)) + (,
(20)
F Fawnyy
(DTV (Ti())), T2(-1s) (T))
DIv(Pi.P9)
(iteratively using Eq. (20)
4-1
[Div (T((()))TT((1)))
= Emppl'
(due to same initial state distributions)
W=0
which completes the proof.
17

---

Published as a conference paper at ICLR 2023
Observe that Lemma B.1 bounds the TVD of joint distribution by the TVDs of its corresponding conditional and marginal distributions, and that Lemma B.2 bounds the difference of two MDPs' state visitations in each time step by the cumulative dynamics differences. Next, provide the following lemma that bounds the difference between the expert's and learned policy's 's occupancy measures from above.
Cp. denote â as its corresponding stationary policy, i.e.. ft = p(s, a)/ Ew i(s,a)), and p² denote the occupancy measure of under true transition dynamics T. Then, the following holds;
Lemma B.3. For each
Drv(p", ,p . 1-7
1
7 Es Es
a)) + Drv(p, Drv(p" ,P),, (21)
where p" is the occupancy measure of expert policy xB under the true transition dynamics.
Proof. For conciseness, let P1 = 0² and p2 p. Using the triangle inequality, it is easy see that DTv(pI. pB)= S DTv(PI,P2) + Drv(p2, PP) + DTv(D .P))
(22)
where of is the empirical occupancy measure of expert policy x². To bound DTv(PI, P2). denoting (s. a) = Pr(s), s, U₁ T, R. μ)) and p (s, a) = Pr(sh s, ag a | T, A. p) (the difference between them is marked in red), we can write
Drv(PI,
Σ 8,a
(s a) p2(s.a)
n (s) - (1- -(1-))
Σ y (s, a)
2
8,8
h=0
h=0
(using the definition of occupancy measure in Section 2)
20
((((s,)--$(s,a)))
2
5;0
4=0
pi(s,a) 18)))
2
h=0
(1
pi(ss (s)
2
h=0
8
DTv (P((8,0),((3..1))
h=0
1h DTv (p1(6))b((s))
< (1
(23)
K=D
where (s) - Pr(ss = U), p)(s) = Pr(sn = π, u)) and the last inequation holds due to Lemma B.1 (note that (f( = - and (s, = (alss) Denote Ti (s'sss = Σ. T(s', als) and T2(s'(s) = Tss'ss where we slightly overload notations using T(s',ala) = (aals) a)) and 7(s',als) = a(als))(<)ls) We obtain
Drv (1)(1)))
1
Σ
(i('lss) -2((s(s))
2
1
Σ
T(s' als) - T(ss' a|s) Σ
2
"To avoid ambiguity, use (rv a).r3))).))_)) (s, a)) and (r (s). (=)) to denote the TVDs hetween the corresponding state-action distributions and state distributions respectively.
18

---

Published as a conference paper at ICLR 2023
T(s al|s) - (ssss
2 s'a
Drv (T(s' als); T(s', alss)
S Bana(ale)
DTV (T(-)s, a), f(1.s
(seeing ((lss) q (1), (x)), [s as 2i (y/z), and (s) as q2(u1z), and then using Lemma B.1)
Based on that, the following holds:
DTv(PI, (1 - 7)
h DrV (p(()))
(from Eq. (23))
h=0
(1
E (
Drv (T(()1),,T(((3)))
(0)
h=1
M==
(using fact Drv (p)(ss) Drv(u, 14) = 0 and Lemma B.2)
Eana(all)) DTv (T(1s,)), a(s.
(1--))
E s (0)
V==0
(using the above result)
h-1 1
=
(1
h
s (s,a)
Drv ((1)8,)))((()..)))
h= 1
h'=0
(noting that (s, ()))a((a)s)
(1 11-21 7) Σ Drv ())
p2 (s.
8,0
(expanding the expectation and rearranging terms)
=7(1 7) Σ
Drv (T((s.
1 (s.
8,a
h=
M'=0
A
=7(1 - 7) Σ
Drv (T(()).)),T((1s..))
Σ
h
h-1
Σ
"h
a)
8,0
R==0
(B is derived by rearranging the terms in A)
=7(1 ((1--)) -) Σ Drv (T(ls.a).T((ss.c) Σ h- P2((sa
1-7
(noting that p2(8, (1 -) Ei P (s,
Σ
Drv (I(1ss.), Σ pp)(s,))
3.a
Σ 3.4
D)y (), Σg
h=1
Drv (T), T(11s, (((s))
E Eg
(24)
Substituting Eq. (24) in Eq. (22) gives the desired result.
L
Denoting pa as the occupancy measure of ग under underlying dynamics model T, We can write
B - J(p))
Σ
- Σ (s) a) (from the definition)
2,0
19

---

Published as a conference paper at ICLR 2023
Σ -sssa))pp"(sa R(s,a)
6,0
(8, a) 0² (s,a))
(due to [R(s, a)l S 1)
= 2Drvv (p",
(25)
Then, based on Lemma B.3, the desired result in Theorem 4.2 can be obtained by combining Eq. (25) with Eq. (21).
B.3 PROOF OF THEOREM 4.3
C Recall that a(s,a) = C. Drv DDDDD(((i)))) (Tvv(((-1s,a) a) al)s. We define
f() = Ex,arplc(s, a)] + 2Drv
Σ (s,,))((ss a) p(s,a) 8F (s)
(26)
Thanks to Lemma 4.1, minimizing the RHS of Eq. (7) is equivalent to the following problem:
min
f(p).
(27)
A)
Let 8(8, a) a(s,a) - pF(s, a). Then, Problem (27) can be transformed to the following one:
min Σ cds ()) + ((s,a)
(28)
5,
s.t. Σ 8(s a) =0
(29)
8(s, a) 2 (s) SES, A.
(30)
For conciseness, rewrite Problem (28)-(30) the following form:
22
min g()) = Σ
c. + |
(31)
s.t.
0
(32)
i=1
8 2 -p E [n]]
(33)
where corresponds to state-action pair, = |S| |A|, [n] = (1) and 5 = {6, i [n]].
Due to Eq. (32) and Eq. (33), [m] can be divided into two disjoint sets, (6) {i E [m] 8, > 0}
and (0) = {i E [m]] 8, I 0} (8) iff all 8, 0). Thus, we can write
g
Σ (c + 1)6; + Σ( 116-1)6..
(34)
(E)1(8)
JEN₂(0)
For any 8 meeting Constraints (32) and (33), we denote 8' (which should be 8/4 if written in full)
satisfying o3 -c; emin 2] pf for all N2 8% for all i (8) and
8 = E)Er((0; 1/c, - cirin > 2] P (8)) for all i E Nm) where Nmin()) {i e
(6) i E arg min ce and emin = min G. Then, we have
g(6') = Σ (c: + +188 +
Σ
(cj - 138;,
(8)
(ENx(8)
Σ
(a + 110 +
Σ
(cr + 1)8%
Σ
(c, - 1));s
(8)
jEAs(()
Σ
+ (1+1)8 +
Σ
(Gr + 1)69,
Σ
1/g
min > 2] (c; 1)7
(8)
(4) (8)
(8)
20

---

Published as a conference paper at ICLR 2023
= =cerin +1)
Σ
Ile3 eminn > 2] -
Σ
1// - -T 2] (eg 1) pp
(EN2(8)
jENa
(due to 8/ = 0)
= Σ (CN
1fc,- - cmii
-ctii > 2] (chiiu - Cj + + og.
(35)
Regarding g(0), the following holds:
2(0) = E (a + +1),+
Σ
(c) (19-1)6) 1)03
(6)
JENa(5)
2 Σ (cTin + +1)d; +
Σ 1(03- - min > 2| (cg - -1)6
(6)
JENa(6)
+
Σ
1 10g
min 2] (c- - 1)63
(8N
Σ
1/c c
min > 2] . (cTriu + +105 + Σ
1/c ci 2) -1)8
=
(8N2(8)
jENa(1)
+ [c]riu +1)
Σ 8, + Σ 1 ci > 2] dj
(8)
jEN2(8)
1/cg-
min S 2] (c3 - 1)6
jEN2(8)
(adding and subtracting EJENS(8) 1/c - -cqin > > (cdtt + 1)6,)
= Σ
1/e-
min > 2| (ej - min - -2)0;
JEN2(1)
+ (cmii +1)
Σ 8, + Σ 1(c3 cpinn > 2).00
(B)
jEN2(8)
1/e=- ctrin
< 2] . (cj- - 1)63
)EN2(8)
2 Σ
1/e
min > 2] . (cfrin Cj + +2)0
JEN2(1)
+ (cnin
+1)
Σ 8z 52+ Σ 1[c3 cpiu > 2) 2).5)
(B)
:ENx(8)
1 1/c-- ctiin
< <2] . mm + +105
(noting that 8, S 0)
)EN2(8)
Σ JEN2(8)
1/c quin > 2] (ctuin Cj + t2)
+ (cnin +1)
Σ 8z 5,+ Σ
-cqiin > 2] + 1/;
2
-cquin I =2) 55
(BN1(8)
jENa(8)
Σ
1/c qmin > 2] - (crrin cj + +2)Ø5- + (crin +1)
Σ (E)
6 +
Σ
63
(E
(
(due to Constraint (32))
9666")
(due to Eq. (35))
21

---

Published as a conference paper at ICLR 2023
Denoting g = {6 RISH-A s.t. (32) and (33)), have the following fact:
8/41 = arg min g 6E9(N)
(36)
where ((N)) = (6 G (i((6) = N₁ and N2(0) = [x]] Wi]. Due to Eq. (35), we have
min g(0) min dEg
((0,)
min
Σ
[ em (chin
Cj + 2) p..
(37)
Nicla]
ci
jEiNILAG
Let emin = min Cj and Ni =
[n] : )min The following fact is true:
9(6v;) 9(0'5))
E
1
mmin > 2] (c Cj +)+
Σ
1cyy min > 2] (cmiin Cji + 2)
J'ENJRW
Σ
1 - min > 2] (cmin - -Cj +
W
Σ
1cy c > 2] (cmin cg + 2) (due to c < <cqiin)
J'ENJAAA
20,
(38)
where the last inequality holds because {j [n]/M] cj ct > 2} is subset of {j' [nl//1 Cyr min > 23. Thus, 8" = =6y; = minseg g(0), and we can express 8* as
if((s, a) S minu
8 (s =
(s)
if if((s, ()). > min + otherwise
(39)
0,
where Wmin -N. Due to 0(8, a) + p a) = p(s, we obtain the optimal solution of Problem (27) as follows:
Eveor Maafo' 200" + 7 (8,
ifc(s,a) < min
Wmil
as)
O,
if ff(ss a) > min + (40) otherwise
(s,
thereby completing the proof.
B.4 PROOF OF COROLLARY 4.1
Because c(s, a) > min when (s a) 0. if (s, a) = 0, then (s,,sa) 0 holds. The desired result can be easily obtained by seeing 3* (8, " (s, a) as 8t (x, a) in the proof of Theorem 4.3.
B.5 MINIMIZING A CHI-SQUARED DIVERGENCE
The f f-divergence between two distributions P1 and P2 is defined as
/ /
((())
Dy(pillo)) E,p2
P1
sup Ex-m [x(X)) Ex-oa [s*(((X)]
(41)
where f is the convex conjugate. The x² -dvvergee is the f-divergence with f(x) = (x 1)2 and
(") E i.e.,
9(X)2 + 9(X)
(p,, (p1,p2) =sup Ex lo(X)]
Ex
(42)
22

---

Published as a conference paper at ICLR 2023
By interpreting 9 -r and X (s, a), the following holds:
x P2) = suu ((s,a)~p2 [r(s, a)] - Bi (r(s, 4 F(sa)pps [T(
(43)
Thus, using convex reward regularizer (b(r) r² enables CLARE to minimize a diveeregee between the target policy and learned policy, i.e., maxpec; aH( Z50 (²)
48
23


Reviews:
[{"review_id": "LkA3XWQq2Y", "reviewer": "Reviewer_CxsR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the problem of reward extrapolation in inverse reinforcement learning (IRL). The authors propose a modification of the MaxEntIRL to properly address the problem by leveraging an additional dataset collected by a behavioral policy and properly adjusting the loss function, introducing additional terms for assessing the reward function on the samples of this dataset. The loss function depends on a auxiliary function $\\beta(s,a)$. A theoretical analysis is presented to show the guarantees of the approach as well as proposing a suitable value of $\\beta(s,a)$. An experimental evaluation is presented showing the advantages of the proposed approach over several baselines on multiple Mujoco environments.", "summary_of_the_review": "Overall, the paper provides a relevant contribution to the IRL literature. The reward extrapolation is a relevant problem in IRL and the approach proposed by the paper is novel and has shown to be effective from both the theoretical and empirical sides. I suggest the authors to expand the explanation of the loss function of Eq (2). My evaluation of the paper is positive.", "strength_and_weaknesses": "**Strengths**\n- The paper addresses a very important issue of IRL\n- The paper provides both empirical and theoretical contributions showing, in both cases, the benefits of the proposed approach\n\n**Weaknesses**\n- The rationale behind the loss function of Eq (2) could be explained better\n- The \"Related work\" section should include additional papers (as detailed below)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "3I75jOsui4", "reviewer": "Reviewer_yF3G", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "CLARE uses a learned dynamics model and conservative policy objective in order to tackle the problem of learning a reward function from a dataset of expert trajectories, i.e. \"offline IRL\". Their contribution tackles both the reward extrapolation problem in offline IRL as well as improves generalization of the learned policy. The first well-known issue is tackled using the conservative reward update and a notion of safety constrained policy update. Secondly, the generalization to unseen states is handled by learning a dynamics model. Additionally, the authors provide a theoretical framework for balancing the exploitation-exploration tradeoff with a derived upper bound on the return gap between the learned and expert policies. ", "summary_of_the_review": "Overall, this is a nice paper with empirical and theoretical contributions and I would lean towards acceptance. ", "strength_and_weaknesses": "Strengths:\n- Paper is well written - especially in tackling two related, but still distinct issues in offline IRL, it is easy to follow the contributions and their exposition is well done.\n- Along the above point, the combination of different techniques to improve overall performance of the offline IRL algorithm is quite a reasonable one, i.e. combinging model-based conservative policy learning into one technique.\n- Results are very strong and convincing on known benchmarks\n- The pointwise weight parameters is a novel and interesting contribution.\n\nWeaknesses:\n- Why is the reward regularizer chosen as r^2. I could not find any further elaboration on this, and the rest of the section on regluarizers discusses the conventional policy regluarization as a KL divergence. Theorem 4.1 states the reasoning behind various reward regularizations, which I could just be overlooking as I am not familiar with the cited works, however, it would be good to hear more about the reasons r^2 was chosen in terms of why it performs well with respect to the datasets used. \n- As far as I can tell, the generalization ability coming from the policy updates under the learned dynamics model is not a novel contribution. I think it would benefit this paper's visibility on the overall contribution if they reframed their approach as being only the conservative reward function learning which is compatible with an off-the-shelf offline RL method rather than a two step update algorithm. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0EXshdA5xB", "reviewer": "Reviewer_Gh9M", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the reward extrapolation challenge in offline inverse reinforcement learning. The setting considered is access to a dataset of expert demonstrations as well as an offline dataset, collected by a behavior policy that is not necessarily an expert policy. The authors propose a conservative IRL algorithm called CLARE that integrates conservatism into the learned reward function and exploits learned transitions. They provide some analysis of their approach and show improved empirical performance over prior methods on continuous control tasks.\n", "summary_of_the_review": "Despite the improved empirical performance, I did not find algorithmic design and theoretical analysis to be significant. The paper discusses well-known insights and does not provide any new insights.", "strength_and_weaknesses": "**Strengths:**\n\n- The topic of IRL and issues related to covariate shift and generalization are important and of interest to the community.\n- Empirical results show significant improvement over prior inverse offline RL methods.\n\n**Weaknesses:**\n- Novelty and contributions are limited. The benefit of combining expert and offline data via pessimism on rewards and model-learning is already introduced in prior work such as Chang et al. 2021. The multiplicative weights are also used by Yu et al. 2021. No new insights are revealed, e.g. the discussion is limited to the well-known covariate shift issue in offline learning, penalizing rewards of less visited state-actions pairs, etc. \n- Theoretical results are very weak. An important issue is that while the paper focuses on issues such as covariate shift and generalization in high-dimensional and/or continuous environments, in several places state-action pair counts are used such as empirical data distributions based on counts (below Eq. (2)) and Assumption 4.1. These are not aligned with the high-dimensional problem considered where typically $|D(s,a)| = 0, 1$. Indeed, Assumption 4.1 can only hold in tabular using concentration inequalities. Furthermore, the statements are not very interpretable and do not give bounds on standard quantities such as suboptimality of learned policy with respect to target policy expressed in terms of sample size, data coverage, and hypothesis class cardinalities. Theory is also not very insightful e.g. Theorem 4.2 is very similar to Theorem 4.4 in Yu et al. 2020.\n- I don't think it's meaningful to discuss exploration (e.g. in the abstract, Figure 1, Introduction, etc.) in the context of purely offline learning from datasets. Running iterations on a learned model is not exploration.\n- Generally, the paper makes several strong claims that are not supported by either the analysis or experiments. For example, it is not correct to say that optimizing the upper bound leads to balancing exploration-exploitation. Or the findings do not show the algorithm \"can effectively capture the expert intention\" which is claimed in the paper. The claim in paragraph 3 of Section 3 is not well-supported \"it is clear that utilizing a learned dynamics model is indeed beneficial due to its capability of providing broader generalization (compared to model-free counterparts)...\". The question of generalization of model-based vs. model-free approaches is still debated and unclear.\n- Empirical evaluations can be improved by providing ablation studies such as showing the impact of expert data sample size on performance, the impact of conservative reward weighting, and accessing the impact of offline data on performance. Finally, tests on environments with high-dimensional image inputs on Atari strengthen the work on the empirical aspects.\n\n\n\n\n**References**\n\nChang, J., Uehara, M., Sreenivas, D., Kidambi, R., & Sun, W. (2021). Mitigating Covariate Shift in Imitation Learning via Offline Data With Partial Coverage. Advances in Neural Information Processing Systems, 34, 965-979.\n\nYu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine, S., & Finn, C. (2021). Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34, 28954-28967.\n\nYu, T., Thomas, G., Yu, L., Ermon, S., Zou, J. Y., Levine, S., ... & Ma, T. (2020). Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33, 14129-14142.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-hodWw6cDm", "reviewer": "Reviewer_5Qn6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a model-based reward learning algorithm. The key idea is to smartly balance the use of the expert data and the low-quality data in learning the reward function. Theoretical analysis is provided to justify the weight for each low-quality data. Empirical study also verifies the efficacy of the proposed algorithms.", "summary_of_the_review": "see above", "strength_and_weaknesses": "Strength: \nThe paper is well written and easy to follow. I in particular like the baseline MOMAX. This is the simplest way to use offline data in the IRL pipeline and comparing with this baseline makes the empirical study more convincing. The proposed algorithm outperforms baselines by a large margin.\n\nWeakness:\n1) It is not clear why the y-axis in Figure 2 means. I presume it is the performance of the learned policy, since the algorithm needs to alternatively learn \\pi and r. But I do not understand why BC is included, since it is not a inverse RL algorithm at all.\n2) Table 1 may benefit from having second order (e.g., variance) and averaged statistics (e.g., expectation) instead of just the highest score", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster