Learning from Sparse Offline Datasets via Conservative Density Estimation
Authors: Zhepeng Cen
Zuxin Liu
Zitong Wang
Yihang Yao
Henry Lam
Ding Zhao
Venue: ICLR 2024 poster (2024)
Keywords: offline reinforcement learning
stationary distribution correction estimation
Primary area: reinforcement learning
TL;DR: We develop an offline RL method Conservative Density Estimation for scarce data and sparse reward settings.

Abstract: Offline reinforcement learning (RL) offers a promising direction for learning policies from pre-collected datasets without requiring further interactions with the environment. However, existing methods struggle to handle out-of-distribution (OOD) extrapolation errors, especially in sparse reward or scarce data settings. In this paper, we propose a novel training algorithm called Conservative Density Estimation (CDE), which addresses this challenge by explicitly imposing constraints on the state-action occupancy stationary distribution. CDE overcomes the limitations of existing approaches, such as the stationary distribution correction method, by addressing the support mismatch issue in marginal importance sampling. Our method achieves state-of-the-art performance on the D4RL benchmark. Notably, CDE consistently outperforms baselines in challenging tasks with sparse rewards or insufficient data, demonstrating the advantages of our approach in addressing the extrapolation error problem in offline RL.

Published as a conference paper at ICLR 2024
LEARNING FROM SPARSE OFFLINE DATASETS VIA CON- SERVATIVE DENSITY ESTIMATION
Zhepeng Cen', Zuxin Liu', Zitong Wang", Yihang Yao', Henry Lam", Ding Zhao' Camegii Mellon University, 2 Columbia University {zcen, zuxinl, {zw2690, henry. 1am] @collclluoo umbia edu, dingzhace cmu, edu
emu.edu
ABSTRACT
Offline reinforcement learning (RL) offers a promising direction for learning poli- cies from pre-collected datasets without requiring further interactions with the environment. However, existing methods struggle handle nut-of-distribution (OOD) extrapolation errors, especially in sparse reward or scarce data settings. In this paper, we propose novel training algorithm called Conservative Density Estimation (CDE), which addresses this challenge by explicitly imposing con- straints on the state-action occupancy stationary distribution. CDE overcomes the limitations of existing approaches, such as the stationary distribution correc- tion method, by addressing the support mismatch issue in marginal importance sampling. Our method achieves state-of-the-an performance the D4RL. bench- mark. Notably, CDE consistently outperforms baselines in challenging tasks with sparse rewards or insufficient data, demonstrating the advantages of our approach in addressing the extrapolation error problem offline RL. Code is available at https: /ttps:///gihuse htt.hbttthu.....o com/ czpl w/tal1/ccce cde offline rl
INTRODUCTION
Reinforcement Learning (RL) has witnessed remarkable advancements in recent years (Akkaya et al. 2019] Kiran et al.) 2021). Nevertheless, the success of RL relies on continuous online interactions, resulting in high sample complexity and potentially restricting its practical applications in real-world scenarios (Levine et al., 2016; Gu et al. 2022). As compelling solution, offline RI. has been brought to the fore, with the objective of learning effective policies from pre-existing datasets, thereby climinating the necessity for further environment interactions (Fu et al., 2020 Prudencio et al. 2023).
Despite its benefits, offline RL is not devoid of challenges, most notably the out-of-distribution (OOD) extrapolation errors, which emerge when the agent encounters state-actions that were absent in the dataset. These issues pose significant hurdles when learning policies from datasets with sparse rewards or low coverage of tttt-actioon spaces (Levine et al. 20220. To address OOD estimation errors in value-based offline RI,, current efforts primarily revolve around two strategies: pessimism-based methods (Xie et al. 2021a Shi et al., (002) and the integration of regularizations (Kostrikov et al.) 2021a). However, these approaches hinge on assumptions of the behavior policy. In addition, many works push policy to behavior policy to achieve pessimism (Fujimoto et al. 2019 Shi et al. (2022). which is more challenging to select the level of pessimism when the data distribution estimation difficult (Liu et al. 2020: Xie et al.) 2021a) (e.g., in high- nigh-dimensional state-action space), while regularization methods may struggle with the tuning the regularization coefficient (Lee et al.. 202T
Lyu et al. (022). As such, striking the optimal balance of conservativeness remains challenging goal particularly in sparse-reward settings.
Recent attention has been drawn towards an alternative method that employs importance sampling (IS) for offline data distribution correction (Precup) 2000: Jiang & Li 2016). Among these, Dis- tribution Correction-Estimattin
(DICE)-based methods have garnered substantial interest, which uses a single marginal ratio to reweight rewards for each state-action pair and has a relatively low estimation variance (Nachum al. 2019b Zhang et al. 2020 Lee al. 2021). DICE provides a aghossir-agnostic estimation of stationary distributions, presenting a more direct approach for offline learning. However, DICE-based techniques rely implicit assumption of the dataset's

---

Published as a conference paper at ICLR 2024
bj Li et al. 2022). otherwise the stationary distribution support mismatch between the dataset and policy can cause an arbitrarily large IS ratio, resulting in unstable training and poor performance, which can be significantly severe with insufficient data.
concentrability Munnn 2007 Xie et al.) b0
To address these challenges, we introduce a novel method, the Conservative Density Estimation (CDE), that integrates the strengths of both pessimism- based and DICE-based approaches. CDE employs the principles of conservative Q-learning (Kumar et al.) 2020) in a unique way, incorporating pessimism within the stationary distribution space to achieve theoretically-backe occupation distribution. On the one hand, CDE does not rely on Bellman update-style value esti- mation, favoring a direct performance in sparse reward scenarios. On the other hand, by constraining the density of the stationary distribution induced by OOD state-action pairs, CDE significantly enhances performance in data-limited settings. This stands in contrast to the significant performance degradation observed in baseline offline RL. methods with diminishing dataset sizes, as CDE maintains high rewards even with only % trajectories in challenging D4RL tasks (Fu et al. 2020
conservative
stationary distribution correction that improves
We introduce the first approach to explicitly apply pessimism in the stationary distribution space. Notably, CDE outperforms conservative value learning-based approaches in sparse reward settings and demonstrates superior performance over DICE-based methods in handling scarce data situations.
2. We present method that automatically bounds the concentrability coefficient without resorting to the common concentrability assumption (Rashidinejad et al. 202T; [Shi et al., 2022] Zhan et al. 2022 underlining its robustness in managing the OOD extrapolation issue inherent in offline RL.
3. We demonstrate the resilience of CDE in maintaining high rewards even with significantly reduced dataset sizes, such as 1% of trajectories, while prior methods fail. Therefore, our method provides viable solution for real-world applications where data can be scarce or costly to obtain.
2 RELATED WORK
Offline RL with regularization or constraints. To mitigate OOD issues, Q-value-based methods are often enhanced with regularization or constraint terms (Levine et al. 2020 Prudencio et al. 2022). These techniques restrict the learned policy's 's deviation from the behavior policy in the dataset, whether through explicit constrained policy spaces (Fujimoto et al., 2019) or regularizers in the objective (Wu et al. 2019 Peng et al. 2019 Kumar et al. 2019 Nair et al. 2020 Fujimoto & Gu 2021]. Alternatively. value regularization is employed to yield lower estimates for unseen states or
actions, resulting in conservative policies (Kostrikov et al. 2021a| Kumar al., 2020). However,
those methods may suffer instability from approximation error when learning value with Bellman
update iteratively (Fujimoto et al., 2018 Fu et al. 2019 Brandfonbrener al., (02T], often failing
in sparse reward settings even with expert demonstrations. Meanwhile, the reliance on heuristic regularization can lead to overly conservative policies and degrade performance.
Offline RL with marginal importance sampling. The DICE method represents class of ap- proaches that directly address distribution shift using marginal importance sampling, offering reduced estimation variance compared to naive importance weighting (Precup) 2000)) These methods reframe the learning objective as maximizing expected reward, using the primal-dual correspondence between value-function linear programming and distribution optimization (Nachum et al. 2019b; Nachum & Dail 2020). DICE calculates the importance ratio using either a forward method that minimizes the residual error of the transposed Bellman equation (Zhang al. 2020), or backward method that optimizes the value function via duality (Nachum et al. 2019a).. Some variations add a regularization to the objective function, yielding closed-form solution for the importance ratio (Nachum al.) 2019b] Lee et al. 202II. Despite their ability to provide unbiased policy evaluation, DICE-style methods yield arbitrarily large importance ratios when the dataset lacks sufficient state-action space coverage, a challenge particularly acute in scarce data settings.
3 METHOD
3.1 PRELIMINARIES
We formulate reinforcement learning problem in the context of a Markov Decision Process (MDP) M (S,A, T, r, 7, Po), where S is the state space, A is the action space, T S x A S [0,
2

---

Published as a conference paper at ICLR 2024
specifies the transition probability T(s|ss, a), r X A R is the reward function, Y is the discount factor, and Po : S [0, 1] is the initial state distribution. The policy т : S x A + [o. 1] maps from state to a distribution over actions. Given policy オ, consider the trajectory {so, 40, $1, (1, .i,-... sampled by π. i.e., 8p ~ Pn, 4t ~ TT- $t+1 ~ T(-)st 0t), the stationary state-action distribution is defined as dd (s, = (1 - 7) -t= 7¹ Pr(st 8, 8, a). The goal
of RL is to learn a return- mauummaaxiiiztion policy π arg max, ET~ 1I== 2'r(st, a.), which is equivalent to reward maximization (Puterman 2014): ' arg max Bs [n(s.
In offline RL, the agent learns the policy from a pre-collected dataset D = {(s,, aq, rs, +))};= For simplicity, we denote the empirical state-action distribution of offline dataset as dD, The DiCE-style
methods apply marginal IS to estimate the expection of certain function g: Es [g(s, a) =
E. aa t (s,a) g(s, a)] with d" dd as target and proposal distributions. The IS estimation can thus be approximated by sampling from offline dataset.
d (s.a)
3.2 CONSERVATIVE DENSITY ESTIMATION
In this section, we present Conservative Density Estimation (CDE), which aims to learns a policy that induces the distribution with conservative density in unseen state-action region. We first consider f-divergence regularized policy optimization problem (Nachum et al., 2019a b):
max E. (r() - aD (dF(d)))P) d 20
s.t. Σ d" -))p((s) + (s), (1)
where Dy(d"ldd) E, if d" (0.0) }] is the f-divergence between two distributions, is the (2a)
hyperparameter of regularization, T.d"(s) Ew,a' T(sss', a a'dd" (s', a') is the transposed transition operator: Here we adhere to state-wise Bellman flow constraint as [ee et al. (2021) to incorporate the stochasticity of action distribution next state a' (-1s') since the state-action-wise constraint can lead to overestimation for 'lucky' samples (Kostrikov et al. 2021b) and instability during training. Particularly, we have following assumption on f function selection:
Assumption 1. The f function in diivergeece is strictly convex and continuously differentiable, and (x) 2 0, Va E IR.
The previous DICE methods (Nachum et al. 2 Nachum & Dai 2020 Lee et al. 202 transform constrained optimization problem to unconstrained one in Eq. [ by Lagrange or Fenchei-RRokaffel duality and evaluate the unconstrained objective by marginal IS with d as proposal distribution. However, one implicit assumption behind DICE methods is that the support of dataset distribution is wide enough and otherwise the density of unseen vate-actions in dd can be zero arbitrarily small. Therefore, when the support of dl mismatches dP, there will be a large extrapolation error for OOD actions and variance in IS estimation. Meanwhile, the f-divergence regularization is enforced on the support of data distribution and approximated by single or several sample points, failing to serve as an effective supervision to explicitly reduce extrapolation errors for unseen state-actions.
To overcome the above issues, we consider a new constraint on the density of d' (s,a) by p(s, a): d' (s, a) S eq((s,a) Vs, supp(n), where juss, a) is distribution OOD state-action pairs, i.e., supp(n)) n ") = 0. The new optimization problem is formulated as
max Eg- (r(s, a)] aDy (cP)
d' 2020
s.t. Σ d" (]s,a)=(0 (1- - T.d T,d'
(3)
d' d' a) < cp(s, a). Vs, a E supp(u).
(4)
The corresponding unconstrained problem is max min L(d v, A), where aD, (d lcd) - Ex [Ass, a) (d" /p(s, a)
L(d", v, A) = Ex- [A(s, a)] + (1 Ep
e)
(5)
and A(s,a) := r(s, a) + "
- u(s) is regarded as advantage function if we interpret v(s) as the V-value of state The derivation is attached in Appendix A.1
In practice, we restrict the state marginal of μ to match the state distribution of dataset ( as previous OOD querying methods (Kumar et al. 2020: Kostrikov al. 2021a Lyu et al. 2022) and
(s)
3

---

Published as a conference paper at ICLR 2024
shrink the OOD region to unseen actions with existing states. Given state in dataset, suppose there exists n. actions a!")..... ..... (n)) such that (s, a))) E DD i= ...... we define the set of OOD actions as (SoD(s) := {a min; ||a -I 2 Aa). See more details in Appendix B.2. We further adopt a uniform distribution n* (als) over the unseen action space AOoD(s) as the policy of i... ps a) (s))))((( (alss) We want to emphasize that our method is also compatible with other OOD sampling distribution with proper inductive bias.
3.2.1 POLICY EVALUATION AND IMPROVEMENT
Based on the unconstrained objective in Eq.(5) ( we first adopt marginal evaluate policy given its stationary distribution. To avoid the support mismatch issue, we consider new proposal distribution (s(s,, a) Çd the mixture coefficient. Therefore, the support of new proposal distribution can cover the target distribution d=. We further replace the original f-divergence regularizer by Dy(d"Ild?) to constrain the density of both OOD and in-support state-actions. Besides, we substitute the importance ratio w(s,a) .) = d= (s.a)/ a)/aa((aa (s, a) for d= as dd is fixed. The new objective function
(s. a) + (1 - () a) in importance sampling, where & E (0, 1) is
(w, v. A) <E [w(s, a) A(s, a) a af(((s a))l + (1 Pn
(6)
+ (1 E, [w(ss ()(,s)))) - X(s,a)) a (w](s a)) + Ex(s, a)),
where ē = r- The derivation is attached in Appendix A. With assumption D the objective in 2 is convex and thus is equivalent to the minimax problem minx20,e max. L(d v, X) by Slater's s condition. Moreover, the inner maximization has a closed-form solution (Nachum & Dai 2020 Nachum et al. 2019b a) and the outer minimization is convex optimization problem. The proofs are in Appendix A.3 A.4.
Proposition 1. With assumption max C' (w. U. A) is
the closed-form solution to inner maximization problem
w² (s(,a)) (A's A(s,a)==A(ssa) a(s,a) a( E supp(n)) M(s, a) denotes regularized advantage function and 1(- is the indicator function.
where A(x, .) ==
Proposition 2. The outer minimization problem minx20,0 L'(w", .. A) is a convex optimization problem. Suppose the optimal solution is (X*,e*) v*), then X has closed-form solution
X X0 a) max (0, (s, a) (?)), Vs, a E supp(u),
where as) = r(s,a) + (E..
2" (s') - 2 (s). The optimal regularized advantage is (s),)))) (E), [minfa]"(O)A (s, supp(u)
(s, E supp(dP)
as
Based on the closed-form relation between stationary distribution d" and value function, we can thus improve the policy by maximizing w.r.t. value function. Since requires the reward r(s, a) and transition T(-ss, a) to compute regularized advantage function Á(s, a), which available only for (s, a) E D, we consider function approximation for both V-value v and regularized advantage A by parameters ip and c. The optimization is in two steps: 1) We first optimize U. by minimizing the value of states in distribution:
min E, [w"(s, (r(s)) a) + (5') vs)s - af (s, (s,a))]+( +1-))+(11))BBB))) Eo Iu (80)]. (10)
2) Then we regress the regularized advantage Ag to the optimal Á in Eq. Specifically, we regress the OOD advantages to (z) if they exceed it and regress the in-distribution advantages to the values from up: (x a) = r(s, a) ". (8). ).). In summary, we optimize the regularized advantage function by following mean squared error (MSE):
min (E (Ao.ss Ap a)) (1 (1, a) (()) (11)
&
and obtain the approximated optimal importance ratios for both in-distribution and OOD state-actions: wn (8, a) =
a)/a).
(12)
By definition, the optimal distribution is d' (3(s))) = x " another constraint to enforce Ex, d² (sx a) (Zhang et al. 2020 See Appendix A.2 for full
(s. a 2 (s.a) In practice, we introduce
derivations.
4

---

Published as a conference paper at ICLR 2024
3.2.2 POLICY EXTRACTION
Finally, we extract the policy from the learned importance ratios. Since one policy is uniquely determined given its corresponding stationary distribution, we extract the policy by minimizing the KL divergence between the stationary distributions of optimal policy and parameterized policy ma. Meanwhile, as we have no access to d= (s, a) in offline setting, we estimate it by d" (s) (als), where the optimal state marginal d" can be viewed as a state distribution of successful trajectories in dataset. The objective of policy extraction is
dd (s. + log
To(als) #a(als) d3(s)
+ log ds
min DKL IId") z min E Ed aawro
log d'
(13)
min E Exnddaans log (s, + Bgnd [DKI.[TO ( (|)b))=(())))] (-16)]],
(14)
where # = (a)ss) + (1- (1-()((aass) is the mixed behavior policy and π T (als) denotes the empirical behavior policy. We will analyze the error induced by state marginal approximation in Theorem The final objective in Eq, [I. consists of two components: the maximizing of w and minimizing the divergence with mixed behavior policy, indicating the trade-off between performance improvement by maximizing the value and conservative learning to reduce extrapolation error.
2
The key steps of complete training proce- dure are summarized in Algo. [ See Ap- pendix[B.2 full algorithm and training details. One noteworthy difference from other actor-critic methods is that CDE up- dates the policy after the value function converges, which improves the learning sta- bility and computation efficiency.
Algorithm Conservative Density Estimation
Initialize value functions Up. Ag, behavior policy xD, policy ma.
1: & policy evaluation and improvement
2: for training iteration do
3: Sample batch {(s,, a4 Ti, s{)3 from D and n
OOD actions (a(
(n) for each s;
The advantages of CDE over previous DICE methods are two-fold: 1) the pro- posal distribution (i.e., dP) has wider cov-
4: Update V-value Uh by Eq, I00)
I
Update regularized advantage Ag by Eq 11
6: Update z0 by behavioral cloning.
erage than d,, which mitigates the support
7: end for
mismatch in importance sampling and pre-
> policy extraction
vents the arbitrarily large importance ratio;
9: for training iteration do
2) CDE produces a conservative estimation
10: Update policy ny by Eq.
of density in OOD region. Compared to previous conservative methods, CDE de-
11: end for
termines the degree of conservatism precisely by optimal À in Proposition mitigating overly pessimistic estimation and loss of the generalization ability (Lyu et al. 2022 Furthermore, CDE disentangles two optimization steps, i.e., learning the value function by convex optimization and extracting the policy from the optimal importance ratio, thereby reducing the compounded error amplified by the interleaved optimization (Brandfonbrener et al. (022).
3.3 THEORETICAL ANALYSIS
CDE adopts proposal distribution with broader support in marginal IS and explicitly constrains the stationary distribution density of the OOD region, resulting in theoretical bound for the importance ratio, also known as concentrability coefficent (Munos) 2007; Rashidinejad et 202T).
Proposition 3 (Upper bound of concentrability ratio on OOD state- acte-actions). With assumption ! the theoretical optimal importance ratio is upper bounded by w" (s, a) S E, V(s a) E supp(μ).
The proof of Proposition]] is in Appendix A.5 should be noted that an unbounded importance ratio methods (Shi et al. 2022) We further
can cause unstable training for importance-sampling
bound the function approximation v= in (12) with following continuity assumption: Assumption 2 (Lipschitz continuity of As(s,a); There is a constant L such that
as, As(s, a')l s L -|a a'lba, Ya, E Aoon Vs E D.
Theorem 1 (Upper bound of function approximated concentrability ratio). Suppose that 1) the action space is d-dim, Le., A Rdd 2) the diameter of A is M, iee, ||an 0z 0zlloc S M, Val, 02 A, and
5

---

Published as a conference paper at ICLR 2024
3) there are at least N OOD action samples from 11 given any state D. When the continuity assumption 2 holds, with probability at least - 6, >> O. we have
f(0) 1 - (A00+
Ma log
!
1/d
N 8
((s,)) E supp(u)
(15)
where & the maximum residual error of OOD regression in Eq. Da the radius of in- distribution region previously defined.
The proof of Theorem iis in Appendix [A66] Proposition Jand Theorem show that CDE inherently bounds the OOD concentrability coefficient. This coefficient is frequently assumeed to be bounded in the variance or performance analysis in both off-policy evaluation and offline RL domains (Rashidine- jad et al. 2021 Ma al. 2022 Zhan et al.] 20222, as unbounded concentrability coefficient can lead to instability during training. As such, the CDE framework shows promise as a potential tool for reducing variance establishing performance lower bounds in future research.
Meanwhile, CDE evaluates the policy within the stationary distribution space, enabling the compu- tation of performance differences between policies based the discrepancies in their respective stationary distributions. Consequently, we can establish the following bound on the performance gap between the learned and optimal policies.
Theorem 2 (The upper bound of performance gap). Suppose the maximum reward is Rmox maxs,a (Ir(s,a) let V* y²(pp) = Essorpo (V* denote the performance given a policy x. For
policy π optimized by Eq.14
14 and N transition data from dp, under mild assumptions, we have
2.maxx
V"o))) - (P0) S 1-7 Drv (d)((()))((())) +EN
(16)
and eN converges in probability to zero the rate N N Yh 0, ie., NItE N N=+00 0 in probability. Here. (s) d' (s) denote the state marginal of dp, d', and (po) denotes the performance of optimal policy.
The full assumptions and proof are in Appendix A.7 The performance gap bound comprises two elements: 1) the discrepancy between the state distribution of the data and the optimal policy, and 2) the number of training samples. The first element stems from the state- mare-naagginl approximation in 41)) during policy extraction. Importantly, this bound explicitly highlights two crucial factors influencing the final performance of the learned policy: the performance of behavior policy x. and the size of the offline dataset. It provides quantitative illustration of how the offline RL problem difficulty increases as the performance of behavior policy degrades and the dataset size decreases.
4 EXPERIMENT
In this section, we aim to study if CDE can truly combine the advantages of both pessimism-bant -base methods and the DICE-based approaches. We are particularly interested in two main questions:
(I) Does CDE incorporate the strengths of the when handling sparse reward settings?
correction training framework
(2) Can CDE's explicit density constraint effectively manage out-of-distribution (OOD) extrapolation issues in situations with insufficient datasets?
Tasks. To answer these questions, we adopt Maze2D datasets, 8 Adroit datasets, and MuloCo (medium, mediumm-expert datasets from the D4RL benchmark (Fu al. 2020p. The original rewards of Maze2D and Adroit tasks are sparse so we adopt the normalized score as the evaluation metric. Since the MuJoCo tasks are with dense rewards, we convert them to sparse-reward ones in the setting of goal reaching. Specifically. we first set the 75- 75-percentile of all returns (the sum of rewards) in the dataset as return threshold. Subsequently, we assign a reward of 0 to all trajectories in the lower 75% of returns, and reward of I to the top 25% of trajectories if the cumulative dense reward surpasses the threshold, which means the agent reaches the "goal". We compare the success rate of different methods Muaroe-MuJoCoo tasks. We adopt "-v1" tasks for Maze2D and Adroit domains and "-v2" tasks for MuloCo domain. To assess the performance under scarce data conditions, we employ a random sampling strategy the full dataset.

---

Published as a conference paper at ICLR 2024
Baselines. We compare CDE method with collection of state-of-the offline RL baselines spanning different categories. These include: 1) behavior cloning (BC); 2) BCQ (Fujimoto et al.
2019) direct policy constraint method; 3) CQL (Kumar et al.) 2020 value regularization
method: 4) IQI. (Kostrikov et al. 2021b) as an asymmetric Q-learning method: 5) TD3+BC (Fujimooo
& Gu 202T) as implicit policy regularization method; AlgaeDICE (Nachum et al. 2019b) as
poliicy-rrddintt-ass DICE method; and 7) OpúDICE (Lee et al. 2021 as an in-sample DICE method. More details regarding the tasks and baselines are available Appendix B.1
Training details. We use tanh-squashed Gaussian policy for CDE's policy π following SAC Haarnoja al., 2018) and uunhsqqashee Gaussian mixture model for empirical behavior policy n² to improve the expressivity for mulli-modalle of offline data from composite policies (c.g.. medium-expert tasks MuloCo) or non-Markovian policies (e.g., Maze2D tasks). Following Lee et al. (02T). ) we adopt soft-chi function f= in f-divergence and thus the (f*)=1 is equal to ELU function (Clevert al. 2015) plus one, which satisfies Assumption and also avoids the gradient vanishing problem for small values when computing importance ratios. For consistent evaluation and fair comparison, we keep hyperparameters the same for experiments in the same domain. evaluate all methods every 1000 training steps and compute mean value over 20 trajectories. The reported scores the average of last evaluation values with 5 seeds. We adopt the scores of baselines if they are reported in original paper. Full experimental details are included in Appendix B.2
4.1 RESULTS ON D4RL SPARSE REWARD TASKS
Table and present the normalized scores and success rate, respectively. See more results in Appendix B.3 CDE consistently matches surpasses the performance of the best baseline iacross nearly all tasks, achieving particularly noteworthy margin of improvement in the Maze2D domain. On Sparse-MuloCo tasks, BCQ, CQL,, and TD3+BC, while achieving high scores in dense-reward settings, display vulnerability to value function approximation errors due to reward sparsity, resulting in inferior performance. The substantial improvement CDE exhibits over over standardRRL-base methods highlights its capability to mitigate compounded value estimation error by leveraging a closed-fon optimal value solution instead of Bellman bootstrapping value update.
Table Normalized scores CDE against other baselines on D4RL sparse-reward tasks. We bold the mean
values that 2 0.99 highest value.
Algae- DICE
Task
BC
BCQ
CQL
IQL
TD3+BC
OptiDICE
CDE
maze2d-umaze
3.8
32.8
5.7
41.5
-15.7
111.0+8.
134.1+10.4
-mediuu
30.3
20.7
5.0
31.0
76.3
10.0
145.2 251.5
1413.1
maze2d-large
5.0
47.8
12.5
58.0
77.8
-0.1
155.7
210.0:13.5
pen-human
63.9
68.9
37.5
71.5
2.0
-3.3
42. 12153
72,1+15.8
commer-human
1.2
0.5
4.4
1.4
1.4
0.3
0.3
1.920.7
door-human
2.0
0.0
9.9
4.3
-0.3
0.0
0.1
7.743.3
relocate-hunde hhman
0.1
0.2
-0.3
-0.1
-0.1
0.3
pen-expert
85.1
114.9
107.0
111.7
79.1
-3.5
80,9+314
1+1.3
hammer-expert
125.6
107.2
86.7
116.3
3.1
0.3
127.0=3.0
126,3+3.4
door-expert
34.9
99.0
101.5
103.8
-0.3
0.0
103.4:2.
105,9
relocate-expert
101.3
41.6
95.0
102.7
-1.5
-0.1
99.7+42
102.6=1.9
Table 2: Success rate (%) of CDE against other baselines on sparse-MuloCo tasks.
Task
BCQ
CQL
IQL
TD3+BC
OptiDICE
CDE
halfcheetah-meel
57.8&13.2
97.62-1.1
76.625.8
41.6217.6
80.0a3.4
82.0s8.6
malker2ddmediun
41.0+11.
17.7+10.4
19.5+42
21.0=16.7
38.4+13.3
53.0
hopper-medium
2.0=4.0
74.0x5.0
0.0+0.0
0.0s0.0
81.0=5.7
85.5+5.7
halfchectah meaichm-miimmmmiii expeert
24.8+9.8
4.2+5.8
95,4+4.2
0.0+0.0
90.8=5.0
95.2+2.9
malkkerd--mdiium-eee exxpett
87.0+13.4
61.6#23.5
94.625.9
32.2s22.
69.
69.0a18.8
97.0a2.
hopper-mediium-exper
20.0+11.
0.0+0.0
94.8+2.8
22.0=10.8
97.4=1.0
97.0=1.4
Another notable observation is that CDE exceeds both AlgaeDICE and OptiDICE in most tasks.
AlgaeDICE falls short because it updates the policy via high-variance policy gradients, opposed
7

---

Published as a conference paper at ICLR 2024
to extracting from optimal importance ratios. OptiDICE exhibits comparable performance in full data setting because the out-of-support issue is alleviated with adequate data coverage on state-action space. To further substantiate the efficacy of the constraints unseen regions, further investigate into situations where offline the data is scarce and the empirical state-action occupation is sparse.
4.2 COMPARATIVE EXPERIMENTS IN SCARCE DATA SETTING
In this section, we examine the performances across datasets of varying sizes. Given the extreme difficulty of the Adroit "-human" tasks due to their high-dimensional space, narrow data distribution, and data scarcity (25 trajectories), we select Maze2D and sparse-MuloCo tasks as our testing platforms. We randomly sample 1%, 3%, 10%, and 30% trajectories from standard datasets to create our sub-datasets. The evaluation process aligns with the full dataset experiments.
umaze
maze2d medium
maze2d
125
150
Nomallzed Scorss
125
75
100
150
100
25-
-25
0-
14
35
10%
30%
100%
17
3%
10%
30%
100%
1%
3%
10%
30%
100%
ha haltcheetah--md um
KalkerZo-mediumg
hopper-medium
80
50
60
30
40
40
23
10
dataset proporionn
dlataset propartion
dataset propartion
OPEDICE
[Ouns)
Figure 1: The results sub-datasets with different dataset sizes.
The comparison results are shown in fig. D we defer the results on medium-expert MuJoCo tasks to Appendix B.3.3] CDE consistently achieves the highest scores across all dataset sizes in Maze2D domain. On the sparse-MuloCo tasks, CDE outperforms most baselines, which may have high success rate with enough data, especially when data are scare (e.g., S 3%). Notably, OptiDICE displays less robustness against scarce data, undergoing a sharp performance drop despite achieving comparable performances full-data setting. Morcover, considering that the original Adroit "-human" " tasks already contain scarce data, OptiDICE also falls short as shown in Table This is because the narrow distribution of scarce data exacerbates the support mismatch problem in OptiDICE, leading to a significant bias in the importance sawpice--mmllling-baee mixed data policy, successfully mitigating large distribution shifts between the stationary distribution support of the dataset and the learned policy.
off- policy evaluation. Conversely, CDE employs
4.3 PARAMETER STUDIES
The results underscore the effectiveness of incorporating conservatism into density estimation in CDE. However, it prompts the question: given that conservative value function estimation is also prevalent in HtandardRRL-baaee to CDE? To answer this question, we conduct parameter study é, which upper bounds the OOD importance ratio by Theorem [ to delve deeper into the relationship between the performance of CDE and the degree of conservativeness distribution mixture coefficient Ç. More results on parameter studies are attached Appendix B.4
offline RI. methods (e.g.. CQL.), why they underperform compared
parameter study. Besides, also run the study on the
Max OOD IS ratio č. We select the maze2d-large large task where the agent's position directly represents the stationary distribution of the learned policy. We contrast CDE and CQL, two representative categories of pessimism augmentation in offline learning.
8

---

Published as a conference paper at ICLR 2024
+F
LE
(a) CDE, E 0.3
(b) CDE, c= 0.03
(c) CDE, 0.003
(d) CQL
The heatmaps agents with different levels conservatism maze2d-large environment. Yellow denotes the high occupation probability. The starting point of each trajectory may vary but the destination (red star) is the same. Smaller indicates more conservative policy. The yellow accumulation points except the destination indicate that the agent is stuck those regions.
Figure 2:
The heatmap in fig reveals that the probability mass at the midway and starting points increases as the level of conservatism escalates. In particular, in fig the agent is trapped at yellow points due to the overly strict constraint. Although stronger regularization can reduce OOD extrapolation error, excessive conservatism can harm generalization and lead to significant performance degradation. Compared CDE, CQL policy is less likely to be trapped in single points but still struggles to reach the destination. There are two principal reasons: 1) the sparse reward setting poses significant challenges to the precise estimation of the value function; 2) CQL. incorporates an additional mini- mization term the unseen Q- to induce pessimism and balances it with penalty coefficient, which lacks a theoretical foundation on the conservatism adjustment. contrast, the CDE employs the closed-fome relation between value and density to explicitly constrain the stationary distribution space, allowing for more precise control over the level conservatism.
Mixture coefficient Ç. We test the performance on sparse-MuJoCo tasks with varying Ç. Fig, shows that the performance is not very sensitive to when closed to but the success rate decreases when Ç is smaller. This is because the theefdde f-divergence regularizer enforces learned policy to remain close to mixture behavior policy, which tends to take unseen actions more frequently when II II]
ç decreases; and 2) a lower can cause the objective in eq.
be dominated by OOD learning, overshadowing in-distribution learning. Therefore, we choose
0.9 for all tasks in practice.
haltcheetall-meduum
walker2d-medium
hopper-medium
50
80
Success rana (%6)
60
60-
40-
0=0
30-
C=0.5
0.5
(-0.7
20
[-0.7
(-0.7
0.0
[-0.8
< 0.9
[=0.9
0.9
(-
Mmw
(-0.85
0.95
50
100
150
200
50
100
150
200
50
100
150
200
treining steps (x10*)
training steps (x10*)
treining steps
Figure 3: The performances with different Ç.
CONCLUSION
In this work, we propose CDE, new offline RL approach, derived from the perspective of stationary state-action occupation. CDE applies the pessimism mechanism on stationary distribution and enjoys the benefits from both fields, which makes it perform advantageously sparse reward and scarce data settings. We further provide the theoretical analysis for the importance-sampling ratios and performance of CDE. Extensive experimental results demonstrated remarkable improvements over previous baselines in challenging tasks, highlighting its practical potential for real-world applications.
The major limitation of CDE that requires the strict alignment of initial state distribution in offline data and online environments, which restricts its performance in inconsistent settings. To overcome this drawback, one future direction can be extending the current framework to goal-conditioned RL setting.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The research is partly supported by the National Science Foundation under grants CNS-2047454.
REFERENCES
Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv: 2019.
David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off- aff-policy evaluation. Advances in neural information processing systems, 34:4933-4944 2021.
Guang Cheng and Jianhua Huang. Bootstrap consistency for general semiparametric m-estimation. 2010.
Djork-Aréé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07285: 2015.
Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep q-learning algorithms. In International Conference on Machine Learning, pp. 2021-2030. PMLR, 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Sara A Geer. Empirical Processes in M-estimation, volume 6. Cambridge university press, 2000.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. arXiv preprint 1rXiv:1912.0 19t.9992........ 2019.
Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205. 10330, 2022.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arxiv preprint arXiv: 18/2.05905, 2018.
Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning, pp. 652-661. PMLR, 2016.
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(6):4909-4977 2021.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning
with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783, PMLR, 2021a.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit gelearning arXiv preprint arXiv:21 2021b.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off off-policy gllaannin via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
10

---

Published as a conference paper at ICLR 2024
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 13:1179-111 2020.
Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference PMLR, 2021.
on Machine Learning. pp.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-13733 2016.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0164; 2020.
Gen Li, Laixi Shi, Yuxin Chen, Yuejic Chi, and Yuting Wei. Settling the sample complexity of model-based offline reinforcement learning. arXiv preprint arXiv:2204.05225, 2022.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch off-policy reinforcement learning without great exploration. Advances in neural information processing systems, 33:1264-1274, 2020.
David G Luenberger. Optimization by vector space methods. John Wiley Sons, 1997.
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2206.0447 2022.
Jason Yecheng Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. Offline goal-conditioned
reinforcement learning via f-advantage regression. Advances in Neural Information Processing Systems, 35:310-323, 2022.
Shuangge Ma and Michael R Kosorok. Robust semiparametric m-estimation and the weighted bootstrap. Journal of Multivariate Analysis, 96(1):190-217, 2005.
Rémi Munos. Performance bounds in I-p-nomm for approximate value iteration. SIAM journal on control and optimization, 46(2):541-561, 2007. 2007.
Ofir Nachum and Bo Dai. Reinforcement learning via fenchel--rockfelllar duality. arXiv preprint a0Xiv:2001.01866, 2020.
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems, 32, 2019a.
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv: (9ti://922.0074,4 2019b.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint (xXiv:2000.09359, 2020.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. gendice regression: Simple and scalable off- off-policy reinforcement learning. arXiv preprint 4XXIv:19100001177220199 2019.
Doina Precup. Eligibility traces for off- policy policy evaluation. Computer Science Department Faculty Publication Series, pp. 80, 2000.
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey offline reinforcement learning: Taxonomy, review, and open problems. arXiv preprint arXiv:2203.01387 2022.
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John
Wiley & Sons, 2014.
11

---

Published as a conference paper at ICLR 2024
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline rein- forcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 14:11702--1716 2021.
Takuma Seno and Michita Imai. d3rlpy: An offline deep reinforcement learning library. The Journal of Machine Learning Research, 23(1):14205-14224 2022.
Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. In International Conference on Machine Learning, pp. 19967-20025, PMLR, 2022.
Aad W Van der Vaart. Asymptotic statistics, volume Cambridge university press, 2000.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offiine reinforcement learning. arXiv preprint arXiv:1911.1130 2019.
Tengyang Xie, Ching-An An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consistee consistent pessimism for offiine reinforcement learning. Advances in neural information processing systems, 34:6683-6694, 2021a
Tengyang Xie, Nan Jiang, Huan Wang. Caiming Xiong, and Yu Bai. Policy finetuning: Bridg- ing sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 34:27395-2743
2021b.
Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. Advances in Neural Information Processing Systems, 33: 137377--95749 2020.
Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking goal-conditioned supervised learning and its connection to offline rl. arXiv preprint (rXXi:2202.044477 2022.
Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory, pp. 2770-2775. PMLR, 2022.
Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offline estimation of stationary values. arXiv preprint arXiv:2002. 09072, 2020.
12

---

Published as a conference paper at ICLR 2024
A SUPPLEMENTARY DERIVATIONS AND PROOFS
A. DERIVATION OF (6)))
Derivations of (5. and Slater's Condition.
Given the optimization problem:
max En (r(8, a)] -dD,(d111 d' 20
8.4. Σ d' -= = (1-7))((3) - T.d (s)
d' (s.a) S ep(s, a), Vs, E supp(u).
The corresponding unconstrained problem is
min L(d", u, A) ,, Ex [A(s,a)] + (1 (1- aDD A20,0
- -E -M,(Mssss(dd/((sss0 - -c]]
: where a(s,a) == r(s,a) + (() o(s).
Proof. The Lagrangian for constrained optimization
max min L(d U. A) = E (s.a)rdd [r(s,a)]- X20
Σ ((s.a)ld((ssa)) cu(s,
T(11(a))
5,1Esupp(11)
Σ e((S)(( ())
Σ
- aDD())l
(17)
=E, (r(s, a)] - 1x a(s,a) - )] - adD)(d
+ (1 Rop [v(())]
+ Σ 0(8) Σ
E.- [e()]
(18)
3,
d" ( -
-E (r(s a)] - FFp A(s,a)
(d' (dP)
+ (1 - Ep [u()))]]
+ Σ v(5')
((s')s,add
Ex [us)]
(19)
8.0
d
= E. (rss, a)] Ep a(s,a)
(d (ldP)
1
+ (1 - E (2( (so) + Ba (' [u((')] - EEd [u(s)]
(20)
=E,r (r(s,a) a) Eg ( [v(s')] v(s)] - EE ((s,a)
dtt -e-e) (
aDD
(21)
=E,- -Eg-(A(s,a)] (1 7) [=() aD (d Ey, ((s,a)) a) (d ( -
(22)
The Slater's Condition for Problem in 2
The corresponding Slater's condition for optimization problem in eq. eq.2 is that there exists strictly feasible state-action distribution d(s, a) s.t. the equality constraint in eq. (3) holds while constraint in eq. 4 is satisfied with strict inequality. One strictly feasible solution is d (s, a). the stationary state-action distribution of behavior policy πp:
1. Bellman flow constraint E. d= d' a) = (1- (1))))((s)) + T,d (s) holds because the Tp is valid policy and d (8, a) is its corresponding stationary distribution;
13

---

Published as a conference paper at ICLR 2024
2. n (s, a)), Vs, Vs, e supp(u) by definition of μ..
Therefore, the Slater's condition holds. Derivations of 6
Replace (y ()) by (ddd"l as new regularizer, and use ds = (dl(s) + (1-
()p(s,a) a) as proposal distribution, we can obtain
(w, e [woss, a) A(s,)) af(w(ss a))] + (1- Bo [((0]]
+ (1 E (u(s.a)(A(ss))) - M(s,a) af(w(s, a)) + (\(s,)))
Proof.
E2 A(3,a) ( -c)
d
(w = Ex- [A(s,a) + (1 B) [v(sa)] -aDy(d -
(23)
=B,r
d" A(s,a) -af
(p))
d" )
+ (1 Po [u() - E2 [((.aa)
(o=
d
-1)]
(24)
d
-B ()sss))A(ss -afass.))-((((((s. a]]] + p - Ep A(s,a)
d"
(25)
=EE -ff (1- [v(s)]] a(s,a) + (1 - E) -Xsss) a))
(26)
A.2 DERIVATION OF NORMALIZATION FOR STATIONARY DISTRIBUTION
In practice, the optimal distribution d' may not satisfy Ex,o d' (s, due to function approxima- tion error. Therefore, we explicitly enforce the Σ., d'(s 1 (Zhang et 2020) to make d' valid distribution, which equivalent to Ej w (s, a) =1.
With new normalization constraint, the corresponding unconstrained problem becomes
min max [(w; u, A, = (Ear [w(s, ()lss - af(((s + (1 - p.. [v(s)]]
A A20,0 or
+ ED as))s)))(((ss))) X(s,a)) -a(((( 7A(s a)] 7/1 E w" (s, a))
(27)
= =EE,o () a) 7) -
+ (1 () [w(x, ()),))))( (A(s,
-)) ( - a)) E\(s, a)) +1,
(28)
where 1) the dual variable of the normalization constraint. Therefore, we only need to replace Á by Ä 17 for optimization w.r.t. Ei and my. Meanwhile, we will also update nj by gradient descent. See
more details in full algorithm in Appendix B.2.5
A.3 PROOF FOR PROPOSITION I
With assumption D the closed-form solution inner maximization problem max C'(w, v, X) is w (s a) = (f')- a)/a).
where A(s, a) == A(s.a) a) [s a) E supp(n)) A(s, a) denotes regularized advantage function and 1f.3 is the indicator function.
Proof. Let (C.()))))) 0 and we have
(A(o(A(s,a) a (w(s, a))] (1- (1- - afw((ss
(29)
Separate state-action space S A into the support of dd and μ. then can get the solution as (2). Meanwhile, w" 2 always holds by assumption] T Therefore, the solution valid and is exactly the optimal solution.
14

---

Published as a conference paper at ICLR 2024
The closed-form solution to optimal importance ratio also be derived by Maachel-Rockafellar dual form of f-divergence (Nachum et al. 2019b: Nachum Dai 2020), which leads to the same results.
A.4 PROOF FOR PROPOSITION 2
The outer minimization problem m C'(w", U, X) is convex optimization problem. Suppose the optimal solution is (X*, v²), then X has closed-form solution
X (= a) max A A a) (+)), Vs, a E supp(u),
where As r(s, a) + ( 2" (s') u The optimal regularized advantage is
as) as),,
(s,a) supp(d")
((), (s,a) supp(n)
Proof. Notice that the convexity of dual function, which corresponds gu, A) == max L' U, A) in our setting, is proved by previous literature (Proposition 1, section 8.3 in (Luenberger 1997)).
Then we prove the closed form of optimal X* Consider the partial differential of L' C'(w,, u. 2) w.r.t
OL (w" u. A)
θ (1 E
(f")
-
(A - A) aaf (f))
+ Ae
0%
0%
a
(30)
AA A a
A A-
=(1 - 5) ((f)) 11
1
(A A) -ff)
a
A A-1 1
A (A-1)
1
-af' (f))
(1F)-1)
+
(31)
a
a
=(1 -(1- - B) -(()-'y
A (A-1
(1)-1
1 AA-1 1
a
a
A (A-1)
A-A +
+((F)-
(32)
AA-1 -11- -C-)
(33)
We omit (s, a) for A and À functions for brevity. By assumption]] f is convex and f' is monotonic
increasing. Therefore, when A(s, a) S (ē), the gradient of is always non-negative for À 2 0; otherwise, the gradient equals to zero when À = A(s, a) (().
Therefore, the optimal solution of A is
a²(s.a) max(0, (0, A(s, a) (E))
(34)
Plug-in the X to Proposition and then we can get the optimal regularized advantage function Ä
A.5 PROOF FOR PROPOSITION 3
With assumption V(sss) a) E sup
I the theoretical optimal importance ratio is upper bounded by w" (s a) S
Proof. Combine equation and equation g
w. (s,a)) = (f A(s,a) as),,a)
(35)
A(ssa) max(0, A(s, af)
-f-)-1
(36)
min A((s,a) a
=(")-1
((0)
(37)
15

---

Published as a conference paper at ICLR 2024
By Assumption 1 f is strictly increasing and so is (f*)=1. As result,
w ws, (f')
min { [A(s,a)
f(0) (in((("""((sssa)//)))
(38)
a
A.6 PROOF FOR THEOREM
We first give the following lemma.
Lemma 1. Suppose that 1) the action space is d-dim, i.e., A C ACR. (R 2) the diameter of A is M, i.e., ||aı - 02 |loe S M, Va1,a2 02 E A. and 3) there are N action samples from μ given any state s D, denoted by (s, 01) ui)...., (8. (N), and μ is uniform distribution over OOD action space. Let >
(8, a) E DD à Aoon(s). We have
min l|ā 0:l100 > 28) <<1 (1-00
8d Da
P
(39)
Ma
Now consider the theorem:
Suppose that 1) the action space is d-dim, i.e., A C RR 2) the diameter A is M, i.e. ||a - aazl I M. Val, ag E A, and 3) there at least N OOD action samples from μ given any state D. When the continuity assumption 2annnnnoollls with probability at least 1 - 5, > O, we have
wh (s, S (J')-
('(e + ((+++
L
Aa
Md
log
1/d
v(s,a) a) supp(μ)
N
where is the maximum residual error of OOD regression in Eq. Aa is the radius of in- distribution region as previously defined.
Proof. Let B (x, y) {ar IRd | 2'11 y) denote the d-dim Euclidean Ball under ||
The volume of B. (I, y) then given by Vol S5 (r, y)) = 2d We have
TP([Iā ai 2x > 6) = P(Iāā - a1 bc S 1- PPa1 E B (ā, 5)) Recall that (s, (s..)).....)) (s, aN) are i.i.d. samples from uniform distribution on A Boc (a, Aa). Thus, can establish the following equality
(40)
P(a1 E (,)
1(x Bx (ā, 8))u(x)drr
(41)
Ed
1fx E A Bo, dr Vol(A) Aa))
1fx E B (ā,
(42)
VoI(A), Boo (a, Da)) JAu 1fx E B (ā, 6) n A (a),
Aa)
(43)
Vol(Boo (ā, 5) 0 A B (a, Aa)) Vol(AA) 3o (a, Aa))
(44)
Since the action space A is bounded with radius M,
In addition, notice that Vol(A))Booee, Aa)) S Vol(Bao(a,
(45)
Vol (a, (A Aa)) 2 Vol (ā, 5)) Vol(Boo(c (a, Aa))
(46)
Combining the above inequalities and plugging in the formula for d-dim ball under || Ilac we have
P([íã a1 11o0 > 6)
(47)
=1 - P(a1 E B (ā,
(48)
<1 Vol(B (ā, 6)) - Vol (a, Aa))
(49)
Voll Vol(B.o M))
8d Aad
(50)
Md
16

---

Published as a conference paper at ICLR 2024
By independence between the OOD samples
P min i
llā - ailloc failoo > 6)
(51)
n (llā Uill--aslla -ola-asllen as1lo0 > 61 = -1lla-aall>> > =8/² 1 <(1-
8d -da
N
=P
(52)
Md
This finishes the proof.
As remark, if we consider || Ilp instead of ||| Il the result would still be the same. Now give the proof of Theorem
Proof. Let (s. a) E and suppose that (s, a1),... ..... (s, ax) are the i.i.d. samples from p. Let a' {ai...., (N) } be the OOD sample that is closest to under I1 (i.e., a' = arg (x I|x- allxo). Since the maximum regression residual error is &. we have
A S (E) +
(53)
Then, by assumption 2 we have
as S As(s, d') (o(s, a) - Ao d')l ()) + L. ||a a' 1lx +
(54)
Let ê > 0 and 8' = 44 + 8) ('( ()). Suppose 8' 0, by Lemma and using the fact that
1+ S e", Vx E R, we have
P(Ila' - alloo I 6')21- (1- 8'd Aa
21-6 21-e-Net
Noddd and ard
(55)
Md
N oni Aad
Combine equation 54 and equation 55 we have, with probability at least 1 - c-
aed
a) af + L8 + af
(56)
Recall that i = (==) (1') a/))). By equation 56 have
w ( a) := (f) (A ()/aaa///)))) s (() 8)) 2
(57)
with probability at least 1 - - - م.. in equation 57 follows from the fact that f' is increasing.
where 8' = (f 8) - (2) - 2). The inequality step
xrd
Let 5 E (0, 1). Consider 8 = ())) + + (A + x log f)² - First, we verify 8' > 0
with this choice of 6.
((E+ + 8) - ((()) $)
8' L
(58)
a
('(8) + a
+
L
Aad
d
log x
1}²
1 5/2 (())
E
(59)
T
a
M
1
1
sa +
log
(60)
N
5)
0
(61)
Substitute 5 back into equation
57
We get
(ss (E+ (f(0) +
5 + a
L
sa
+
Ma
N log
5)
1 #1
(62)
1 $1
=(f')1 [((2) + $ +
1
Md
san+
N log
(63)
a a
17

---

Published as a conference paper at ICLR 2024
with probability of at least
1 - -00
Md
(64)
-N (N((((()+) - f(() 2))²² - Da"
20y)
=1 ox
(65)
Md
N (2) + $ 2/2 (Aa"
+ M1
log f) - f(() 2) Da"
=1 exp
N
(66)
Md
-N Aa a + Ma N log 1 á - Daa
=1 exp
(67)
Md
=1 8
(68)
This finishes the proof of Theorem I
A.7 PROOF OF THEOREM
2
In this section, we consider the performance of our policy as the sample size N grows.
Let dd denote the data distribution from which DD is obtained. Thus can viewed as í.i.d. samples from d.. In this section, we use the notation with subscript Dx to denote D to address the number data and avoid ambiguity.
Recall that my minimizes the empirical objective + E (x))m as an esti- mation in 14..
We make following assumptions:
Assumption 3. Denote the space of parameter @ in the policy extraction step by θ. Let go(s) := DKL (-1s [) (-Is)), where (s) denotes the state marginal of d". Then, the function class F = (go(-) : S + R E O} is d -Donsker. And Var (x) (g) < oo for all E θ.
Assumption 3 guaraattees the consistency of f trained with dataset D which is common assumption
when considering training with finite samples Van der Vaart (2000); Geer (2000); Ma & Kosorok (2005); Cheng & Huang (2010). A sufficient condition for Assumption is θ being bounded, together with Lipschitz-type condition on F Van der Vaart (2000).
Assumption 4. Suppose the policy extracted from Eq. is n, define the state marginal of dP dd,, d' d (s), dt (s), d² (8), then
(dv( S (T)(((())(d
(69)
The Assumption 4 holds in general because the performance of learned policy is empirically in between π and π.".. indicating that the stationary state distribution of learned policy d is closer to the optimal state distribution than dataset distribution.
Then we introduce the following lemma based on Lemma in Xu et al.
2020
Lemma 2.
Suppose the maximum reward is Rmax max,,a [ (s, al V (Po)
= -E (
denote the performance given policy æ, then with assumption
2Rmax
2.mmax
(P*) ((")))
V'(o)) S
1- (Tv(d + 1-7 Eg (aw
(70)
where ds dP), denote the state marginal of dt dp and dp (s, == -d (s))s(())s)
18

---

Published as a conference paper at ICLR 2024
Proof.
(P* - -V*)
B ((xa))a- [r(,
(71)
1-7
Rmax
dd' (s, - ds(s,))
(72)
2R 2mmax
Drv(d"Ild))
(73)
2 Rmmax
(Drv(d"Ild'ss)) ④) + Drv(d (s). +|(d))
(74)
2.maxx
2
1-Y (sv(d" + 1-7 Ed
(75)
2 Rmmax
2Rmax B (s) (P
(d (s)lld Drvdd
(76)
The 74) follows the triangle inequality of TV distance.
Now we give the complete statement and proof of Theorem 2
Theorem 3. Suppose the maximum reward is Rmax maxs,a ||((s let (=(po)
Esg=ool [V (so)] denote the performunce given a policy и. For policy ny optimized by Eq. 14)
and transition data from dP, if no is a universal approximator, under Assumption 3 Band we have
(Po) TTH (po) S 2Bmax
1-y Drv (d (s) - EN
and en converges in probability to zero at the rate N Thे vh 0, Le., N NeteeN No000 0 in probability.
Proof. By Lemma it remains to establish the vanishing rate of en == 2fs Eg 1-7
By
Pinsker's inequality and Jensen's inequality,
(enao(Drr((((ss)) (+1s)) Ewodd 2D (T) (-|s)
(77)
SB (T)(
(78)
Recall that the me minimizes an empirical expectation
N DKL (TO(-|s))] (=(ss)
min
I Σ (d" (dd) (8i, 8 N
Σ
(79)
N
iee, the objective is equivalent to minimizing the KL divergence over policy distribution. When is a universal approximator, it exactly minimizes the objective to
Use notation go(s) DKL (=(1)))
as defined in Assumption 3
By Assumption 3
VN (B (go( N D gs:))) converges in distribution normal distribution with mean and variance Var (go(s)) < oc. (see e.g., Van der Vaart (2000))
As result, for any O.
N Σ go(s))
Ninn
B (Bnooogo [ge
1
N-000 0, in probability
(80)
N
Therefore,
EsnaP [Drv(To(+1s)]]=
$ / 2E Bannr (DLL(TO( (T) (-(s))
(81)
/2B 2E a (
(82)
1 N
E Eanao [go(s)]
Σ go())
(83)
N
19

---

Published as a conference paper at ICLR 2024
Combine with equation 80 for any h > have
N Nathey =N 2Rmax
g DrV DDVV (DIv(T(((s)(IT(+s))) 1-
(84)
2mxx
2N
Bondd (90(s)]
1
Σ go(s)
N-xc 0, in probability
(85)
1
N 1=1
This finishes the proof.
20

---

Published as a conference paper at ICLR 2024
B EXPERIMENT DETAILS
B. TASKS AND BASELINES
Tasks. We adopt "-vI" tasks for Maze2D and Adroit domains and "-v2" tasks for MuJoCo domain. We do not adopt antmaze tasks because the initial distributions significantly different between training and test settings: the starting point can be anywhere in the offline dataset but is always at the same corner when tested. CDE learns the value functions by optimizing objective 6 w.r.t. initial distribution and dD, which is not compatible with antmaze tasks. This inconsistency also partially explain why trajectory optimization methods (e.g., decision transformer) fail but the Bellman bootstrapping methods are less affected.
We make the rewards sparse in MuJoCo tasks as stated in main text, and return thresholds (i.e., the 75-percentile of the trajectory returns in dataset) are listed in table 3 During evaluation, a trajecotory viewed successful one if its reward return exceeds the threshold.
Table 3: The return thresholds for sparse-MuloCo MuloCo tasks.
Task
Return threshold
4909.1
walker2d-medium
3697.8
hopper-medium
1621.5
10703.4
walker2d-medium-e medium-expert expert
4924.8
hopper- wwet--mmmiimmxpeeee
3561.9
Details baselines. We adopt the results of baselines if reported in the original paper. We rerun the baselines for these tasks using their official codes (IQL, TD3+BC, OptDICE) the d3ripy library (Seno & Imai] 2022) (BCQ, CQL), because 1) d3rlpy keeps the same hyperparameters as the original papers, and 2) the performance of d3ripy is better and more stable than the original implementation for BCQ, CQL (c.g.. the performances of BCQ on Maze2d and Adroit).
B.2 FULL ALGORITHM AND PRACTICAL DETAILS OF CDE
In this section, we present the full algorithm and implementation details. Without otherwise state- ments, the policies crities defaults to be parameterized by neural networks (NN).
B.2.1 VALUE FUNCTIONS SEPARATION
In CDE, we learn both the V-value function and the advantage function. The former incorporate the stochasticity of action distribution to reduce the instability, and the latter is to generalize the optimal importance ratios to OOD regions since the reward and transition probability functions for unseen transition (8, a, rr,' are absent in offline datasets. Meanwhile, we take two steps to train V-value and advantage functions instead of optimizing the objective function in Eq. Note that the objective can be separated into in-distribution and OOD parts:
CEE [wos, A(, a) af(w(s
Bn [e(s)]
+ (1 Ey [us
X(s, af(w(s, a)) + EA(s, a)],
(86)
EE
+ (1 (1- (IE
X(s, -af(-aaaasss a)) E\(s, + (1 7) (s))))))
(87)
where the in-distribution part corresponds to the learning objective of the V-value function in Eq. 10
which is also the dual form of following constrained optimization:
max E.g [r(8, a)] - (d (dP))
(88)
d 20
s.t.
d (s. (1 7)pu T.d (s), Vs. supp(dP).
(89)
21

---

Published as a conference paper at ICLR 2024
The main difference of it from the previous one in Eq.I) is that it constrains the state-action in the support of offline datasets. Therefore, the objective for V-value function learning in (q.(10) is still a convex optimization problem.
B.2.2 POLICY EXTRACTION
In policy extraction, we need to estimate the KL divergence between learned policy and mixed behavior policy a in eq. (4). As we only train the behavior policy я from offline dataset, apply Jensen inequality to get an upper bound of the objective:
D-L[Ta(- {)]| (-1s)] = Bew- [log (als) log(S (+ss) + (1 ()
(90)
Ea-ra [log (als)) loo P (-1s) (1 () () log="(ass)
(91)
which is adopted in policy extraction step in practical implementation.
Meanwhile, we simply set the optimal state distribution d' (8) as uniform distribution on the states in the successful trajectories (i.e., the trajectories with returns larger than 0).
B.2.3 OOD ACTION SPACE
Remember we define the OOD action region AOoD based on the Aa in Sec. 3.2 Here Aa defines the radius of the in-distribution action region and AOOD can cover the action space without overlapping with in-distribution action. As we cannot precisely tell the in-distribution region given an offline dataset, which only provides one action for each state especially in continuous tasks. Therefore, we estimate it by behavioral cloning: we employ NN to approximate the behavior policy x that outputs
Gaussian distribution (Nxx((s) (s), oD (s)) for each state; then the standard deviation o dP((s can be measurement for the breadth of in in-distribution region on. In practice, we adopt Aa = o (s)) when computing (s). We further give a parameter study on Aa in appendix B.4.3
B.2.4 HYPERPARAMETERS
Before training NN, we standardize the observation and reward and scale the reward by multiplying 0.1. To extract the policy after the optimization over value functions converges, we set the warm-up training step following Lee et al. (2021) and the policy To will not start until warm-up training ends. As shown in table 4 we set the same f-divergence coefficient a for each domain because the divergence between the optimal and behavior policies varies in different domains. Note this is still significantly different from previous DICE paper (Lee et al. (027)) that finetunes and assigns different hyperparameterss for every single task. The other shared hy pyprraaraeeerss are summaries in table More details can be found in codes provided in supplementary materials.
Table 5: The shared hyperparameters.
Hyperparameters
values
hidden layers of policy me
[256,256]
hidden layers of π7"
[256,256]
Table 4: The f-divergence coefficient
number of mixtures of T
3
hidden layers V-value vip
[256,256]
Domain
hidden layers of advantage A
[56,256]
Maze2D
0.001
activation function of networks
ReLU
Adroit
0.01
NN optimizer
Adam
MuJoCo
0.1
NN learning rate
3
discount factor Y batch size
0.99
512
mixture coefficient
0.9
max OOD IS ratio
0.3
number OOD action samples
B.2.5 FULL ALGORITHM
In this section, we present the full algorithm in Algorithm. 2
22

---

Published as a conference paper at ICLR 2024
Algorithm 2 Full Algorithm of Conservative Density Estimation
Initialize value functions Uig, Ag. behavior policy x2, policy wæ. All the advantage functions will be subtracted by 1 during computation as proved Appendix A.2
1: for training iteration do
2: > policy evaluation and improvement
3: Sample batch {(s;, aj rss s() from D and OOD actions {a()) (e())....(")) for each
(
4: Update V-value Vip by Eq.(10)
5:
Compute in-distribution ad advantage function Ap(s, a) via Upi
6:
Update regularized advantage Aø by 11
7:
Update distribution normalizer by gradient descent: 7) + -a,(1 (1 E[w" (s, a)))
8:
Update n² by behavioral cloning.
9:
> policy extraction
10: if 2 warm-up steps then
11:
Update policy अy by Eq. (4) with entropy regularization. 12: end if'
end for
B.3 MORE EXPERIMENT RESULTS
B.3.1 TRAINING CURVES
The training curves of full dataset experiments shown in fig. The training steps start from a non-zero number because of the warm-up step, i.c., we learn the policy after the value function almost converges. The warm-up step is 20,000 for the maze2d environment and 40,000 for other environments.
halfcneetah
walker22d
hopper
120
80
100
Normaized reverd
150
02
00
1.00
40
50
med
umaze
20
medium expert
25
medlum-expert
40
mediumeexpen medium
0-
medium
0-
medium
60
80
1.00
100
150
200
50
1000
150
200
50
100
150
700
pen
relocate
150
Normalizes reware soore
150
Tumnan
125
100
100
100
100
75
00
75
expertt
expert
export
50
40
25
200
350
200
Unii Kng Statis (x10P)
Iuiiiing slapss [x10P)
Iraii ing skape (x10P)
Briininng sleps [xL0P)
Figure 4: The training curves of CDE. The shadow region indicates the standard deviation of mean values across different seeds. Here we report the normalized reward scores for MuloCo tasks measured by dense rewards instead of success rate, which has been reported in previous tables.
We can observe that our method converges extremely fast and is very stable during training. This is because CDE employs convex optimization to solve the value function and extracts the optimal policy in manner of supervised learning. On the contrary, the previous methods (e.g., Q-learning- based methods (Kumar et al.) 2020 Kostrikov et al.) 2021bb) prone to over-fitting in training due the interleaved optimization of value and policy (Brandfonbrener et al. (02I), which may lead to large compounded errors and performance decrease especially with long training steps.
23

---

Published as a conference paper at ICLR 2024
B.3.2 THE EXPERIMENT RESULTS ON MORE TASKS
We present more comparisons MuJoCo "-medium-replay" tasks and Adroit "-cloned" tasks in table respectively.
Table 6: Success rate (%) of CDE against other baselines sparse-MuJoCo MuJoCo "-medium-replay" tasks.
BCQ
CQL
IQI.
TD3+BC
CDE (Ours)
halffchetahh medium-replat replayy
91.0+2.2
48.0214.9
70.7+3.7
61.0#16.3
95.2+3.5
walker2d-medium medium-repthe replay
85.748.4
78.3
4.024,9
84,427.3
90.4+5,0
hopper-medium-replay
91.4+5.2
87.0+5.1
0.0=0.0
89.0+4.1
93.3+4.6
Table 7: Normalized scores of CDE against other baselines on Adroit "-cloned" tasks.
BCQ
CQL
IQL
CDE (Ours)
pen-cloned
44.0
39.2
37.3
56.9+13.2
hammer-cloned
0.4
2.1
2.1
3.3+1.7
door-cloned
0.0
0.4
1.6
0.340.1
relocate-cloned
-0.3
-0.1
-0,2
0.4+0.3
By the results listed above, observe that CDE exceeds most baselines sparse MuJoCo " "-medium-replay" and Adroit "-cloned" tasks.
B.3.3 COMPARISON ON SPARSE MEDIUM EXPERT MuJoCo TASKS IN SCARCE DATA SETTING
The experiment results on MuloCo medium-expert tasks are shown in fig. 5 CDE obtains similar performances OptiDICE and IQL on halfcheetah and hopper tasks and exceeds all others on walker2d task. Mcanwhile, CDE has higher success rate with a small proportion of data.
halfcheetth-mmddiu---prrr
walkerzddmediiumeexpet
hoaper-addum-ecpprr
80
80
(86)
40
40
40
01
186
355
100
30%
100%
15
31%
30%
30%
100%
155
3%
10%
30%
100%
-+-
COL
-
BCQ
-
IQL
-B-
TD3+BC
- OpúDICE
CDE (Ours)
Figure 5:
The results on sub-datasets with different dataset sizes for MuJoCo medium-cxpert tasks.
B.3.4 THE EXPERIMENT RESULTS ON ORIGINAL D4RL MuJoCo TASKS
We provide the performances of CDE and comparison with baselines on dense-reward MuJoCo tasks in table g
We can observe that while CDE may not outperform the baselines in dense reward settings, achieves a significantly higher success rate when reward is sparse, which indicates that the baseline performances are predominantly reliant on dense rewards and highlights the effectiveness of CDE for sparse-reward offline data.
B.3.5 PERFORMANCES ON REWRRD GOAL-CONDITIONED TASKS
To further illustrate the effectiveness of CDE sparse-reward tasks, test it goal- tasks from Yang et al. (2022), where the agent receives a "+1" reward only it reaches a desired goal and () reward otherwise. Follow Ma al. (202)) choose 5 relatively hard tasks and the offline
24

---

Published as a conference paper at ICLR 2024
Table 8: Normalized scores on original dense-reward MuloCo tasks.
BCQ
CQL
IQL
TD3+BC
CDE
halfcheetah- mediium
40.7
49.1
47.4
48.3
43,3+2,9
walker2d-medium
54.5
82.9
78.3
83.7
73.8+4.4
hopper-medium medium
53.1
64.6
66.3
59.3
51.2+3.7
halfcheetah- meeprtt-ee-e
64.7
85.8
86.7
90.7
75.6+7.2
malker2dd-ccdium-xppett
110.9
109.5
109.6
110.1
107.7+10.4
hopper-medium-expert
57.5
102.0
91.5
98.0
108.6+4.8
datasets consist of expert and random data. We then compare our method with GCSL (Ghosh et al. 2019), WGCSL (Yang et al. 2022) and GoFAR (Ma et al., 2022), where the former two methods are goal-conditioned imitation learning and the latter one is goal-conditioned version of DICE method.
Table shows the average reward returns comparison. The performances of baselines are adopted from Ma et al. (2022). Note that all compared baselines goal relabeling during training while method ignores the goal and only uses the states to train policy. We can find that CDE obtains the 3 tasks and comparable rewards FetchPush. Mcanwhile, CDE does not use goal state during training and may fail to stitch transitions from different trajectories, which can be a part of reason for failure FetchSlide. As this paper focuses the standard offline RL setting, can be a future direction to extend existing method to goal-conditioned setting.
highest rewards
Table 9: The performances comparison on goal-conditioned tasks.
Task
GCSL
WGCSL
GoFAR
CDE
FetchReach
20.9+2.8
21.942.1
28.2+0.6
29.1+1.7
FetchPick
8.943.1
9.8+2.6
19.742.6
27.7+1.4
FetchPush
13.4+3.0
14.742.7
18.2+3.0
16.6#2.0
FetchSlide
1.8+1.3
2.7+1.6
2.5+1.4
1.1+1.0
HandReach
1.4+2.2
6.0+4.8
11.5+5.3
17.0#2.9
B.3.6 COMPARISON ON COMPUTATION TIME
One drawback of method compared baselines is that CDE may require time in training. Therefore, we compare baselines and our method on halfcheetah meahhh-mdiim---ee medium-expert- v2 task. We use the server with AMD EPYC 7542 32-Core CPU and A5000 GPU. To compare different methods fairly, we report the time of 200,000 steps with batch size = 512 and defaulted hyper-parameters for baselines. The final results are shown table I0
computation
Table 10: The computation time comparison.
Method
BCQ
CQL
IQL
TD3+BC
CDE
Computation time (min)
200
150
40
30
~00
B.4 MORE PARAMETER STUDIES AND ABLATION STUDIES
In this section, we present more parameter studies the hyperparameters in method. We mainly choose Maze2d tasks as our testbench as they are more sensitive to the hyperparameters.
B.4.1 PARAMETER STUDY ON MAX OOD IS RATIO
Although we visualize the stationary state distribution of policies with different maze2d-large task, provide performances all maze2d tasks in table II supplement. As ? decreases, constraint on unseen region gets stronger. There is a performance gain with proper conservatism but it also ruins the final performance if the constraint is conservative.
25

---

Published as a conference paper at ICLR 2024
Table 11: The performances on maze2d tasks with different غ
3.0
0.3
0.03
0.003
maze2d-umaze
121.7=12.5
134.1=10.4
132.7=11.4
114.5+17.3
maze2d-medium
154.2+14.9
146.1=13.1
135.2+12.3
89,4429.5
maze2d-large
189.7+19.7
210.0#13.5
174.5+24.7
51.6#37.9
B.4.2 PARAMETER STUDY ON NUMBER OF OOD ACTION SAMPLES
We use sampling on OOD actions in eq. 11 to approximate the out-of-distribution constraint. To study how the action sample number N influences the final performance, we test our method with different N maze2d tasks and show the results in table 12 Theoretically, the approximation enoor on OOD constraint will decrease as we increase the N. practice, we find there no significant performance improvement when N 2 3. Therefore, we simply set N 5 for all tasks in practice.
Table 12: The performances on maze2d tasks with different number OOD action samples N.
5 (adopted)
10
maze2d-umaze 115.2+6.9
131.7+3.5
134. 100.4
141.1=10.
mazeld-medium
132.6+11.9
142.9=13.8
146.1=13.1
146.8+14.8 209.7=14.8
maze2d-large
179.4+21.5
202.0+24.7
210.0=13.5
B.4.3 PARAMETER STUDY ON IN-DISTRIBUTION WIDTH
We the standard deviation of the behavior policy output o (s)) measurement of the width of in-distribution region. However, there may still be some concerns on the overlap between OOD action space and in-distribution region. Therefore, we conduct parameter study on Aa.
Table 13: The performances maze2d tasks with different Aa.
Aa Aa/oP(s))
0.3
0.7
1.0 (adopted)
maze2d-ume 2d-uzzze
131.3+16.4
137.749.3
134. 140.4
maze2d-medium
149.1=12.7
151.2+11.0
146.1113.1 210.0+13.5
maze2d-large
211.5+14.3
204.0+17.3
As listed table 13 there is no significant performance drop when decreasing Aa/oP(s) (i.c., the higher risk of overlap between OOD and in-distribution region). This is because the advantage learning step in II) involves both in-distribution and OOD regression. When an in-distribution action is wrongly recognized OOD action, its value function will exist both two terms. Therefore, the value function of in-distribution action will not be negatively affected while the OOD actions in Aood can always be constrained.
From the above results, we can observe that the performance of our method is robust to the most hyperparameters except ē, which controls the degree of conservativeness on unseen regions.
B.4.4 ABLATION STUDY ON WARM-UP STAGE
We adopt warm-up where only update the value function and stop policy updating. To investigate its influence on final performances, we present the results of ablation study on Maze2d tasks in table
26

---

Published as a conference paper at ICLR 2024
Table 14: The performances on maze2d tasks with/without warm-up stage.
with warm-up
w.o. warm-up
maze2d-umaze
134.1=10.4
136.2+13.2 141.7+17.3
maze2d-medium
146.1=13.1
maze2d-large
210.0#13.5
205.4+15.4
We can find that the warmup stage does not influence the final performances significantly.
27


Reviews:
[{"review_id": "WZAczUe8lr", "reviewer": "Reviewer_voqr", "summary": "In this work the authors propose CDE, an offline algorithm that builds upon a similar formulation as that of the DICE method of Nachum et al. In DICE the state-action distribution is optimized in order to maximize the reward under that distribution subject to an f-divergence regularization between the visit distribution and the empirical distribution under observed data. In additiona this optimization problem is constrained such that the state-action visit distribution is a valid discounted distribution under the initial-state and transition model of the MRP. Where CDE departs from DICE is in introducing an additional constraint such that the state-action distribution function evaluated at any state-action is less than a multiplicative factor of epsilon of some alternative distribution mu defined to have support only over out-of-distribution actions, and where mu is restricted to have the same marginal state distribution as the empirical distribution. Given this formulation the state-action distribution can then be optimized and the policy implied by this distribution can be extracted from the distribution. Additionally the authors extend the f-divergence to include an f-divergence with an extended distribution which mixes the empirical and mu distribution which a trade-off factor.\n\nBased on these results the authors present some theoretical results bounding the importance ratio and bound the performance gap. The authors also provide a number of results on maze2d tasks and sparse mujoco tasks, predominantly showing that their method outperforms other comparable offline RL algorithms.", "questions": "See above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall the paper was quite good and the algorithm was well presented, albeit quite dense. The theoretical results do add to the paper (although I'm not sure how much) and the results were quite good and well presented.", "weaknesses": "Overall the paper was good and from my perspective merits inclusion at ICLR. My only real criticism of the work lies in its presentation, which although good I felt was dense. And the precise differences with e.g. DICE could have been made more clear, i.e. I would have liked to see more discussion of the effects of the additional constraint. To that end I'm not sure how much was added by the theoretical contributions of this work which could have been rather spent on explaining these differences at a higher level. And to be clear, I think the theorem(s) are useful, but since the proof itself and some of the underlying assumptions could not be included due to space constraints I'm not sure if these would not have been better moved into the appendix as a whole.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WcTEkP1sbM", "reviewer": "Reviewer_HRnW", "summary": "This paper proposes a new offline RL method, called Conservative Density Estimation (CDE). It builds upon the DICE work by utilizing the state-action occupancy stationary distribution. However, the DICE type work requires the assumption on the concentrability, and the boundedness of the IS ratio for stable training. This paper tries to address this by introducing pessimism on the state-action stationary distribution, via a similar principle like conservative Q-learning.", "questions": "See Weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper points out the two type of approaches for offline RL, conservative Q-value based, which might perform worse under sparse reward setting due to iterative Bellman update; and DICE based, which implicitly relies on the concentrability. The proposed method nicely combines these two approaches, via a pessimism approach on the state-action stationary occupancy, which is a neat idea especially on the sparse reward setup.\n\n- The paper is very well-written and easy to read. The method is very well-motivated, and clearly described, from policy evaluation to policy extraction. It also shows theoretical guarantees regarding to the concentrability bound, as well as performance gap. \n\n- The experiments on sparse reward settings and small data regime is interesting, and support the CDE's effectiveness, especially at the sparse reward setup.", "weaknesses": "- Some parts of the method need to be dived into more details, for example, the CDE policy update happens when the value function converges. This also marks a difference compared with classical actor-critic methods, is it a key component in improving the empirical performance? \n\n- More hyper-parameters introduced by the algorithm, such as the max OOD IS ratio, the mixture coefficient, etc. The ablation study on the mixture coefficient is interesting, demonstrating the robustness and giving some empirical guidelines on how to pick them. However, the study of max OOD IS ratio, though interesting, it does not provide any meaningful guidelines on how to choose them for various environments and applications. \n\n- A clear understanding between CDE and CQL is missing, one is pessimism on the state-action state-action occupancy stationary distribution, the other is on the value function. Figure 2(d) only probably only shows the performance of CQL for a fixed pessimism-coefficient, it would be great to compare these two methods in detail, both theoretically and empirically.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0DhMM0eKRU", "reviewer": "Reviewer_MKs2", "summary": "The paper proposes a density estimation-based offline RL algorithm that aims to incorporate the strengths of value penalization-based offline RL methods and DICE-based offline RL methods. The proposed method outperforms the certain baselines in considered sparse reward D4RL tasks.", "questions": "- On sparse-MuJoCo tasks, how does the proposed method and the baselines perform if you change the cut-off percentile (which is currently 75)?\n- How does the proposed method perform compared to the baselines on dense reward settings?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed method outperforms the considered baselines significantly in maze2d tasks.\n- The motivation which is to achieve the best of both worlds (between the value penalization methods and the DICE-based methods) is solid.", "weaknesses": "- Some clarifications are required in the introduction section. 1) \"pessimism-based techniques may be prone to over-pessimism, especially in high-dimensional state-action spaces\" => why is over-pessimism a bigger problem for high-dimensional state-action spaces? 2) \"regularization methods often struggle with the tuning of the regularization coefficient\" => Why does the author think this is true? The cited paper seems to be about DICE and not about regularization methods. Additionally, IQL (which is a baseline considered in the paper) uses the same hyperparameters for experiments in the same domain, just like the proposed method.\n- The considered tasks are not thorough. First, there are no experiment results on D4RL Gym *-medium-replay datasets, which contain trajectories that vary significantly in performance. These datasets can test if the proposed method can handle datasets with mixed quality. Second, on Adroit, I am curious on why the authors did not include the *-cloned datasets while including *-human datasets. Third, some important tasks with sparse reward are missing: Antmaze and Kitchen. These tasks are much harder than simple maze2d and at the same time have sparse rewards.\n- Some important baselines are missing. To name a few, DecisionTransformer [1] and Diffuser [2] also claim they have strength in the sparse reward settings.\n\n[1] Chen et al., Decision Transformer: Reinforcement Learning via Sequence Modeling, NeurIPS 2021.\n\n[2] Janner et al., Planning with Diffusion for Flexible Behavior Synthesis, ICML 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f4z9l1dyeC", "reviewer": "Reviewer_6pFf", "summary": "The paper proposes an Offline RL algorithm called Conservative Density Estimation (CDE) incorporating pessimism within the stationary distribution space. Extensive empirical study shows that CDE outperforms existing baselines in both the sparse-reward settings and the scarce-data settings.", "questions": "1. How are the OOD actions sampled? Do you just sample a random action and reject it if it is in distribution?\n\n2. Does Slater's condition also hold for optimization problems dealing with infinite-dimensional vectors (i.e., functions)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The authors present a novel method of mixing pessimistic-based methods with DICE-style methods. Unlike other pessimistic methods that balance pessimism using a hyperparameter called a penalty coefficient, CDE finds the optimal balance based on theoretical grounds. The paper also introduces a new way to convert dense-reward MuJoCo tasks into a sparse-reward setting based on the trajectory return. Together with the scarce-data setting experiments, the two additional benchmarks can help test the robustness of offline RL algorithms. Finally, the arguments provided by the authors are mathematically sound.", "weaknesses": "1. §3.2.1 states that the strong duality holds due to Slater's condition. The fact that the optimization problem of our interest satisfies Slater's condition does not seem trivial.\n\n2. The RHS of (7) may not exist. For example, $\\tilde{A}(s, a)$ might be negative.\n\n3. According to §B.2.2, the authors set $d^*(s)$ as a uniform distribution on the state in the successful trajectories. It would have been better if this explanation were included in the main paper. I am doubtful that this distribution will be close to $\\tilde{w}^*(s, a)\\hat{d}^{\\mathcal{D}}(s, a)$, though.\n\n### Minor comments\n\n1. (18), (19) ⇒ the summation should also be over $s$ and $s'$, respectively\n\n2. (29) ⇒ $d^{\\mathcal{D}}$ in the second expectation → $\\mu$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)