Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning
Authors: Liyuan Mao
Haoran Xu
Xianyuan Zhan
Weinan Zhang
Amy Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Offline Reinforcement Learning
Diffusion Models
Primary area: reinforcement_learning
TL;DR: A new diffusion-based offline RL algorithm that uses a guide-then-select paradigm to achieves state-of-the-art empirical results.

Abstract: One important property of DIstribution Correction Estimation (DICE) methods is that the solution is the optimal stationary distribution ratio between the optimized and data collection policy. In this work, we show that DICE-based methods can be viewed as a transformation from the behavior distribution to the optimal policy distribution. Based on this, we propose a novel approach, Diffusion-DICE, that directly performs this transformation using diffusion models. We find that the optimal policy's score function can be decomposed into two terms: the behavior policy's score function and the gradient of a guidance term which depends on the optimal distribution ratio. The first term can be obtained from a diffusion model trained on the dataset and we propose an in-sample learning objective to learn the second term. Due to the multi-modality contained in the optimal policy distribution, the transformation in Diffusion-DICE may guide towards those local-optimal modes. We thus generate a few candidate actions and carefully select from them to achieve global-optimum. Different from all other diffusion-based offline RL methods, the \textit{guide-then-select} paradigm in Diffusion-DICE only uses in-sample actions for training and brings minimal error exploitation in the value function. We use a didatic toycase example to show how previous diffusion-based methods fail to generate optimal actions due to leveraging these errors and how Diffusion-DICE successfully avoid that. We then conduct extensive experiments on benchmark datasets to show the strong performance of Diffusion-DICE.

Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning
Liyuan Mao
Haoran Xu'
Shanghai Jiao Tong University
UT Austin
Xianyuan Zhan Tsinghua University
Weinan Zhang Shanghai Jiao Tong University
Amy Zhang UT Austin
Abstract
One important property of DIstribution Correction Estimation (DICE) methods is that the solution the optimal stationary distribution ratio between the optimized and data collection policy. In this work, we show that DICE-based methods can be viewed as a transformation from the behavior distribution to the optimal policy distribution. Based this, propose novel approach, Diffusion-DICE, that directly performs this transformation using diffusion models. We find that the optimal policy's 's score function can be decomposed into two terms: the behavior policy's score function and the gradient of a guidance term which depends on the optimal distribution ratio. The first model trained on the dataset and we propose in-sample learning objective to learn the second term. Due to the multi-modalle contained in the optimal policy distribution, the transformation in Diffusion-DICe may guide towards those local- thus generate a few candidate actions and carefully select from them to approach global-optimum. Different from all other diffusion-based offline RL methods, the guide-then-selec paradigm in Diffusion-DICE: only uses in-sample actions for training and brings minimal error exploitation in the value function. We use a didatic toycase example to show how previous diffusion-bart methods fail to generate optimal actions due to leveraging these errors and how Diffusion-DICe successfully avoids that. We then conduct extensive experiments benchmark datasets show the strong performance of Diffusion-DICE.
can be obtained from diffusion
optimal modes.
Introduction
We study the problem of offline reinforcement learning (RL), where the goal is to learn effective policies solely from offline data, without any additional online interactions. Offline RL is quite useful for scenarios where arbitrary exploration with untrained policies is costly or dangerous, but
sufficient prior data is available, such as robotics [8] or industrial control [56). Most previous model-free offline RL methods add a pessimism term to off-policy RL algorithms [50] 2], this pessimism term acts as behavior regularization avoid extrapolation errors caused by querying the Q-function about values of potential -istr--disttitution (OOD) actions produced by the policy [25]. However, explicitly adding the regularization requires careful tuning of regularization weight because otherwise, the policy will still output actions that are not seen in the dataset. DIstribution Correction Estimation (DICE) methods [35] 41 34 provide an implicit way of doing so. By applying convex duality, DICE-based methods solve for the optimal stationary distribution ratio between the optimized and data collection policy in an in-sample manner without querying unseen actions [55].
'quual contribution; more junior authors listed carlier. Corresponding authors. Project page at petttps:/rrynxxhr getthu..i//ifffuii. wtt.:////ffffiioo-DDDD DICE//
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In this paper, we extend the analysis of DICE-based methods: we show that the optimal distribution ratio in DICE-based methods can be extended to a transformation from the behavior distribution to the optimal policy distribution. This different view motivates the use of deep generative models, e.g. diffusion models 42 1.5 431, to first fit the behavior distribution using their strong expressivity of fitting multi-modal distributions and then directly perform this transformation during sampling. We find that the optimal policy's 's score function can be decomposed into two terms: the behavior policy's s score function and the gradient of guidance term which depends the optimal distribution ratio learned by DICE. The first term can be easily obtained from the diffusion model trained on the offline dataset. While it is usually intractable to find a closed- solution of the second term, make a subtle mathematical transformation and show its equivalence to solving a convex optimization problem. In this manner, both of these terms can be learned by only dataset samples, providing accurate guidance towards in-distribution while high-value data points. Due to the multi-modality contained in the optimal policy distribution, the transformation may guide towards those local-optimal modes due to stochasticity. We thus generate few candidate actions and use the value function to select the max from them to go towards the global optimum.
We term our method Diffusion-DICE, the guide- then-select procedure in Diffusion-DICE dif- fers from all previous diffusion- based offline RL methods [49 4 [4 32), which are either only guide- based or only select-based.
True
Dillused
quide-based methods [2] use predicted values of actions generated by the diffusion behavior policy to guide toward high-return actions. Select-based methods [ [4] bypass the need for guidance but require sampling a large number of actions from the diffusion behavior policy and using the value function to select the optimal one. All these methods need to query the value function of actions sampled from the diffusion behavior policy, which may produce potential OOD actions and bring overestimation errors in the guiding or selecting process. Our method, however, brings minimal error in the guide-step by using accurate in-sample guidance to generate in-distribution actions with high values, and by doing so, in the select step only few candidate actions are needed to find the optimal one, which further reduces error exploitation. We use an illustrative toy case to demonstrate the error exploitation in previous methods and how Diffusion- DICE successfully alleviates that. We also verify the effectiveness of Diffusion-DICE benchmark D4RL offline datasets [8]. Note that Diffusion-DICE also provides a replacement for the Gaussian policy extraction part used in current DICE methods, successfully unleashing the power DICE-based methods. Diffusion-DICT surpasses both diffusion-based and DICE- based strong baselines. reaching SOTA performance in D4RL benchmarks [8]. We also conduct ablation experiments and validate the superiority of the guide-then-seled learning procedure.
Dick
guide or select in prestous methods
gulde-then-selee Diflusion-BICE
Figure 1: Illustration of the guide-then-select paradigm
2 Preliminaries
which is specified by tuple M = (S, A, P, r)). Here S and A are state and action space, P(s'|s, |s, a) and dy denote transition dynamics and initial state distribution, r(s, a) and represent reward function and discount factor, respectively. The goal of RL is to find a policy π (als) which maximizes expected return E ' r(s,, a.), Another equivalent LP form of expected return is (ss,a)=d> (r(s, a)}. where d' (s) a) = (1-7) Ee D Pr(ss s, a a) is the normalized discounted stationary distribution T क 336. For simplicity, we refer to d= (s, a) as the stationary distribution of ग. Offline RL considers
We consider the RL problem presented as Markov decision process 45
the setting where interaction with the environment prohibited, and one need to learn the optimal from a static dataset D
{8;, 04 TG, M
We denote the empirical behavior policy of D as πe ?
2.1 Distribution Correction Estimation
DICE methods 41 incorporate the LP form of expected return [(T) B(e.e)=a- (r(s,a) with regularizer (d = E( [f( d (s.a) }}, where Dy is the f-divergence induced by
((2 (( (s.a))
2

---

convex function f [3] More specifically, DICE methods try to find an optimal policy x² that satisfies:
arg max ((.a))aaaa- a)] aD
(I)
This objective is generally intractable due to the dependency on (s, a). especially under the offline
setting. However, by imposing the Bellman-flon constraint [33] E d(s,a) a) (1 7))o(s))
(sr.ar) d(s', a (')pps)s', s',,') aa) on states and applying Lagrangian duality and convex conjugate, its dual problem has the following tractable form:
min (1 () (s)) (B (TV -(()) V
Here I a variant of f's convex conjugate and (V, a) r(s: -) [Vs')] represents the Bellman operator on V. In practice, one often uses prevalent semi- aami-gradient technique in RI. that estimates (V a) with (ss, a) and replaces the initial state distribution dg with dataset distribution d to stabilize learning [4] 34]. In addition, because D usually cannot cover all possible s' for specific (s, a). DICE methods only use a single sample of the next state s'. The update of Q(s, a) and (s) in DICE methods are follows and we refer to detailed derivation in Appendix A
min B(sa))=oD [11 V
((s)]/aa/
(3)
min E Q
Q(s, Q(s,a))
Note that learning objectives of miCh-methods can be calculated solely with a (s, a, 8') sample from D. which is totally in-sae.le. One important property of DICE methods is that Q" and V have relationship with the optimal stationary distribution ratio w (8, a) as
d' (s) = max w (s,a) = (s)
where d' is the stationary distribution m² To get ' from w". previous policy extraction methods in DICE methods include weighted behavior cloning [34], information projection [27] or policy gradient [37]. All these methods parametrize an unimodal Gaussian policy in order to compute
log (a) 8) 13 which greatly limits its expressivity.
2.2 Diffusion Models in Offline Reinforcement Learning
are generative models based on a Markovian noising and denoising process. Given random variable Tn and its corresponding probability distribution qo (20). the diffusion model defines a forward process that gradually adds Gaussian noise to the samples from To (T 0). Kingma et al. [20] shows there exists a stochastic process that has the same transition distribution ( (zo) and Song et al. [43] shows that under some conditions, this process has an equivalent reverse process from T' O. The forward process and the equivalent reverse process can be characterized as follows, where w, is a standard Wiener process in the reverse time.
Diffusion models 42
15 43
(20(2)) = (a of1) dæ: = [(())=+ - g v. log (z+))dt + g(t)dWr, or ~ gr (5)
Here we slightly abuse the subseript t to denote the diffusion timestep. ar ae are the noise schedule and ft). g(t) can be derived from a₁, de [31] For simplicity, we denote qıa (x (z)) and pur (Zg (2:)) as q( (zo) and p( 22)). respectively. To sample from (o( by following the reverse stochastic differential equation (SDE), the score function Vz, log qe (Ix) is required. Typically, diffusion models denoising score matching to train neural network (z,,, t) that estimates the score function by minimizing (o.=2((,))) (6)000((20) (eno(tt))e(((..) [ - <||²], where x, = 0y + ave. As we mainly focus on diffusion policy in RL, this objective is usually impractical because go(wx) is expected to be the optimal policy x² (als). A more detailed discussion is given in Appendix A
46 15
To make diffusion models compatible with RL, there are generally two approaches: guide- based and gelect-basede Guide-based methods 32 17 incorporate the behavior policy's 's score function with an additional guidance term. Specifically, they learn a time-dependent guidance term J, and use it to drift the generated actions towards high-value regions. The learning objective of T. can be generally formalized with L(T((a)), w(s { ())),, where wss, (at) K are critic-computed values on K
diffusion behavior samples. L can be a contrastive objective 32 seran--quure-erroo objective [17]
3

---

After training, the augmented score function Va log (T(a)s)))))))))))))))){{)) = Va log (a(lss) + Jo is used characterize the learned policy.
Select-hased methods 4 [4] utilize the observation that for some RI. algorithms, the actor induced through critic learning manifests a reweighted behavior policy. To sample from the optimized policy, these methods first sample multiple candidates {at) (e))=1 from the diffusion behavior policy and then resample from them using ritic-computed values w(s, fap (i=))) [ap)mm))
More precisely, the sampling w(s.a)))
procedure follows the categorical distribution Pr(a a s) = Σ w((3,05))
Error exploitation As we see, both guide-based and select-based methods need the information of wsss, (1)) to get the optimal action. However, this term may bring two sources of errors. One is the diffusion model's approximation error in modeling complicated policy distribution, and the other is the critic's s error in evaluating unseen actions. Although trained on offline data, the diffusion model may still generate OOD actions (especially with frequent sampling) and the learned critic can make erroneous predictions on these OOD actions, causing the evaluated value on these actions to be over-estimated due to the learning nature of value functions [11 (23]. As result, the generation of high-quality actions in existing methods is greatly affected due to this error exploitation, which we will also show empirically in the next section.
3 Diffusion-DICE
In this section, we introduce our method, Diffusion-DICE. We start from an extension of DICE-basee method that considers DICE as a transformation from the behavior distribution the optimal policy distribution, which motivates us to use diffusion models to perform such transformation. We then propose a guide-ther then-select procedure to achieve the best action, i.e., we propose in in-sample guidance learning for accurate policy transformation and use the critic to do optimal action selection to boost the performance. We also propose piecewise /-divergence to stabilize the gradient during learning. We give an illustration of the guide-then-select paradigm and use a toycase to showcase the error exploitation in previous methods and how Diffusion-DICC successfully alleviates that.
3.1 An Optimal Policy Transformation View of DICE
As mentioned before, DIstribution Correction Estimation (DICE) methods, an important line of work in offline RL and IL provides us with an elegant way to estimate the optimal stationary distribution ratio between d' (s, a)) and (s,s, a) [2 41 (34]. We show that this ratio also directly indicates proportional relationship between the optimal in-support policy ' (als) and the behavior policy D (als). This proportional relationship enables us to transform x into Tr".
Our key observation is that the definition of (s inherently reveals bijection between и r[a)s) (s) and d= (s, a). Given a relationship between d' (s, a) and d' (s, we can use this bijection to derive relationship between x" (als) and (als).
(ndd((()s)) We formalize the bijection and derived relationship as:
(s.a)
d= (s,
(aas))
* (als)
(s.a)
T P (als).
JaeA d= (s,a)da
SoeA
(e(a))) (".o)) n (a)ss)d
The proof is given in Appendix B Since the denominator in the second equation involves an integral over a A and unrelated the chosen action, the relationship indicates T (a x d'(a,a) (a)ss, This means that the transformation between ds a) and d' a) can be extended (s.0)
to the transformation between T (als) and n" (a (s). This transformation motivates the use of deep generative models, e.g. diffusion models. to first fit the behavior distribution using their strong expressiveness and then directly perform this transformation during sampling. More specifically, the score function of the optimal policy and the behavior policy satisfy the following relationship:
d' (s. ap
Va, log o. (ayls) = Va. log oP (axls) + Va, log (Pooaa
(6)
dD ag
The proof is given in Appendix B Intuitively, Va, log x. (a. Is) tells how to generate actions from D and Va, log T (s) ( (x, an)] tells us how to transform from n2 to in-support # during the reverse diffusion process. As mentioned before, when representing x² with diffusion model,

---

all we need is its score function Va, log "ī (agls). Equivalently, we focus on the right-hand hand side of (q. The first term is just the score function of T², which is fairly easy obtain from the offline dataset [14 4] To perform the transformation, we still require the second term. Fortunately, DICE allows us to acquire the inner ratio term in an in-sample manner directly from the offlin dataset, and we will show in the next section how to exactly compute the whole second term using offline dataset.
3.2 In-sample Guidance Learning for Accurate Policy Transformation
Although DICE provides us with the optimal stationary distribution ratio, which the cornerstone of the transformation, the second term on the right-hand side of Eq.66) is still intractable due to the conditional expectation. Our key observation is that for arbitrary non-negative function f(x),, the optimizer of the following convex problem is unique and takes the desired form of log oggexpectattio..
Lemma 1. Given a random variable X and its corresponding distribution P(X), for any non- negative function f(x), the following problem is convex and its optimizer is given by y log (())
min P(n -M v).
It is evident that the optimizer can be used to derive the second term in Eq.(6). . In fact, we show the second term can be obtained directly from the offline dataset, we optimize the following objective.
Here De is a guidance network parameterized by 0 and we denote ('s.a) as w" w"(s, (s,a) for shorthand.
d"(s.a)
Theorem 1. Va, log En (aol (oolo..e) [w" (s, an)] can be obtained by solving the following optimization problem:
(als) Bay p(oilo)))))) w= (8, (eaa))=999l g0(o,ar.) + go az l)]:
min (, (als)
as the optimal solution 0 satisfies V Va,90 100000000 (s, at, t) Va, log (oola (w(s, an)].
This objective has two appealing properties. Firstly, compared to guide-based and select-based methods, we don't It need to use multiple actions generated by the behavior diffusion model, we only need one sample from π (alas) ( to estimate the second expectation, enabling an unbiased estimator of this objective using only offline dataset possible. Secondly, because w(s, a) must be non-negative to ensure valid transformation, this objective is convex with respect to go a, t). This indicates guarantee of convergence to ge- (s, az, t) under mild assumptions (see proof in Appendix B
Directly inducing Va, log E, (aglas. s) [w= (s, ao)] from the dataset is crucial in the offline setting. Firstly, it avoids the evaluation OOD actions and solely depends on reliable value predictions of in-sample actions. Consequently, the gradient exploits minimal error in the critic, enabling it to more accurately transform P into in-support x²" Moreover, because all values of w (s, a) used in our method are based on in-sample actions, applying this gradient in the reverse process will guide the samples towards in-sample actions with high w w"(s, a) value.
We refer to this method as In-sample Guidance Learning (IGL). Note that previous methods either have biased estimate of the gradient [1 6), or have to rely on value predictions for diffusion generated actions, which could be OOD 322 We refer to Appendix Cor an in-depth discussion of IGL against other guidance methods.
Stablizing gradient using piecewise f-divergence. In practice, there exists one issue when computing the guidance term. Note that in Eq.(4). w ws, a) could become zero, depends on the choice of f. Given specific a,, if w (s, au) = 0 for actions under π x (aglla, s), the second gradient term will not be well-defined due taking the logarithm of O. In practice, this can result in an unstable gradient. To avoid such issue, (f')-1(xr) should be positive for any given Also, to facilitate the optimization process, (x) should possess a closed-form solution and good numerical properties. Unfortunately, none of the commonly used f fdiivergence can accommodate both. However, considering the following two f-divergences:
Divergence
f(x)
(x)
(f')-((x) 6"
Reverse KI.
x: llggr
6
Pearson x²
(xr 1)2
max ( (-1 1, T
+ +x)
max (0, $ + 1)
5

---

It is obvious that Reverse KL divergence possesses the positive property but exhibits numerical instability given large z due to the exponential function in (z), while Pearson x² divergence avoids the instability in the exponential function, its (z) value is negative when x < -2
To take advantage of both divergences while avoiding their drawbacks, we propose the following piecewise f f-divergence that has properties similar to Pearson x² when æ is large while has properties similar Reverse KL when is small:
((x) - 1
I 2 1, r 1olog=-0 x+
(8)
0 x <1
In Appendix B we prove that f(z) is a well-defined f-divergence. We also prove that f(x) has the following proposition:
Proposition 1. Given the piecewise f-divergence in Eq. formulation:
(x)) and (*) has the following
++ x 120 0 x x<0
2 + 2 0
(x)
(x) =
T
e
e00
x <<
We can see that given any x, the exponential function in (z)) ensures a strictly positive value. Meanwhile, (xx) is most quadratic polynomial, which ensures numerical stability in practice.
Note that (xx enables a stable policy transformation and can also be applied to other DICE-based methods when the optimal stationary distribution ratio needs be positive.
3.3 Boost Performance with Optimal Action Selection
After transforming π into in-support σ², we now introduce the select-step to boost the performance during evaluation. One issue is the multi-modality in the optimal policy, this partially arises from the policy constraint used to regularize its output ( 25 55 34]. Under policy constraint, the regularized optimal policy is unlikely to be deterministic and may be multi-modal.
More specifically, in our method, as we strictly follow the score funetion Va, log मं (arls) of in- support optimal policy л¹ to generate high-value actions during the reverse diffusion process, any mode that has non-zero probability under x (alss) could be generated, which lead sub-optimal choice. Similar issues have been discussed in previous works [14] [3]. In Diffusion-DICE, it's natural to leverage the optimal critic Q" derived from DICE in 33. to identify the best action. Given a specific state actions from rr (als). Then we evaluate these actions with Q" and select the action with the highest Q" (s, a) value the policy's 's output
first follow Va, log कद (a,ls) in the reverse diffusion process to generate a few
T(s) a"
arg max
as)
(10)
(E (et....... ou (als))
Different from previous guide-only or select- only methods, we base the select stage on the guide stage. As mentioned before, using IGL ac- curately guides the generated candidate actions towards suppup actions with high value. With the assistance of the guide stage, we can sample candidates from π (a]ss, which has a high prob- ability of being high-quality. This means only a small number of candidates are required to be sampled, which reduces the probability of sam- pling OOD actions. This leads to minimal error exploitation while still attaining high returns. We give an illustration of the guide-then- then-select paradigm in Figure I
Algorithm 1 Diffusion-DICE
1: Initialize value function Q Vo2, diffusion behav- ior model ev. guidance network go
2: // Training
3: Pretrain the diffusion behavior model by minimizing
el
4: for t 1,2,-..,,A"" do
5: Sample transitions a,, ~
6: Update Qo, and Vo, by minimizing Eq.
7:
Update go by minimizing Eq.
end
9:
// Evaluation
10:
Sample [a from x² following 55. with
Va, log πī (a.ls) (u(.....)) Va, go a,,t)
We term this guide-then-select method 11: Select action according to (
Diffusion-DICE and present its pseudo-code in Algorithm [ During the training stage, Diffusion- DICE estimates the optimal stationary distribution ratio using DICE with our piecewise f-divergence and calculates the guidance with IGL, both in an in-sample manner. During the testing stage,

---

Diffusion Policy
Diflusion-DICE
IDQL (select-only)
(guide-then-seeett
(guide-only)
6:15
ok
020
(15)
Guide
05
cm
620
600
-10
Solect
Reward Function
Learned Reward Function
15
04
Diffusion Trajectryy
60
-14
45
-19
Figure 2: Toycase of 2-D bandit problem. The action in the offline dataset follows a hivariate standard normal distribution constrained within an annular region. The ground truth reward has two peaks extending from the center outward. use a diffusion model H² the behavior policy a reward model ft to fit ground truth reward R. Both #P and A fit in in-distribution data well while making error in out-of of-distribution regions. Diffusion DICE could generate correct optimal actions outer cirele while other methods tend expolit error information from ft and only generate overestimated, sub-optimal actions.
Diffusion-DICE selects the action from n² (als) with the highest value. By selecting from a small number of action candidates from x" ("(ss), minimal error of the action evaluation model will be exploited. Besides serving as diffusion-bart novel policy extraction method from other DICE-based algorithms. By leveraging expressive generative models, Inffiusion-DICE can capture the multi-modality in R- and better draw upon the knowledge from d' (s.a) and n. (a,a)
offline RL algorithm, Diffusion-DICe also introduces
Toycase validation. We use a toycase to validate that the guide-then-select paradigm used in Diffusion-DICEE indeed brings minimal eтrоr exploitation that other diffusion-based methods suffer from. Here we aim to solve a 2-D bandit problem given a fixed action dataset. The action space is continuous and actions in the offline dataset follow a bivariate standard normal distribution constrained within an annular region. We show the dataset distribution TP, the learned diffusion behavior policy #P, the ground truth reward R and the predicted reward ft in Figure 2 Note that the true optimal reward occurs the outer circle. However, due to limited data coverage, the learned reward function exploits overestimation error on unseen regions, e.g., actions inside the inner circle have erroneous high values. What's worse, due fitting errors, the diffusion behavior policy may generate such overestimated actions. We take Diffusion-DICC with guide-based method, QGPO [32), and select-based method, IDQL [4] for comparison.
More specifically, we visualize the sample generation process in different methods. For guide-based method QGPO, as utilizes actions generated by $ and the value of on them to obtain guidance, the guidance exploits overestimation errors of OOD actions in the center and drifts the generated actions towards them. Note that adding a select step in QGPO will not help as the guide-step already incorrect. IDQL, because free of guidance, requires large number of generated candidate actions ensure the coverage of optimal action. IDQL utilizes R select from those suboptimal or -iitt--iistiittiinn actions, which however, are the sources of error exploitation R. For Diffusion-Dice because the first guide-step accurately guides the generated actions towards iupoupp while high-value regions (i.e., actions around the outer circle), in the select step, the learned value function Å could make a correct evaluation and successfully select the true best actions in the outer circle.
7

---

Table 1: Evaluation results on D4RL benchmark. We report the average normalized scores the end of training with standard deviation across random seeds. Diffusion-DICE (D-DICE) demonstrates superior performance compared to all baseline algorithms in 13 out of 15 tasks, especially on more challenging tasks.
Gaussian policy
Diffusion policy
DARL Dataset
CQL
IQL.
SQL.
O-DICE
Diffusion QL.
QGPO
IDQL.
D-DICK (nurs)
halfcheetah-m
44.0
47.4
48.3
47.4
51.1 +0.5
45.9 4:2.2
54.1 +0.4
$1.0
60.0
+0066
hopper-m
58.5
66.3
75.5
86.1
90.5 24.6
57.1 1:.1
98.01:26
65.4
100.2 13.2
walker2d-m
72.5
72.5
84.2
84.9
87.0 +0.9
77.9 4:2.5
86.010077
82.5
89.3 11.3
halfcheetah-m-rr
45.5
44.2
44.8
44.0
47.8 +0.3
37.1 11.7
47.6
45.9
49.2 +0.9
hopper-m-r
95.0
95.2
99.9
101.3 #0.6
86.2 1:91
96.9:::.66
92.1
102.3 +2.1
walker2d-m-r
77.2
76.1
81.2
83.6
95.5 1.5
65.1 4:5.6
84.41:4.1
85.1
90.8:12.6
halfcheetah-m-c
90.7
86.7
94.0
93.2
96.8 #0.3
92.6 +.05
93,5
95.9
97.3 0066
hopper- m-
105.4
101.5
111.8
110.8
111.1 #1.3
108.6 +-2.1
108.0
108.6
112.2 +0.3
walker2d-m-e
109.6
110.6
110.0
110.8
110.1 #0.3
109.8 1:0.2
110.7:10.6
112.7
114.1 -055
antmuze-u
84.8
85.5
92.2
94.1
93.4 #3.4
92.0 1:2.1
96,41:1
94.0
98.1 11.
antmaze-u-d
43.4
66.7
74.0
79.5
66.2 #8.6
85.3 4:36
74,4199.7
80.2
82.0
antmaze-m-p
65.2
72.2
80.2
86.0
76.6 +-10.8
81.3 +-2.6
83.6
84.5
91.3 +3311
antmaze-m-d
54.0
71.0
79.1
82.7
78.6 10.3
82.0 13.1
83.813.5
84.8
85.7
44.8
antmaze-l-p
38.4
39.6
46.4 +8.3
59.3 4:14.3
66.6
63.5
68.66
antmaze-I-d
31.6
47.5
52.3
54.0
56.6 47.6
45.5 4-6.6
64.8--5.5
67.9
72.0
+6655
kitchen-p
49.8
46.3
60.5+6.9
+3344
kitchen-m
51.0
$1.0
62.6
67.8
1:422
pen-human
37.5
71.5
72.8 4227
73.9
84.4
+:2.7
pen-cloned
39,2
37.3
57,3+3.6
54.2 +5.11
83.8
+3.2
4 Experiments
In this section, we present empirical evaluations of Diffusion-DICE. To validate Diffusion-DICE's ability to transform the behavior policy into the optimal policy while maintaining minimal error exploitation, we conduct experiments on two fronts. On one hand, we evaluate Diffusion- DICE on the D4RL offline RL benchmark and compare with other strong diffusion- based and DICE-based methods. On the other hand, we use alternative criteria to demonstrate that the guide-then-select paradigm results in minimal error exploitation. Experimental details are shown in Appendix D
4.1 D4RL Benchmark Datasets:
We first evaluate Diffusion-DICE on the D4RL benchmark ( and compare it with several related algorithms. For the evaluation tasks, select MuloCo locomotion tasks and AntMaze navigation tasks. While MuloCo locomotion tasks are popular in offline RL, AntMaze navigation tasks are more challenging due to their stronger need for trajectory stitching. For baseline algorithms, we selected state-of-the-arr methods not only from traditional methods that use Gaussian-policy (including DICE- based methods) but also from diffusion- based methods. Gaussian-policy -baedd baseline includes CQL (26). in-sample based methods IQL (24). SQL (55) and DICE-based method O-DICE [34]. Notably, O-DICE a recently proposed DICE-based algorithm that stands out among various DICE-bast based Diffusion-policy-
offline RL methods. Diffusion- policy- based baseline includes Diffusion- -LL 44 SiBC 4 QGPO
32 and IDQL [4]. We also compare Diffusion-DICe DICE with Gaussian policyy counterpart to show the benefit of using Diffusion-policy in Appendix D While SIBC and IDQL simply sample from behavior policy candidates and select according to the action evaluation model, Diffusion-QI. and QGPO will guide generated actions towards high-value ones. It is worth noting that Diffusion-QI. will also resample from generated actions after the guidance.
The results show that Diffusion- DICE outperforms all other baseline algorithms, including previ- ous SOTA diffusion-based methods, especially on MuJoCo medium, medium-replay datasets, and AntMaze datasets. The consistently better performance compared with Diffusion--O -L and QGPO demonstrates the essentiality of in-sample guidance learning. Compared with SfBC and IDQL, Diffusion DICE also shows superior performance, even with fewer action candidates. This is because the guide stage provides the in-support optimal policy for the select stage to sample from, which under- scores the necessity of sampling carefully from an in-support optimal action distribution, rather than from the behavior distribution. We refer to the comparison of candidate numbers under different envi- ronments in Appendix D. Furthermore, the substantial performance gap between Diffusion- -DCC and

---

O-DICE reflects the multi--mddlle of DICE's ' optimal policy and firmly positions Diffusion-DDL DICE as a superior policy extraction method for other DICE-based based algorithms.
4.2 Further Experiments on Error Exploitation
We then continue to verify that the guide-then-select paradigm used in Diffusion-DICE indeed exploits minimal error. This is obvious for the guide stage because we only leverage in-sample actions for both critic training and guidance learning. For the select stage, additional evidence is needed, as Diffusion-DICE also selects actions from generated candidates. Our key observation is that if the action value remains high during evaluation but results in low return trajectory, these action values must suffer from overestimation error.
Based on this, we choose to compare the aver- age state-action value and the average return be- tween actions selected by our guide-then-select paradigm and simply select from the behavior policy. Given identical critic and behavior pol- icy, we run Diffusion-DICE and its guidance- free variant for 10 episodes, comparing their average Q(s, a) values and their normalized scores. The results are shown in Figure [ We also compare the learning curves of their aver- age Q(s, a) values over time in Appendix D It's clear from the results that our guide-then-select paradigm achieves better performance while having less overestimated Q(s, a) values. This result validates the minimal error exploitation in Diffusion-DICE
Seacs Aehenk
Select bchari
Fredcadd
50 Natto7 bargegee We Fageen
Figure Actions generated by the guide-then-select paradigm result in better performance while have less overestimation error.
5 Related Work
To tackle the distributional shift problem, most model-free offline RL methods augment existing off-policy RL methods with behavior regularization term. Behavior regularization can appear explicitly as divergence penalties [50] 25 5 91. implicitly through weighted behavior cloning 48 38 [2], or more directly through careful parameterization of the policy [ [8]. Another way to apply action-level regularization is via modification of value learning objective to incorporate some form of regularization, to encourage staying near the behavioral distribution and being pessimistic about OOD state-action pairs [26] 24 555 [7]. There are also several works incorporating action-level
Offline RL
regularization through the use of uncertainty
2 or distance function
29
All these methods are based on the actor-critic framework and use unimodal Gaussian policy. However, several works indicate their limited ability model multi-modal policy distribution 49
14
4.
thereby leading to suboptimal performance. To remedy this, it's natural to employ powerful generative models to represent the policy. DT [5] uses transformer as the policy, Diffusion-QL [49]
SiBC (4)
QGPO (22) and IDQL 14) leverage diffusion models to represent policy. Other generative models
like CVAE and consistency model have also been used as policy in offline RL
588 54
7 Another
line of methods utilizes diffusion models for trajectory-level planning by generating high-return
trajectories and taking the corresponding action of the current state [7 M. While generative models are widely used in offline RL, few methods consider the existing errors in these models and whether the generation process exploits these errors.
DICF-based methods The core of DICE-based methods revolves around the ratio of the stationary distribution between two policies. This ratio can serve as the estimation target in off-policy evaluation [36] |7] or as part of a state-action level constraint term in RL [37 41 34]. In offline IL [53], it can connect the optimal stationary distribution of a regularized MDP with the dataset distribution
22 12 19.. In constrained RL, this ratio can induce the discounted sum of cost without an additional function approximator (28). Under the imperfect rewards setting, this ratio can reflect the gap between the given rewards and the underlying perfect rewards 30) All of these DICE methods consider this ratio as a transformation from one stationary distribution another. In this paper, however, we extend this ratio as transformation from the behavior policy to the optimal policy and propose a novel way of doing so by using diffusion models, which also serve as replacement for the Gaussian-han bbsed policy extraction in DICE.

---

Conclusion
In this work, we propose new duffusiin-basse offline RL algorithm, Diffusion-DIee Diffusion- DICE uses guide-then-select paradigm to select the best in-support actions while achieving minimal error exploitation in the value function. Diffusion-DICE also serves as a replacement for the Gaussian-bas based policy extraction part in current DICE methods, successfully unleashing the power of DICE-based methods. Through toycase illustration and extensive experiments, we show that Diffusion-DICE outperforms prior SOTA methods on a variety of datasets, especially those with multi-modal complex behavior distribution. One limitation of Diffusion-DICC of diffusion models is costly and slow. Another limitation is the training of diffusion models may suffer in low-data regimes. One future work is using more advanced generative models better choice.
the sampling process
40) as
Acknowledgement
HX thanks Harshit Sikehi for insightful discussions. The SJTU team is partially supported by Shanghai Municipal Science and Technology Major Project (2011SHZDZX0102) and National Natural Science Foundation of China (62322603, 62076161). HX and AZ are supported by NSF 2340651.
References
[1] A. Ajay, Y. Du, A. Gupta, J. Tenenbaum, T. Jaakkola, and P. Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15633 2022.
[2] G. An, S. Moon, J.-H. Kim, and H. O. Song. Uncertainty-based offline reinforcement learning with diversified g-ensemble. Proc. of NeurlPS, 2021.
[3] S. Boyd, S. P. Boyd, and L. Vandenberghe. Convex optimization. Cambridge university press, 2004.
[4] H. Chen, C. Lu, C. Ying. H. Su, and J. Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209. 14548, 2022.
[5] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Proc. of NeurlPS, 2021.
[6] H. Chung, Kim, M. T. Mccann, M. L.. Klasky, and J. C. Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv: 14687, 2022.
[7] Z. Ding and C. Jin. Consistency models as a rich and efficient policy class for reinforcement learning. arXiv preprini wrXiv:2309.16988 2023.
[8] Fu, A. Kumar, O. Nachum, Tucker, and Levine. D4rl: Datasets for deep data-driven reinforcement learning. ArXiv preprint, 2020.
[9] S. Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145 2021.
[10] Fujimoto. H. van Hoof, and Meger. Addressing function approximation error in actor-critic methods. In Proc. of ICML, pages 1582-1591, 2018.
[11] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In Proc. of ICML, pages 2052 2062, 2019.
[12] D. Garg, Chakraborty, C. Cundy, J. Song, and S. Ermon. Iq-leam: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 01:4008840333 2021.
[13] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
10

---

[14] P. Glansen-Estruche I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.1057. 2023.
[15] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 118800-6551, 2020.
[16] Ho, T. Salimans, Gritsenko, W. Chan, M. Norouzi, and Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633 8646, 2022.
[17] M. Janner, Y. Du, J. B. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint SXKi::2205009911,, 2022.
[18] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. ArXiv preprint, 2021.
[19] G.-H. Kim, S. Seo, J. Lee, W. Jeon, H. Hwang, H. Yang, and K.-E. Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In Proc. of ICLR, 2021.
[20] D. Kingma, T. Salimans, B. Poole. and J. Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-21100 -27077 2021.
[21] D. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proc. of ICLR, 2015.
[22] I. Kostrikov, O. Nachum, and Tompson. Imitation learning via off-policy distribution matching. In Proc. of ICLR, 2020.
[23] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. ArXiv preprint, 2021.
[24] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06166 2021.
[25] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances neural information processing systems, 32, 2019.
[26] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1191, 2020.
[27] Lee, W. Jeon, B. Lee, Pineau, and K.-E. Kim. Optidice: Offlinn policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pages 6120-6130. PMLR, 2021.
[28] J. Lee, C. Paduraru, D. J. Mankowitz, N. Heess, D. Precup, K.-E. Kim, and A. Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. arXiv preprini ar arXiv:2204.08955: 2022.
[29] J. Li, X. Zhan, H. Xu, X. Zhu, J. Liu, and Y.Q. Zhang. When data geometry meets deep function: Generalizing offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022.
[30] J. Li, X. Hu, H. Xu, J. Liu, X. Zhan, Q.-S. Jia, and Y.-Q. Zhang. Mind the gap: Offline policy optimization for imperfect rewards. arXiv preprint arXiv:2302.01667 2023.
[31] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787 2022.
[32] C. Lu, H. Chen, J. Chen, H. Su, C. Li, and J. Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning, pages 22825-22855. PMLR, 2023.
[33] A. S. Manne. Linear programming and sequential decisions. Management Science, 6(3): 259-267, 1960.
11

---

[34] L. Mao, H. Xu, W. Zhang, and X. Zhan. Odice: Revealing the mystery of distribution correction estimation via orthogonal-
update. arXiv preprint arXiv:2402..0338. 2024.
[35] O. Nachum and B. Dai. Reinforcement learning via fenchhel-rockafellar duality, arXiv preprint arXiv:2001.01866 2020.
[36] O. Nachum, Y. Chow, Dai, and L. Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in neural information processing systems, 32, 2019.
[37] O. Nachum, B. Dai, Kostrikov, Chow, L. Li, and D. Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXi:: 2019.
[38] A. Nair, M. Dalal, A. Gupta, and S. Levine. Accelerating online reinforcement learning with offline datasets. ArXiv preprint, 2020.
[39] A. Rényi. On measures of entropy and information. In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume 1: contributions to the theory of statistics, volume 4, pages 547-562. University of California Press, 1961.
[40] A. Shocher, A. Dravid, Y. Gandelsman, I. Mosseri, M. Rubinstein, and A. A. Efros. Idempotent generative network. arXiv preprint wrXiv:2331.014460 2023.
[41] H. Sikchi, Q. Zheng, A. Zhang, and S. Niekum. Dual rl: Unification and new methods for reinforcement and imitation learning. arXiv preprint arXiv:2302.08560. 2023.
[42] Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[43] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arX1v:2011.13456. 2020.
[44] Y. Song, P. Dhariwal, M. Chen, and I. Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023.
[45] R. S. Sutton, A. G. Barto, et al. Introduction to reinforcement learning. MIT press Cambridge, 1998.
[46] Vincent. A connection between score matching and denoising autoencoders. Neural Comput., 1367)1166 201 doi: 10. 102166EEOOO L00142, URL httph:ttt///t (efoo.oorg/11 or.//1.. 1162/ HECO_a_00144
[47] X. Wang, H. Xu, Y. Zheng, and X. Zhan. Offline multi-agent reinforcement learning with implicit global-to-local value regularization. In Thirty-sevnnth Conference Neural Information Processing Systems, 2023.
[48] Z. Wang, A. Novikov, K. Zolna, J. Merel, J. T. Springenberg. S. E. Reed, B. Shahriari, N. Siegel. Ç. Güllçehre, N. Heess, and N. de Freitas. Critic regularized regression. In Proc. of NeurIPS, 2020.
[49] Z. Wang, J. Hunt, and M. Zhou. Diffusion policies as expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.066 06133, 2022.
[50] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. ArXiv preprint, 2019.
[51] H. Xu, X. Zhan, Li. and H. Yin. Offline reinforcement learning with soft behavior regulariza- tion. ArXiv preprint, 2021.
[52] H. Xu, I.. Jiang, L. Jianxiong, and X. Zhan. A policy-guided imitation approach for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 35, pages 4085-4098, 2022.
12

---

[53] H. Xu. X. Zhan, H. Yin, and H. Qin. Discriminator seeeriiinatorwwiightedd offline imitation learning from suboptimal demonstrations. In International Conference on Machine Learning, pages 24725- 24742. PMLR, 2022.
[54] H. Xu, X. Zhan, and X. Zhu. Constraints penalized q-learning for safe offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8753 8760, 2022.
[55] H. Xu, L. Jiang, J. Li, Z. Yang. Z. Wang, V. W. K. Chan, and X. Zhan. Offline rl with ood actions: In-sample learning via implicit value regularization. arXiv preprint MxXi::3333.58/11 2023.
[56] X. Zhan, H. Xu, Y. Zhang, Zhu, H. Yin, and Y. Zheng. Deepthermal: Combustion optimiza- tion for thermal power generating units using offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 4680 4680,4688,4 2022.
[57] R. Zhang, Dai, L. Li, and D. Schuurmans. Gendice: Generalized offline estimation of stationary values. In International Conference Learning Representations, 2019.
[58] W. Zhou, S. Bajracharya, and D. Held. Latent action space for offline reinforcement learning. In Conference on Robot Learning, 2020.
[59] Z. Zhu, M. Liu, L. Mao, B. Kang, M. Xu, Y. Yu, S. Ermon, and W. Zhang. Madiff: Offline multi-agent learning with diffusion models. arXiv preprint arXiv:2305. /7330, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our main claims can be reflected by Section 3 and Section 4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
Seed Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Experimental details are given in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
Code provided the project link.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
http://udddesSoeeubbi.icc cissinnnPicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
httpa////ggideee/eeeubbbbiiiiiia Sot////ooeSSeiiaaeeeeeeee wwwsssssioiiiyyss for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental settings are given in Section 4ad Appendix D in Appendix(D
Details are given
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We use sufficient random seeds and provide standard deviation of the normal- ized scores in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details of computation resources are given Appendix E
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: This paper doesn't 't involve human subjects or participants and the datasets in Section are common open-sourced datasets.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Potential positive impacts and negative impacts are discussed in Appendix Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets we use are properly credited, and the corresponding license are given in Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
datasets.paperrsst
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19

---

A A More Detailed Discussion of DICE and Diffusion Model in RL
Derivation of learning objectives in DICE DICE algorithms consider the following regularized RL problem as a convex programming problems with Bellman-flow constraints and apply Fenchel- Rockfeller duality or Lagrangian duality to solve it. The regularization term aims at imposing state-action level constraints, [35] 27 34).
max ((( (r(8, ((s,))aa-(((s,)) - aDy(d (D
(s, ())a))
(11)
Here (s a) (s,a)) is the f-divergence which is defined with ()(P))
=
E [ [1 M()
((w)) ]]
Directly solving x" is impossible because it's intractable to calculate d' (s.
However, one can change the optimization variable from to d= because of the bijection existing between them. Then with the assistance of Bellman- -loow constraints, we can obtain an optimization problem with respect d:
max (s.a) (s.a) aD (d)(d d20
s.t.
Σ d(s.a)= REA
+
d's'' d'), (s'
Σ
(12)
Note that the feasible region has {d: Vs E S,a E A, A.d(s,a) a) 2 0} because should be non-negative to ensure valid corresponding policy. After applying Lagrangian duality, we can get the following optimization target following [27]
min max ((s.a)~al((s a)] (oD(((s))))
(e) 420
(1- +7 Σ
d(s', d') (s/aa
> d(s
min max 1 Edosss))
((a) w20
+ E R w/s (o(s,))(((s00)) + EV6Slo,))V( v(s)) aIE sa [[(](s, a))
(13)
(s.a)) for simplicity. By incorporating the non-negative constraint of d and again solving the constraint problem with Lagrangian duality, we can derive the optimal solution
Here we denote w(s, a) as d(s.a)
w (s, a) for the inner problem and thus reduce the bilevel optimization problem to the following
optimization problem:
min(1 - Ed)) [V(s)]] + alE f r(s,,))+ V(())
a) + 2. p(s') p(s).,))))) -
(14)
Here f variant of f's convex conjugate. Note that in the offline RL setting, the inner summation Ev p(s'|s, (s")) is usually intractable because of limited data samples. To handle this issue and increase training stability, DICE methods usually use additional network Q(s, a) to fit r(s,a) - Ee p(s'|s, (s'),, by optimizing the following MSE objective:
min E E [(r(s, + 7V(()) Q(s, ((s,a))
(15)
Q
And because of doing so, the optimization objective in 14 can be replaced with:
min Es (1 7) [s)] + E(saa)a [af" (IQ(s, a) V(s) /a)]
(16)
Also note that to increase the diversity of samples, one often extends the distribution of initial state du to dd by treating every state in trajectory initial state [22
Discussion of diffusion policy and other methods in offline RL In offline RL, as discussed before, most diffusion-based algorithms utilize diffusion models to represent policy æ. This policy can either be the behavior policy [1 4. the optimal policy [49 32]. When representing policy with diffusion model, the ultimate goal is to model the optimal policy n with diffusion model. Different
20

---

from other tasks in image generation or video generation 43 [6]. RL tasks will not provide data samples under π², while the agent has to learn such optimal policy. This makes directly leveraging the diffusion model to fit the optimal policy impossible. However, besides these methods that are based on diffusion policy, some methods utilize diffusion model fit the entire trajectory's s distribution 17 M. These methods try to generate the optimal trajectory based on the current state. After generating the entire trajectory, these methods simply take the action corresponding to the current state. It's worth noting that although these methods avoid modeling '², extra effort is required to generate high-quality trajectories.
B Additional Proofs
Optimal policy transformation with stationary distribution ratio: Given that x² the behavior d=.a,)) and x² are the optimal stationary distribution correction and its corresponding optimal policy, x² and n² satisfy the following proposition:
policy, (=(s,0)
d (s.0)
π (als)
(x,a)
π D (als)
(17)
d' (s.a) SacA (s.a) T (als)dda
Proof. Using the definition between d' (s) and d= (s, we have for any n, dt (s. ((ass) Then have:
d' (s.a)
(e,0)
(s.a)
T (alss)
((ssa)
7 (a(ss)
(s.a) SaEA ("(aa)) v (als)da
Lasn d' (s,a)dd
(3)
(18)
d P (a)s) (s)
d' ds,
π (s)
(a|ss
Verification of 66 The score function of the optimal policy x² (als) in DICE and the behavior d'(s.an)
policy - (a(s) satisfy: Va, log m/ (acls) VV, log n. (aels) Va, log E Ea (aglo..e) (oufo..))la"(s..) (s.()
We mainly follow the proof in Lu et al.
322 to verify this equation. Given the relationship between
nt" (a)s) and x (a|s) in Eq. 18) we have:
म (arls)
p(acloo)Tó s s)dan
/ plaelao, s)T" (aols) (aols))))) doo
d' (a.00)
/ p(arlao, 8)
(s.0g]
π (ao (s)d
SaEA (s.0)))
(*.0)
(als sd
d' (e.an)
- f placlass s)T (auls)
(8.00)
(19)
day
JoeA (P,a) n (a) s)da
(s.a)))
(s.00)
/ n. (anlat, 8) (axls)
d (s.0a)
dag
SoEA a (s.a) n (als)da
d' (a.0)
d" (2.00)
= =)
d (x.0g)
d=(aa)) SaEAA (s,0) n P (als)da-
Note that the second equation holds because p(ay (ao) represents the forward transition probability in the diffusion process, which is unrelated and xo is exactly the optimal policy л². The
21

---

forth equation holds for the similar reasons. We exchange the condition variable to derive the final relationship between zz (als) and मद (a]s). Having this relationship, we can directly calculate their score functions as follows:
d' (#.00)
V. 0, log qi (arls) Vo, log me (arls) + Vag log Bag P ar,
(s,0g)
JaEA (2,0)
(=,00))
(a)s)da
(s,a) E Bg (aoles.))
d' (s.00]
Va, log mi (arls) + V Va, log
(3,00)
(20)
SacA (a,a))
d' (s. ao)
Va, log t. (axls) + Vay log Engna P (ap a(.s)
d 0n)
Note that the second equation holds because the integral SasA and because of so, its gradient with respect to becomes O.
d' ('(aa) dd (x.0) π (als)da unrelevant to do.
Lemma 1. Given random variable X and its corresponding distribution P(X). for any non-negative function fx),, the following problem is convex and its optimizer is given by yf log P()) (x))
min P(x
(21)
Proof. Because is the constant optimizer, this problem can be reformulated as:
min ((x) *
(22)
We first verify that this problem is actually a convex problem with respect to y. Because f(x) is non-negative function, can take the second derivative of this objective and get:
[x= -
(23)
Because f(x) is non-negative function, the second derivative is also non-negative, which means this problem is convex problem with respect to y. For convex problem, the optimizer can be induced from the first-order condition:
(E v²) 0
(24)
8" log (x(](((
Theorem 1. Va, log (w oo)) can be obtained by solving the following optimization problem:
min Bo (P E plaas (p)) ((sssa))e ag)e (0(sa.))) go (s, ae 0]
(25)
The optimal solution 0* satisfies Va, Vo.go (s, az, Va, log (aolen oo))
Proof. Because the forward diffusion process simply adds noise to ag and is unrelated to s. taking as condition doesn't i affect p(a))ap) (o)). This means p(arleo, t) p(a, ag). By changing the conditional variables from (1:K) to (I-K)) we have:
E4~u((, E (agls) Epiar ao) w(s ao)e ao)e -8ll....)) + go at,()
(o (agls) (p(arlaa.)) w(s, aole go(s,ait) + go at))
a0(s)aed) + go(s, ar, (a w(s agle - so(a,a)))) + go t) = Exuus0.2 (a E 1a s) (w)s ao(s go(s, a,,
EE- E, (a, w(s -g)e--olsraedt)
(26)
EE-U4(0.T)) (a
(aole
22

---

We first take the first-order derivative with respect to each go a) for this objective and solve its saddle point:
V go(em R (T [1o(s, ao)]e -9((s,).))) + (0(0,,r.))
(27)
T (ayls)) ( (oolar)) [wss, av)l =o(s))))) +1)
Because any possible intermediate action az has positive value of π (ae(s)), the saddle point 9o- g0-(8, at, t) must satisfy:
- Ex (aol(a)) (0ss, aol]e 90- (c,00.)) + =1=00
(28)
go- (s, az t) log Pao ( an)]
and then Va,90- (s, ar = Va. log E ( s) (ws,, ao)l We then take the second-order derivative with respect to each go (s, az, t) for this objective:
v 9o(e,a, En (n Ex [
[wo(s, ao)] go(s.))))) + go(s, ()
(29)
P (a. (s) E (aolaa.)) (ss an)] =0(s,a).))
T
Because w(s, an) has to be non non-negative for any ag to ensure a valid transformation from TP to x². it's obvious that the second-order derivative always non-negative. This means this objective convex with respect to go(8, or, and has convergence guarantee given unlimited model capacity. D
Lemma 2. The following (a( is a well-defined f f-divergence:
(x-1)² 1)2 fo(z) lolog= - 0 0 x <<
I > 1
(30)
Proof. By the definition of f-divergence in Rényi 139, fz(x) should be a convex function from lim fz(2).
+00) + too]. fz) should be finite for all x and f1( (0)
We first verify that (x) is convex. By taking first-order and second-order derivatives on fp(1)
have:
(2(x-1) - 1) x21
2 x21
((x)
f((x)
(31)
log x 05I<1
0<z<1
It's obvious that (x( is continuous and (x) is strictly positive. According to the second-order condition for convex functions, fp) is convex.
We then move on to verify the rest of the properties. It's obvious that given specific > 0. fp(a)) is finite and (1) = O. In mathematics, the value of x log æ when x is defined by its right limit, which means (0) lim,-ot fz(z)= 1.
Proposition 1: Given the piece-wise f diverrgenc in eq eq
30,,
(f)))(()))) and (x) has the following
formulation:
3
(z)
+
fz(x)
4
+ x
x x20
(32)
c
c
x <<0
Proof. We begin with the proof of (c)) Given Lemma 2, the form of (x) directly comes
from the continuity and monotonic increasing property of (x)) One can easily verify (x))ss form by calculating the reverse function of (x))
We then continue with the proof of ft (z). By the definition Fenchel conjugate we have:
(z)
sup
I. 2 f2(2)
(33)
=E(0,+00)
23

---

Because here fp(x) is piecewise function, the supremum could be achieved in either [0,1) 1) or [L, (..) This means we should first take supremum over two intervals respectively and then choose the greater value as J (x)) More specifically, we perform the following decomposition:
(z) max
sup x 2 -f((),, =0(1,too) =E(0.1)
sup x. 2-2-fp(2) Sp(2))
(34)
When constrained either [o, 1) or [1, +o0), fp(2) is no longer piecewise, and the inner supremum becomes the Fenchel conjugate, albeit under a narrower interval. To calculate Fenchel conjugate under limited interval, one should observe that for any given 2, the derivative of the inner objective satisfies:
d(x 2 fp(2)) = x f/(2)
(35)
dz
By Lemma 2, we have that (2) is strictly increasing. This means d(x-2-1<(s)) has only three possible behaviors any interval: always positive, always negative, transitioning from positive to negative with a zero crossing within the interval. Then for any interval (a, bj, sup=e(a,t x. f2(2) has the following formulation. Note that similar results can be extended to open intervals.
da
σ (F)) (x) fo(( fp(b) else
if (f)) E [a,
(z) = max x a a
(36)
Leveraging this formulation, one can easily derive the close form solution of 34 In fact, (x) has the following formulation:
(x) max ( + f x>0 x
2 20)
(37)
60 - x
x
The final form of f (z) could be derived by directly taking the maximum.
C In-depth Discussion of In-sample Guidance Learning and Other Guidance Methods
Difference between IGL and other inexact guidance methods One important property of IGL is that it can induce the exact optimal policy, rather than a policy similar to the optimal policy, the optimal policy can be represented
weighted behavior policy. Some of the previous methods
inexact guidance terms and thus have no guarantee of the induced policy [17 16 6
Among them,
Janner et al. [17 uses mean-squure--rrrr (MSE) objective to train the guidance model. Ho al. [16), Chung et al. [6] leverage training-free guidance by reusing the pre-trained diffusion model in the data prediction formulation characterize intermediate guidance.
Difference between IGL and Contrastive Energy Prediction (CEP)
Besides the inexact guidance methods mentioned above. Contrastive Energy Prediction (CEP) [3 is guidance learning method that could induce exactly the desired policy if it can represented as a weighted behavior policy. However, CEP is still prone exploit overestimation enror in the critic. We provide additional explanations for the differences between Contrastive Energy Prediction and In-sample Guidance Learning, well as the reasons for these differences. Intuitively, the difference mainly stems from CEP's inherent property as contrastive loss.
In order to obtain Va, log Ep(aolas .)) [wr(s, no)]. CEP will first train model go (s, ar, 1) to fit log Ep(agla,s)) [an(s, oo)] and then take the gradient of it. Its optimization objective has the following contrastive form:
e0 - (s(a.))).))
E E (100)).33))) p (oo)(t:81,)) .)) [-E w (s, (0) log
(38)
Σ 20 (0(0(()))))
Note that here K must satisfy K to ensure the optimal condition is go- (s, ar, t) Va, log Ep (wss, ao]]. This means under offline setting, additional fake actions are required on every state in the dataset. CEP first uses diffusion model to pre-train a behavior policy a from the
24

---

dataset and then generates multiple fake actions (ao)(1:k) ~ (agls)) for each state in the dataset. The second expectation in the objective will be calculated using these fake actions. It's clear that when R mistakenly generates OOD actions, CEP still uses action evaluation model to predict w(s, ag ( on these OOD actions. The introduction of wo(s, a0 (DOD) can make Va, log Pp((ola...) [wr(s, (o) pre- fer these a0on. This is because Va, log (p(oolae.s) (wss, ao)] will guide the samples towards actions with high wr(s, ao) value. These high w(ss, (o)) value actions can be OOD actions whose wo(s, do) is overestimated. The catastrophic result is that CEP may guide action samples towards OOD actions with crroncously high w(ss, ag) value rather than real high-quality actions.
Finally and most importantl, we will explain from a probabilistic perspective why the contrastive loss in CEP can't be estimated with one sample from 7T (a|s). The derivation of the optimality condition also depends on changing the conditional variables from (1o)(1:K) to (a.) (1:K);
K
90 (O)
Ep() xN (10)))
E ( ((00)))))
.))
w(s (ao) log
f== ed (s((a))0),0)
K
- (s((()))))))))))
= E(o
Σ
w (ao) log
Σx 20 (s(aa)3o)))
Σ w(s (o) log cea (((()(aa))))
K
[-
= =Er()
(( (((
Σ c (s.(a,)))..))
(39)
To derive the optimality condition that (s. (t.) = log E
p (1an) [wsss, (a)) for
each i, one have to decompose the joint probability (1:0) (1:K), s) into the product of cach
(ao)*s conditional probability:
K
c0 (11aa)))))))
nx nxx p ((0) [-
Σ
w(s (1) log
f== cgu (s((a))o),))
w(s (i)) log e (s(lar))))) .))
K
= =Ep(o (p((a,)0:x11))
Σ
E (aa
E/ e0 (s((a))3))e)
(40)
Note that here we absorb the expectation over 11 (ap) (1:k)) 5) into the sum over and
replace it with p( (00 [a:) s)). This is because each reverse process p((a)) (aa))) (1)
(0)s)
s) is
independent of other (a+) (G))))
The requirement of decomposing joint probability
(1.) s) means that (ao) (1:8) should be mutually independent from each other, once (ar) (ar) (1:K) and are determined. Obviously, if we only have one sample of og. (oo) must be identical and not independent. On the other hand, because the objective of In-sample Guidance Learning doesn't 't contain expectation over any joint distribution, it can be estimated with only one sample from π"" (als).
D Experimental Details
Additional details for the practical algorithm In this part, we further provide the implementation details based on the given pseudo-code. Firstly, the score network we used is based on a U-net architecture, which is fairly common in diffusion- based RL algorithms
59 32 For the DICE part, we use the double Q-learning trick [10] for the training of Q. For sampling, we use DPM-solver 31 to accelerate the sampling of the diffusion model, while the score function is given by IGL. The implementation of our score model and sampling process are based on DPM-solver, which uses MIT license. Iiffusion-DICF contains syper-paramedte in total, which are a in the DICE part and K during sampling.
25

---

Toy case experimental details As shown in Fig. 2 the behavior policy follows a standard Gaussian distribution constrained in an annular region. We sample 1500 times from the ground truth behavior policy to construct the offline dataset. The reward model R represented with 3-layer MLP and because we don't It have to model complicated policy distribution, the score network ég in the toy case is simply multi-layer MLP. For better visualization, we sample 500 candidates during the guide stage. For QGPO, because it doesn't 't have the select stage, the sample distribution remains the same. For the other 2 methods, we select actions from candidates generated either from diffusion behavior policy (IDQL), from guided policy Diffusion-DICE for 64 times. The reverse diffusion trajectories contain 16 action outputs. Because these 3 methods stand for different paradigms, we tuned different hyper-parameter sets for all of them achieve the best performance.
D4RL experimental details For all tasks, we ran Diffusion-Dice for 10 steps and reported the final performance. In MuJoCo locomotion tasks, we computed the average mean returns over 10 evaluations across different seeds. For AntMaze tasks, we calculated the average over 50 evaluations, also across 5 seeds. Because the inference of diffusion model is relatively slow, we conduct the evaluation every 4 - 10 training steps. Following previous research, we standardized the returns by dividing the difference in returns between the best and worst trajectories in MuJoCo tasks.
In AntMaze tasks, we subtracted I from the rewards following previous methods [23 34 Note that for the D4RL benchmark and its corresponding datasets, Apache-2. 2.0 license is used.
For the network, we use a 3-layer MLP with 256 hidden units to represent Q and V. For the guidance network go. we use a slightly more complicated 4-layer MLP with 256 hidden units. Note that the time embedding in the guidance network is the Gaussian Fourier projection. We use Adam optimizer [21 to update all the networks, with a learning rate of 3 10 4, The target network of Q used for double Q-learning trick soft updated with a weight of 5 10-3
For the baseline algorithms, we report the performance of IDQL under any number of hyper- parameters. Other baseline methods' results are sourced from their original papers. For all tasks, we list the chosen significantly fewer action candidates during the select stage and we present the comparison of K between Diffusion-DICH and IDQL in Table (3.
in Table 2 As mentioned before, Diffusion-DICC requires
To further demonstrate the superior performance of Diffusion-DICe DICE compared traditional DICE algorithms, we conduct ablation studies by replacing the policy in Diffusion-DICH with a simple Gaussian policy, while preserving the piecewise f-divergence. The results are given in Figure
D4RL Results for Gaussian Policy and Diffusion Policy
Gaussan Policy Dificiion Policy
Nomralized
Fag
Popper
Balk
1230e
or
Figure 4: Although using piecewise f-divergence, the Gaussian policy in traditional DICE algorithm still results in inferior performance than using diffusion policy.
We also perform hyper-parameter study for candidate number K. To compare the performance of Diffusion-DICE under different K, we select some of the datasets in D4RL benchmark and test its performance following the same protocol before. Because the optimal K for MuloCo locomotion tasks and AntMaze navigation tasks have significant differences in magnitude, we chose to test neighboring orders of magnitude around the optimal K value. The results are given in Table
Comparison of average Q(s, a) curves For a detailed comparison between the guide-then-select paradigm and the select-from-ber behavio paradigm regarding error exploitation, we present their average Q(s,a) a) curves normalized returns in Fig. 5 It's evident from the results that the average Q(s, a) curves of the select- froeee-ro--bbee nroo-behvvior paradigm almost remain above the guide- theeee paradigm, while the latter exhibits better performance.
Learning curves on D4RL. benchmark Figure 6 and Figure] 7
We present the learning curves on D4RI, benchmark in
26

---

Table 2: The chosen and K in Diffusion-DICE.
Dataset
a
K
malfcheetah-mediu--e
0.6
128
hopper-medium:
0.6
32
walker2d-medium--
0.6
32
haffchetal-mcciiu--rpaaaa2
0.6
128
hopper-meddium--eppay-222
0.6
32
walker22dmmediuu--eppyyyy
0.6
32
hallchhetth--meduuueepppp....
0.6
128
hopper-meddium-exxpettV2
0.6
32
wallerr2d-medhhu--exprttzZ
0.6
32
natmaze-umazze-ee
0.06
4
natmazz-umazz--dierree-e
0.8
1
uutmaze--mcdium-play-a
0.06
8
antmaze-medduum-diveres-=2
0.06
8
antmaze-large-play-22
0.06
antmaze-lage-diivrre--2
0.06
8
Table 3: Comparison of K between Diffusion-Dice DICE and IDQL
Dataset
D-DICE
IDQL
32
256
hopper-medium--22
32
256
valker2d-medium--
32
256
halfcheetah medium medium-replay replay-v2
256
hopper-mecium--eeppee -epplluuu-r-p-2yy22
256
walker2d -weeiduddmdmmm-m---e medium-reelay--v replay-v2
32
256
malffhectah-mdddum-cpppe
32
256
32
256
walleer2d-medlumm-expeteee
32
256
unlmaze-umaze-vy
4
32
antmaaze-mediumppaa--222
8
antmmaze-mcddimm-diiersss-/
8
mutaazelarge--play-y
8
32
antmazelarge-diivrre--2
8
32
Table 4: Hyper- parameee Study
K
16
32
64
128
malfchectah-mediun-22
48.8
52.2
55.6
58.4
60.0
hopper-medium-v2
89.6
96.8
100.2
98.9
96.1
walker2d walker2d-medium--22
82.7
85.6
89.3
88.2
85.4
Dataset
sallihetah-mmdduu--cpppeet
94.7
95.4
95.9
96.6
97.3
hopper-medium- -xxeetttt--
109.3
110.6
112.2
111.9
110.8
walker2d medium-v2
111.2
112.7
114.1
114.0
113.2
K
2
4
8
16
antmaze-umaze-diivrrs--2
82.0
76.2
70.5
60.4
51.8
antmazz-mediumm-divensss
78.8
81.2
83.3
85.7
82.9
Dataset
antmaze-large-play-v2
55.8
61.0
65.3
68.6
67.2
antmaze-lagge-divrrs----
60.2
66.8
70.1
72.0
69.9
27

---

walker2d replay.v2
39
70-
00
guide qu ded-the---eeeette value
guide whw.--hhhheeeeeeee ect O value
20
guide then-select score
guidde then-sheert score
select from beh Q valke
colect-ficm-beee bohhccaaooo value
10
wwwect-from www.se--f---bbbrbbeeeeoeoeoee avier score
sellcd-iccm-bbhaatte score
200
400
620
800
1000
200
000
60C
1000
woller22d-meium-22
hopper-medlum-22
100
70-
50
70-
200
400
600
800
1000
200
400
600
50C
1000
Figure 5: Q(s, a) curves for guide-then teile-then--elerr paradigm and select-from- paradigm.
roltcheetal-meddunee
hopper-medium--22
wakkrr26-medlum22
200
100
102
Nomalized
50
0.2
0.4
0.8
0.8
1.0
0.0
0.2
0.4
0.5
0.5
1.0
6.9
0.2
0.4
0.5
0.8
1.0
Training Steps (1e6)
Training (3e6)
Trilning
hoppen-mediuee cespeemmdddu--rpppaaaaa
ankker/6-meddn--ejpyya2
1.00
73
Nonalted
%
1.0
Taaning (1e8)
Training
(1e6)
wellber 2eillere/immiiimeeeeettee.
1.00
Nomakzod
Nomakozd
52
-50
0.
0.0
0.2
0.4
D.6.
0.6
1.0
0.9
6.2
0.4
0.6
0.9
1.0
Traning Steps (1e8)
Training Steps (1e6)
Training Steps (1e6)
Figure 6: Learning curves of Diffusion-DICE D4RL MuloCo locomotion datasets.
28

---

entmaze umaze +2
attmmaze-meoee medium-mmmmmmmmmmm vlay
antmaae large play-v2
73
75
Nomalzze
Nomalized
3.
0.0
044
54
6.0
0.4
0.6
0.6
Training Steps (1eb)
Training Stegs (1eb)
Thaining Steps (1eb)
antmmazeearr gattmaaelareeddieeeeeee waw.aaageiiiieee222222
107
100
100
Somalized
Nomelized
Komalzzd
90
30
3.2
0.2
0.4
0.8
0.5
1.0
0.2
0.4
245
248
142
9.0
0.2
0.A
0.8 (106)
0.8
1.0
Training Stops (1e5)
Tranning (1.6)
Training
Figure 7: Learning curves of Diffusion-DICE on D4RL AntMaze navigation datasets.
E Experiments Compute Resources
For all the experiments, we evaluate Diffusion-DICE on either NVIDIA RTX 3080Ti GPUs or NVIDIA RTX 4090 GPUs. The training process of Diffusion-DICE takes about 6 hours on NVIDIA RTX 3080Ti or hours on NVIDIA RTX 4090. The evaluation process of Diffusion-DICE takes about 3 hours NVIDIA RTX 3080Ti or 2 hours on NVIDIA RTX 4090. Because we parallelize training and testing process, a complete experiment takes about 8 hours on NVIDIA RTX 3080Ti or 4 hours NVIDIA RTX 4090.
F Broader Impacts
Besides the positive social impacts that Diffusion-DICE can help solve various practical offline RL tasks, for example, robotics, health care, industrial control tasks, some potential negative impacts also exists. Part of the Diffusion-DICE algorithm, the In-sample Guidance Learning, can also be applied other generative tasks like image generation. This could be used generate fake images, which could possibly mislead the public after being spread online.
29


Reviews:
[{"review_id": "TZwntAMILP", "reviewer": "Reviewer_QJqE", "summary": "This paper introduces Diffusion-DICE for offline reinforcement learning. Diffusion-DICE motivates from the transformation between the behaviour distribution and the optimal distribution, which inspires the use of generative models for behaviour distribution modelling. Next, Diffusion-DICE decomposes the policy score function into two components, one from the behaviour distribution, another from the guidance, i.e., the transformation. Lastly, Diffusion-DICE employs a guide-then-select paradigm, which uses only in-sample actions for training to avoid out-of-distribution issues. In their experiments, Diffusion-DICE has achieved strong performance compared with baselines on the D4RL benchmark.", "questions": "Could you provide a bit more discussion about the differences of using guidance for inference with the Diffusion-QL style inference, which directly guides the sampling process towards actions with high returns? It would be interesting to understand the pros and cons of these two different paradigms", "limitations": "I think in general this is an interesting work and I don’t see major limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed idea is novel and interesting. I like the way the authors connect DICE with diffusion policies, decompose the score functions, and make the guidance score tractable. Theoretically, they have provided careful analysis and derivations to support the claims. Empirically, they have conducted both toy experiments for intuitive understanding, and demonstrating the strong performance on the D4RL benchmark.", "weaknesses": "There are several weaknesses of the paper I’d like to point out.\n\n1/ The biggest issue is the presentation. Although I do like the idea of the work and recognise its contributions, I found the paper very hard to follow and needed to read through the paper to understand the introduction. The way authors presented the guide-then-select is confusing. I’d suggest authors provide a bit more background and carefully define the “guidance term”, and how it relates to the RL before using it in both abstract and introduction. \n\n2/ Achieving in-sample learning of offline diffusion RL is not new. Efficient Diffusion Policy (EDP) [1] has introduced an IQL-based variant which naturally allows training Q-values using only in-sample data, without querying out-of-distribution actions during policy evaluation. I’d suggest the authors carefully check the claims and avoid over claiming.\n\n3/ The D4RL experiments are only conducted on the locomotion tasks and the antmaze tasks. The commonly tested kitchen and adroit tasks are missing, which weakens the claim of the paper.\n\nReferences:\n\n[1] Kang, B., Ma, X., Du, C., Pang, T., & Yan, S. (2024). Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's time and effort dedicated to evaluating our paper, as well as the constructive feedback provided. In response to the concerns and questions raised, we have prepared detailed answers, which are outlined separately below.\n\n>... The way authors presented the guide-then-select is confusing. I’d suggest authors provide a bit more background and carefully define the “guidance term”, and how it relates to the RL before using it in both abstract and introduction.\n\nWe apologize for the lack of explanation of 'guidance term' in the abstract and introduction due to the page limit. The 'guidance term' refers to the log-expectation term defined in Eq. (6), which 'guides' the diffused action towards high-value regions. In the updated version, we will provide more background information on this term and its relation to RL in the introduction.\n\n>Achieving in-sample learning of offline diffusion RL is not new. Efficient Diffusion Policy (EDP) has introduced an IQL-based variant which naturally allows training Q-values using only in-sample data, without querying out-of-distribution actions during policy evaluation. I’d suggest the authors carefully check the claims and avoid over claiming.\n\nWe acknowledge that EDP is also a diffusion-based algorithm that avoids querying OOD actions' value, and we will add it to the discussion in the updated version. However, the major difference between EDP and Diffusion-DICE is that EDP has no guarantee of the form of optimal policy. According to EDP's original paper, it discusses two types of approaches: \"direct policy optimization\" and \"likelihood-based policy optimization\". It's obvious that the former can not guarantee the form of optimal policy. The latter replaces $\\log \\pi_\\theta(a|s)$ with its lower bound in the optimization objective, which consequently loses guarantee for the form of optimal policy. On the other hand, Diffusion-DICE directly calculates the score function of the optimal policy induced from DICE's objective. As both terms in Eq. (6) can be estimated unbiasedly, the policy distribution induced by Diffusion-DICE can match the exact optimal policy distribution.\n\n>The D4RL experiments are only conducted on the locomotion tasks and the antmaze tasks. The commonly tested kitchen and adroit tasks are missing, which weakens the claim of the paper.\n\nTo validate that Diffusion-DICE also demonstrates superior performance on other more complex tasks, we evaluate Diffusion-DICE in kitchen and adroit environments. Due to limited rebuttal period, we choose 2 tasks from kitchen and 2 from adroit. We compare Diffusion-DICE with other Offline RL baselines (either diffusion-based or not). The results are either copied from their original papers (if available) or LD[1] (if not). The results and the chosen hyperparameters are as follows:\n\n|                 | Diffusion-DICE | EDP | LD[1] | Diffusion-QL | QGPO|IQL|$f$-DVL|\n| ---------- | ------ | ------ | ------ | ------ | - |-|-|\n| kitchen-partial| **78.3** | 46.3 | -  | 60.5 | - | 46.3 | 70.0 |\n| kitchen-mixed |  **67.8**  | 56.5  | - | 62.6  | - |51.0| 53.8 |\n| pen-human | **84.4** | 72.7 | 79.0 | 72.8 | 73.9 | 71.5 | 67.1 |\n| pen-cloned  | **83.8** | 70.0| 60.7 | 57.3 | 54.2 | 37.3 | 38.1 |\n\n|      | $\\alpha$ | K    |\n| ---- | -------- | ---- |\n|   kitchen-partial   |    0.6      |   4   |\n|    kitchen-mixed  |   0.6       |   4   |\n|   pen-human   |    0.6      |   4   |\n| pen-cloned | 0.6     | 8 |\n\nNote that we follow the same experimental settings in Appendix D. The results further substantiate our claim that Diffusion-DICE achieves optimal policy transformation while keeping minimal error exploited, and thus exhibits SOTA performance even on more complex tasks.\n\n>Could you provide a bit more discussion about the differences of using guidance for inference with the Diffusion-QL style inference, which directly guides the sampling process towards actions with high returns? It would be interesting to understand the pros and cons of these two different paradigms\n\nThe major difference comes from the way score function is modeled. Diffusion-QL represents algorithms that directly model optimal policy's score function with one neural network. Diffusion-DICE represents algorithms that indirectly model it as a \"transformed\" score function of the behavior policy, with possibly more than one neural network. \n\nFor Diffusion-QL, as the guidance towards high-value actions has already been encoded in the score network, simply running reverse diffusion process with the learned score network could bring high-value actions, which makes it easier to implement and faster to inference. However, as the marginal probability $\\log \\pi_\\theta(a|s)$ for diffusion model is hard to compute, the policy improvement of such algorithms almost relies on surrogate objectives (See Eq. (3) in Diffusion-QL[2] and Eq. (10), Eq. (12) in EDP[1]). Consequently, the exact distribution of diffused action after inference is unknown, and the form of optimal policy is not guaranteed. \n\nOn the other hand, using guidance for inference allows for the decoupled and exact learning of both the behavior policy's score function and the guidance term. This provides guarantee for the action distribution after inference. Moreover, due to the decoupling between behavior policy's score function and guidance term, it's possible to combine different guidance flexibly during inference, without training the desired score function from scratch. This is especially useful for aligning large diffusion models in the future. However, given a limited amount of data, using guidance for inference may introduce auxiliary models, which increase the computational burdens.\n\n[1]: Efficient Planning with Latent Diffusion. Wenhao Li. ICLR, 2024.\n\n[2]: Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning. Zhendong Wang, Jonathan J Hunt, Mingyuan Zhou. ICLR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hQ2dKP4mQe", "reviewer": "Reviewer_uMN3", "summary": "The paper introduces a novel offline reinforcement learning approach that leverages diffusion models integrated with DICE-based methods. The proposed guide-then-select paradigm aims to minimize error exploitation. The resultant algorithm achieves state-of-the-art performance on D4RL benchmark tasks.", "questions": "Please see the above weaknesses.", "limitations": "Limitations are discussed in the conclusion.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Well-motivated and novel integration of diffusion models with DICE-based methods\n- Well-written theoretical justification for the approach\n- Strong empirical results that improve upon prior diffusion-policy baselines", "weaknesses": "- Missing comparison in Table 1 of more optimal Gaussian policy methods, e.g. EDAC [1]\n- Selection of D4RL datasets is limited, e.g. what about expert/random datasets? Further environments like Adroit would also be interesting\n- Discussion of hyperparameter choice in the Appendix is important and should be included in the experimental section.\n- A comparison of the inference speed of Diffusion-DICE and prior baselines would be valuable.\n\nMinor:\n- Line 70: “M” -> “M=”\n- Line 73: LP abbreviation not explained\n- There is concurrent related work [2] which also performs an analogous transformation between the behavior distribution to an online policy with diffusion models for synthetic data generation.\n\n[1] Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble. Gaon An, Seungyong Moon, Jang-Hyun Kim, Hyun Oh Song. NeurIPS, 2021.\n\n[2] Policy-Guided Diffusion. Matthew Thomas Jackson, Michael Tryfan Matthews, Cong Lu, Benjamin Ellis, Shimon Whiteson, Jakob Foerster. RL Conference, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's time and effort dedicated to evaluating our paper, as well as the constructive feedback provided. In response to the concerns and questions raised, we have prepared detailed answers, which are outlined separately below.\n\n>Missing comparison in Table 1 of more optimal Gaussian policy methods, e.g. EDAC [1]\n\nWe list the results of EDAC and Diffusion-DICE in the following table. For EDAC's results, we copy the results on MuJoCo locomotion tasks from its original paper and copy the results on AntMaze navigation tasks from CORL[1]. It's evident from the results that on MuJoCo locomotion tasks, Diffusion-DICE is comparable to EDAC. While on AntMaze navigation tasks, EDAC totally fails[3]. We suggest that this is because such ensemble-based uncertainty estimation heavily depends on the dataset distribution and on AntMaze datasets, it's not reliable.\n\n\n\n|  | Diffusion-DICE | EDAC |\n| -------- | -------- | -------- |\n|    halfcheetah-m      |     60.0     |    65.9      |\n|    hopper-m      |     100.2     |    101.6      |\n|    walker2d-m      |    89.3      |     92.5     |\n|    halfcheetah-m-r      |     49.2     |    61.3      |\n|    hopper-m-r     |    102.3      |      101.0    |\n|    walker2d-m-r      |    90.8      |     87.1     |\n|    halfcheetah-m-e      |    97.3      |    106.3      |\n|    hopper-m-e      |    112.2      |     110.7     |\n|    walker2d-m-e      |    114.1      |    114.7      |\n|    antmaze-u      |    98.1      |      0.0    |\n|    antmaze-u-d      |     82.0      |       0.0   |\n|    antmaze-m-p      |    91.3      |       0.0   |\n|    antmaze-m-d      |    85.7      |       0.0   |\n|    antmaze-l-p      |    68.6      |       0.0   |\n|    antmaze-l-d  |   72.0   |   0.0   |\n\n>Selection of D4RL datasets is limited, e.g. what about expert/random datasets? Further environments like Adroit would also be interesting\n\nFor the experiments of MuJoCo locomotion tasks, we only choose \"medium\", \"medium-replay\", \"medium-expert\" datasets because \"random\" dataset hardly exists in real-world tasks, and \"expert\" data is typically used for imitation learning settings. Furthermore, these datasets are also rarely used in other diffusion-based offline RL methods. To demonstrate Diffusion-DICE's superiority against other methods, we evaluate Diffusion-DICE on 2 tasks from kitchen and 2 tasks from adroit, following the same experimental setting in Appendix D. Note that we only choose 4 tasks in total due to the limited rebuttal preiod.\n\n|    | Diffusion-DICE | EDP | LD[2] | Diffusion-QL | QGPO|IQL|$f$-DVL|\n| -- | --- | - | --- | - | - |-|-|\n| kitchen-partial| **78.3** | 46.3 | -  | 60.5 | - | 46.3 | 70.0 |\n| kitchen-mixed |  **67.8**  | 56.5  | - | 62.6  | - |51.0| 53.8 |\n| pen-human | **84.4** | 72.7 | 79.0 | 72.8 | 73.9 | 71.5 | 67.1 |\n| pen-cloned  | **83.8** | 70.0| 60.7 | 57.3 | 54.2 | 37.3 | 38.1 |\n\n|      | $\\alpha$ | K    |\n| ---- | ----- | ---- |\n|   kitchen-partial   |    0.6      |   4   |\n|    kitchen-mixed  |   0.6       |   4   |\n|   pen-human   |    0.6      |   4   |\n| pen-cloned | 0.6     | 8 |\n\nThe results and the chosen hyperparameters are listed in the tables above. We compare Diffusion-DICE with other Offline RL baselines (either diffusion-based or not). The results are either taken from their original papers (if available) or from LD[2] (if not). The results on these more complex environments consistently show Diffusion-DICE's superiority.\n\n>Discussion of hyperparameter choice in the Appendix is important and should be included in the experimental section.\n\nWe apologize that due to the page limit, the discussion of hyperparameter choice is placed in the appendix. In the updated version, we'll include this discussion in the experimental section.\n\n>A comparison of the inference speed of Diffusion-DICE and prior baselines would be valuable.\n\nIn the following table, we compare the inference time (seconds/100 actions) of Diffusion-DICE and other baselines. It's worth noting that because we mainly focus on diffusion-based methods, these baselines also contain only diffusion-based algorithms. The results are based on a single RTX 4090 GPU, under antmaze-large-diverse-v2 environment.\n\n\n\n|       | Diffusion-DICE | SfBC |   QGPO  |  IDQL   |  Diffusion-QL   |  Diffuser   |\n| -- | --- | -- | -- | -- | -- | -- |\n| Inference Time (s/100 actions) |   16.86   |  20.12    |  15.50   |   6.32  |   1.65  |  98.64   |\n\n>Line 70: “M” -> “M=”\n\nWe'll fix this typo in the updated version.\n\n>Line 73: LP abbreviation not explained\n\nWe're sorry that due to the page limit, the explanation of LP abbreviation is omitted. In fact, we refer to LP as the expected return's linear programming form (linear with $d^\\pi$). By the definition of $d^\\pi(s, a)$ in Line 74, $d^\\pi(s, a)$ represents the discounted sum of probability that the agent takes action $a$ on state $s$ over all steps $t$. Then it's obvious that $E_{(s, a) \\sim d^\\pi} [r(s, a)]$ equals to the discounted sum of rewards., i.e. $E[\\sum_{t=0}^\\infty \\gamma^t \\cdot r(s_t, a_t) ]$. Due to the bijection between $\\pi$ and $d^\\pi$, maximizing expected return over $\\pi$ is equivalant to maximizing $E_{(s, a) \\sim d^\\pi} [r(s, a)]$ with respect to $d^\\pi$. The latter exactly possesses a linear programming form.\n\n>There is concurrent related work [2] which also performs an analogous transformation between the behavior distribution to an online policy with diffusion models for synthetic data generation.\n\nThanks for pointing out. We'll add this to the discussion in the updated version.\n\n[1]: CORL: Research-oriented Deep Offline Reinforcement Learning Library. Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, Sergey Kolesnikov. NeurIPS, 2024.\n\n[2]: Efficient Planning with Latent Diffusion. Wenhao Li. ICLR, 2024.\n\n[3]: Revisiting the Minimalist Approach to Offline\nReinforcement Learning. Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, Sergey Kolesnikov. NeurIPS, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gy2omdWK30", "reviewer": "Reviewer_UuBV", "summary": "The paper mainly addresses the problem of obtaining the optimal policies in the distribution correction estimation (DICE) setting, which is one popular offline RL approach. Note that the offline RL assumes that we can’t interact with an environment and only have access to a dataset—sets of (state, action, and next state) tuples collected by some behavior policy. For the given history, DICE estimates the optimal value functions $V^*$ and $Q^*$ and obtains the optimal stationary distribution ratio $w^*$, which is the rate between the optimal policy’s state occupancy distribution $d^*$ and the one of the behavior policy $d^D$. Even if it is possible to get the optimal value functions and the ratio via DICE, obtaining the corresponding optimal policy $\\pi^*$ from them is still challenging.\n\nIn this context, the paper proposes a novel training method for diffusion-based models to learn the optimal policy from the DICE-driven optimal value functions and the ratio. Unlike common density modeling, learning the diffusion-based models from the optimal value functions is not straightforward. Such a problem is closer to variational inference than density modeling; specifically, we don’t have samples from the target distribution to diffuse. Moreover, unlike a typical variational inference problem, sampling from the models is not available due to the nature of the offline RLs.\n\nTo do that the paper, the paper first introduces that the optimal policy can be represented by a product of the optimal stationary distribution ratio $w^*$ and the behavior policy $\\pi^D$. Since the product of two distributions is unnormalized, the complete expression includes the normalization constant, which is the expected value of $w^*$ under the behavior policy (Line 155).\n\nNext, the paper assumes that the behavior policy can be represented by a diffusion-based model with the forward process, which will also be used to perturb the policy of interest. Then, the authors show that the optimal policy at the perturbation level $t$ can be represented by using the behavior’s perturbed distribution at the same $t$ and the DICE ratio $w^*$ (Equation 6). In particular, this representation includes the logarithm of the expectation of the DICE ratio $w^*$ under the posterior distribution of the clean action $a_0$ for a given perturbed action $a_t$. While this novel representation doesn’t require sampling from the model policy anymore, computing the logarithm of the Monte Carlo estimate of any expectation is biased in general.\n\nTo circumvent this, the authors propose using the tangent transform, one of the variational approximation methods; thus, the quantity inside the logarithm is represented by an optimization problem, as in Equation 7. Interestingly, the new training objective only requires sampling from the behavior policy, which is suitable for the offline RLs. \n\nIn summary, the paper introduces a new representation of the diffusion-based optimal policy model by using the diffusion-based behavior policy and time-dependent learnable terms that will be learned by the convex problem described in Equation 7. The authors refer to this approach as In-sample Guidance Learning (IGL). While there have been a few previous approaches to learning diffusion-based optimal policy models, the authors point out that the IGL shows some benefits. For example, IGL only requires a single sample from the behavior policy, which can be obtained from the history, while some previous approaches require more than one sample from the behavior, which is not favorable in the offline RLs. \n\nIn addition, the authors suggest a few techniques to improve the stability of IGLs, such as using piecewise $f$-divergence for the DICE regularization term.\n\nFinally, the paper discusses some failure cases of previous approaches and how the proposed method bypasses them. It also demonstrates the efficacy of the proposed method via experiments on several benchmark datasets.\n\n\n-----------------------------------------------------\nUpdated the rating from 7 to 8 after the authors' rebuttal.", "questions": "N/A", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "In my understanding, the paper's contributions are clear, and I also consider that the results are essential for several reasons.\n\nThe paper introduces a novel representation of the diffusion-based optimal policy model and its training method, IGL. In addition, the paper also motivates the solution well. For example, this approach circumvents the drawbacks of previous approaches, which require more than one sample from the behavior policy, and such actions may be out-of-distribution of the environments.\n\nMoreover, the authors provide extensive discussions to help potential readers comprehend the characteristics of previous approaches and the proposed method.\n\nFinally, the paper demonstrates the effectiveness of the proposed method via various experiments, which further supports the authors' claim.", "weaknesses": "Overall, the paper presents a novel method for learning optimal policy in DICE, which is a valuable contribution to the field. However, improvements in the presentation would greatly enhance the clarity and comprehensibility of the manuscript.\n\nIn particular, several equations within the paper omit the definitions of variables, which can lead to confusion—for example, $a_0$ in Line 119. In addition, some variables overlap while they are independent. For instance, in Line 155, the variable of the integration overlaps with the $a$ in the numerator.\n\nI recommend revising the paper to address these issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful for the reviewer's detailed and accurate summary. We also appreciate the time and effort the reviewer has devoted. As for the weakness, we prepared the following responses, which are presented as follows.\n\n>... However, improvements in the presentation would greatly enhance the clarity and comprehensibility of the manuscript.\n\nWe apologize for any unclear expressions or improper organization in the article. We'll adjust our presentation in the updated version.\n\n>In particular, several equations within the paper omit the definitions of variables, which can lead to confusion—for example, $a_0$ in Line 119. In addition, some variables overlap while they are independent. For instance, in Line 155, the variable of the integration overlaps with the $a$ in the numerator.\n\nWe apologize for omitting some definitions of variables and will include them in the updated version. In Line 119, $a_0$ represents the diffused action where the footprint stands for diffusion timestep. We'll also address the overlap of independent variables in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers found the proposed approach novel and interesting. The method was found theoretically sound and the experimental results convincing.  Questions mainly concerned the clarification of the presentation of the mehtod, and the need for better positioning with respect to existing offline RL methods. 

The rebuttal focused on clarifying the contributions of this work with respect to related approaches, and provided aditional results on the kitchen and adroit environments, demonstrating the benefit of Diffusion-DICE as compared to the chosen panel of baselines. These additional clarifications were appreciated by the reviewers during the discussion phase, and a consensus was reached towards the acceptance of this work to NeurIPS 2024.
Decision: Accept (poster)