Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning
Authors: Ruoqi Zhang
Ziwei Luo
Jens Sjölund
Thomas B. Schön
Per Mattsson
Venue: NeurIPS 2024 poster (2024)
Keywords: Offline Reinforcement Learning; Diffusion Model; Stochastic Differential Equation
Primary area: reinforcement_learning

Abstract: Diffusion policy has shown a strong ability to express complex action distributions in offline reinforcement learning (RL). However, it suffers from overestimating Q-value functions on out-of-distribution (OOD) data points due to the offline dataset limitation. To address it, this paper proposes a novel entropy-regularized diffusion policy and takes into account the confidence of the Q-value prediction with Q-ensembles. At the core of our diffusion policy is a mean-reverting stochastic differential equation (SDE) that transfers the action distribution into a standard Gaussian form and then samples actions conditioned on the environment state with a corresponding reverse-time process. We show that the entropy of such a policy is tractable and that can be used to increase the exploration of OOD samples in offline RL training. Moreover, we propose using the lower confidence bound of Q-ensembles for pessimistic Q-value function estimation. The proposed approach demonstrates state-of-the-art performance across a range of tasks in the D4RL benchmarks, significantly improving upon existing diffusion-based policies. The code is available at https://github.com/ruoqizzz/entropy-offlineRL.

retropy-regularized Diffusion Policy with Q O-Ensembles for Offline Reinforcement Learning
Ruoqi Zhang" Ziwei Luo" Jens Sjölund Thomas Schön Per Mattsson
Department of Information Technology, Uppsala University fruoqi ziang, ziwei. jens. sjolund, thomas. thomas.schon, per mattsson)@it. se
Abstract
Diffusion policy has shown strong ability express complex action distributions in offline reinforcement learning (RL). However, it suffers from overestimating Q-value functions on out-of-distribution (OOD) data points due the offline dataset limitation. To address it, this paper proposes novel entropy-regularized diffusion policy and takes into account the confidence of the Q-value prediction with Q-ensembles. At the core of our diffusion policy is a mean-reverting stochastic differential equation (SDE) that transfers the action distribution into a standard Gaussian form and then samples actions conditioned on the environment state with a corresponding reverse-time process. We show that the entropy of such a policy is tractable and that can be used increase the exploration of OOD samples in offline RL training. Moreover, we propose using the lower confidence bound Q-ensembles for pessimistic Q-value function estimation. The proposed approach demonstrates state-of-the-art performance across range of tasks in the D4RL benchmarks, significantly improving upon existing diffusion-based policies. The code is available at
Introduction
Offline reinforcement learning (RL), also known batch RL [29] focuses learning optimal policies from a previously collected dataset without further active interactions with the environment 33.. Although offline RL offers promising avenue for deploying RI. in real-world settings where online exploration infeasible, a key challenge lies in deriving effective policies from fixed datasets, which usually are diversified and sub-optimal. The direct application standard policy improvement approaches is hindered by the distribution shift problem [10]. Previous works mainly address this issue by either regularizing the learned policy close the behavior policy 10 or by making
conservative updates for Q-networks [28 26
Diffusion models have rapidly become prominent class of highly expressive policies in offline RL 44 49 While this expressiveness is beneficial when modeling complex behaviors, it also means that the model has higher capacity overfit the noise specific idiosyncrasies in the training data. To address this, existing work introduce Q-learning guidance and regard the diffusion loss as a special regularizer adding to the policy improvement process 44
17 22 Such framework has achieved impressive results on offline RI. tasks. However, its performance is limited by pre-collected datasets (or behavior policies) and the learning suffers severe overestimation of Q-value functions on unseen state-action samples [31].
One promising approach is to increase exploration for out-of-distributte (OOD) actions, with the hope that the RL agent can more robust to diverse Q-values and estimation errors [50]. Previous online RL algorithms achieve this by maximizing the entropy of pre-defined tractable policies such
Corresponding authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

dataset
Training curves
2.20
Roward function Step samples
Single Crilic ersemb es Approx LCB
Step-2 samples
1.48
Roward
3.25
Q-value
2.5
3.20
62-
Diffusion lty
Maetin Diflacon Policy
60-
6.25
Ma Dffi-Po wi Ensembles
0.-
-875
1.00
-1.00-0755
0.5 0:38
action
Figure 1: A toy RL task in which the agent sequentially takes two steps (starting from 0) to seek a state with the highest reward. Left: The reward function is mixture of Gaussian, and the offline data distribution is unbalanced with most samples located in low-reward states. Center: Training different policies on this task with 5 random seeds for 500 epochs. We find that diffusion policy with entropy regularization and Qeenseblles yields the best results with low training variance. Right: Learned Q-value curve for the first step actions in state The approximation of the lower confidence bound (LCB) of Q-ensembles is also plotted.
Unfortunately, directly computing the log probability of diffusion policy is almost impossible since its generative process is a stochastic denoising sequence. Moreover, is worth noting that entropy is seldom used in offline settings because it may lead distributional shift issue which may cause overestimation of Q-values on unseen actions in the offline dataset.
Gaussians B5 13 15
Another line of work addresses the overestimation problem by enforcing the Q-values to be more pessimistic [28 20). Inspired by this, uncertainty-driven RI. algorithms employ an ensemble of Q-networks provide different Q-value predictions for the same state-action pairs [ [3 The variation these predictions serves as a measure of uncertainty. For state-action pairs exhibiting high predictive variance (e.g.. OOD data points), these methods preferentially adopt pessimistic Q-value estimations as policy guidance.
In this paper, we present an regropy-regullrrize diffusion policy with Q-ensembles for offline RL. At the core of our method a mean-reverting stochastic differential equation (SDE) [32] which allows us to sample actions from standard Gaussian conditioned on the environment state. We show that such an SDE provides a tractable entropy regularization that can be added in training to increase the exploration of OOD data points. In addition, we approximate the lower confidence bound (LCB) of Q-ensembles to alleviate potential distributional shifts, thereby learning a pessimistic policy to handle high uncertainty scenarios from offline datasets. As illustrated in Figure I both entropy regularization and Q-ensembles can improve RL performance on unbalanced offline datasets. The LCB approach further reduces the variance between different trials and provides better estimation of unseen state-action pairs.
Our model achieves highly competitive performance across range of offline D4RL benchmark tasks 8 and, in particular, significantly outperforms other diffusion-based approaches the Antmaze environment. The superior performance demonstrates the effectiveness of the entropy-regullrrizatoo and Q-ensembles. Overall, the proposed method encourages policy diversity and cautious decision- making, enhancing exploration while grounding the policy in the confidence of its value estimates derived from the offline dataset.
Background
This section reviews the core concepts of offline RL and then introduces the mean-reverting SDEs and shows how we sample actions from its reverse-time process. Note that there are two types of timesteps for RL and SDE. To clarify that, we use i E NN to denote the RL trajectories' step and E {0, (O..... .......TT T) to index diffusion discrete times.
Offfine RL. We consider learning a Markov decision process (MDP) defined as M (S,A, P. R, 7, do), where S and are the state and action spaces, respectively. The state tran- sition probability is denoted P(5 s; a.) and R S A + represents reward function,

---

E (0, 1] is the discount factor, and dg is the initial state distribution. The goal of RL is to maximize the cumulative discounted reward Σi= ? [r,s, a.]] with a learned policy æ. In contrast online RL which requires continuous interactions with the environment, offline RL directly learns the policy from the static dataset D {S.; as, rı, 8i+1 11) N In the offline setting, two primary challenges are frequently encountered: over-conservatism and limited capacity to effectively utilize diversified datasets [3.]. To address the issue of limited capacity, diffusion models have recently been employed to learn complex behavior policies from datasets [44].
Mean--Reertine SDE. Assume that we have random variable a" sampled from unknown distribution (a). The mean-reverting SDE [3] a diffusion process (a" hejo.T that gradually injects noise to a.:
da - di o dw, a ~~~ aP>pn(a);
(1)
where w is the standard Wiener process, 0g and o, are predefined positive parameters that characterize the speed of mean reversion and the stochastic volatility, respectively. Compared to IR-SDE [32], we set the mean to to let the process drift to pure noise to the RL environment. The mean can however be tuned to high-reward actions in the offline dataset or prior knowledge. By setting
of = 20, for all diffusion steps, the solution to the forward SDE (T < () is given by
p(a' aa)= =a a Drn - 20, -20.. )f),
(2)
where 0r == S 0g dz are known coefficients [32]. In the limit + oo, the marginal distribution Pe(a) = p(a' aa) converges to a standard Gaussian No I). This gives the forward process its informative name, i.e "mean-revertting" Then, Anderson [ states that can generate new samples from Gaussian noises by reversing the SDE (1 as da
0 of Va Po)) dt + o2 deo
where a. ~ NO 1) and u is the reverse-time Wiener process. This reverse-time SDE provides a strong ability fit complex distributions, such as the policy distribution represented in the dataset D. Moreover, the ground truth score Va log Pr(a) is acquirable in training. We can thus combine it with the reparameterization trick
a' = a de c V1 e-29. c2
29.
(4)
to train dime-dependent neural network Es using noise matching randomly sampled timesteps:
Lar(d) = [ [||s a)]|]
where E ~ (O0 I) is a Gaussian noise and {a M== denotes the discretization of the diffusion process. See Appendix A. for more details about the solution, reverse process, and loss function.
Sample Actions with SDE. Most existing RL algorithms employ unimodal Gaussian policies with leameed mean and variance. However, this approach encounters a challenge when applied to offline datasets, which are typically collected by mixture of policies and therefore hard to represent by simple Gaussian model. Thus we prefer to represent the policy with an expressive model such as the reverse-time SDE. More specifically, the forward SDE provides theoretical guidance to train the neural network, and the reverse-time SDE (3) generates actions from Gaussian noise conditioned on the current environment state as a typical score-based generative process
4
3 Method
We present the three core components our method: 1) an efficient sampling strategy based on the mean-reverting SDD; 2) an entropy regularization term that enhances action space exploration; and 3) a pessimistic evaluation with Q-ensembles that avoids overestimation of unseen actions.
3.1 Optimal Sampling with Mean- Rean-Reverting SDE
We have shown how to sample actions with reverse-tima SDEs in Section However, generating data from the standard mean-reverting SDE 32 requires many diffusion steps and sensitive to the noise scheduler [36]. To improve sample efficiency, we propose generating actions from the posterior distribution p(a t-1 a²) conditioned on a!. This approach ensures fast convergence the generative process while preserving its stochasticity.
3

---

Proposition 3.1. Given an initial variable a", for any diffussio state a' at time t E [1, the posterior of the gean-reevrtiin SDE (I) conditioned on a is
p(a' a' a - NN
i jula", a)), 3.1),
(6)
which is a Gaussian with mean and variance given by:
As(a", a²) = 1 1
c---26 2-
l a' + 1 e-c-20 8. a and 3, = (1 c 28, :)
c
20,
20,
1 20,
1 -e-20
where 02 = - 0,tt and 0g substitute fo:t for clear notation.
The proof is provided in Appendix A.2 Moreover, thanks the reparameterization trick 25 we can approximate the variable ao by reformulating Eq. 4 to
à = e (a' v1 e e ea 1)),
where ce is the learned noise prediction network. Then we combine Eq. (8) with Eq. 6 to iter-
atively construct the sampling process. In addition, it can be proved that the distribution mean (aa"", a²) is the optimal reverse path from a' to a (see Appendix (A.3). Here, we illustrate simple example of data reconstruction with different diffusion steps in Figure 2 Additional results are shown are provided in Ap- pendix B.I] The proposed optimal sampling process from Proposition requires only steps,
Data Distribution
Gaussian Neise
Reconstruction (N=S)
Reconstructi on (N-10)
30)
Reconstruction
30 for the standard process. clearly shows that the proposed optimal sam- pling is more efficient than the stan- dard reverse-time SDE process.
Reversedimee Optimal Sameling
Figure/ 2: Comparison of the reverse-time SDE and optimal
sampling process in data reconstruction.
Notation Note Recall that we have two distinct types of timesteps for RL and SDE denoted by and t, respectively. To clarify the notation, in the following sections, we use a to represent the intermediate variable of an action taken RL trajectory step with SDE timestep t, as at a' at state s,. Therefore, the action take for state is the final sampled action denoted by a!. Hence, the policy is given by
T 8;) pa(a")
While we cannot sample directly from this distribution we can efficiently sample the SDE's reverse joint distribution as
Po(a 0.T ) II (a a'),
(10)
where p(a") = (0, I) is Gaussian noise and the generative process conditioned on the environ- ment state 8;. So take action from (a st), we sample from the joint distribution using Eq. (6) and Eq. (8) and finally pick out ag as our sampled action. The visualization of is out method provided in Appendix A.5
3.2 Diffusion Policy with Entropy Regularization
The simplest strategy of learning a diffusion policy is to inject Q-value function guidance to the noise matching loss (5), in the hope that the reverse-time SDE would learn sample actions with higher values. This can be easily achieved by minimizing the following objective:
(Ø) = LLan()) Eg, D.a.
[Q (s,, a.)]
(11)
where Q2 is the state-action value function approximated by neural network, Section 3.3

---

This combination regards diffusion loss as cehavvoor-cloning term that learns the overall action distribution from offline datasets. However, the training is limited to existing data samples and the Q-learning term is sensitive to unseen actions. To address it, we propose to add an additional entropy log T 8;)] to increase the exploration of the action space during training and
term H = E >D [-
rewrite the policy loss to
(Ø) (0) - Daf ( a,) log (a [.)]
(12)
where is hyperparameter that determines the relative importance of the entropy term versus Q-values, and À = 17 /e..)) a]]] to normalize the scale of the Q-values and balance loss terms. Iteratively generating the action a. though a reverse diffusion process is computationally costly but, with an estimated noise c, from diffusion term (5), we can thus directly use it to approximate a based on Eq. for more efficient training.
Entropy Approximation. It is worth noting that the log probability of the policy (o ss)) in general intractable in the diffusion process. However, we found that the log probability of the joint distribution in Eq. (10) is tractable when conditioned on the sampled action af. Proposition 3.1 further shows that the conditional posterior from a: to at is Gaussian, meaning that
- log Te (a s;) = 1 log To (a: ss)) +C,
(13)
where C is constant and at can be approximated using Eq. 2 similar a.. The proof is provided
in Appendix A.4 Then can focus the conditional reverse marginal distribution ps(a) a s.) that determines the exploration actions and is acquirable via Bayes' rule:
Pa (a) af x:) Po
a 8:) po(a; ..)
(14)
(a a. 8:)
Since all terms in Eq. (4) can be computed with Eq. (),
rewrite the policy objective
(+(0)= = Lan(d) - D... (Q. a?) a log(p(á)
a!", s)))).
(15)
where a? and à are approximate values calculated based on samples from the diffusion term. Note that the temperature a usually plays an important role in the maximum entropy RI. framework and we thus provide detailed analysis in Section 4.3
3.3 Pessimistic Evaluation via Q-ensembles
Entropy regularization encourages diffusion policies to explore the action space, reducing the risk of overfitting pre-collected data. However, in offline RL, since the agent cannot collect new data during training, this exploration can lead to inaccuracies in value estimation for unseen state-action pairs I.. Instead of staying close the behavior policy and being overly conservative, considering the uncertainty in the value function is an alternative approach.
In this work, we consider a pessimistic variant of a value-based method to manage the uncertainty and risks, i.e., the lower confidence bounds (LCB) with Q-ensembles. More specifically, we use an ensemble of Q-functions with independent targets to obtain an accurate LCB of Q-values. Each Q-function is updated based on its own Bellman target without sharing targets among ensemble members [11, as follows:
Ja(w") = Es Qu (s;, a,) - ynth (ri, 81+1, To) ym = r; + + F.a Q 3m (8++1, 1;+1)]
(16)
where "h" Thth are the parameters of the Q network and Q-target network for the mth Q-function. Then, the pessimistic LCB values are derived by subtracting the standard deviation from the mean of the Q-value ensemble,
Q LCB
Eens 1Q (8, a)) - B Venn
(8, a)]
(17)
where 3 2 0 is hyperparameter determining the amount of pessimism, V(Q is the variance of the ensembles, and m E {1,...,M M) where M the number of ensembles. Then, Q is used in the policy improvement step to balance entropy regularization and ensure robust performance. Finally, use QLCB as the Qu to (15). We summarize our method in Algorithm |
5

---

Experiment
In this section, we evaluate our methods on standard D4RL offline benchmark tasks 8 and provide a detailed analysis of entropy regularization, Q-ensembles, and training stability.
4.1 Setup
Datasets We evaluate our approach on four D4RL benchmark domains: Gym, AntMaze, Adroit, and Kitchen. In Gym, we examine three robots (halfcheetah, hopper, walker2d) across sub-optimal (medium), near-optimal (medium-expert), and diverse (medium-replay) datasets. The AntMaze domain challenges quadrupedal ant robot to navigate mazes of varying complexi- ties. The Adroit domain focuses on high- dimensional robotic hand manipulation, us- ing datasets from human demonstrations and robot-imitated human actions. Lastly, the Kitchen domain explores different tasks within a simulated kitchen. These domains collectively provide a comprehensive frame- work for assessing RL algorithms across di- verse scenarios.
Algorithm 1 Diffusion Policy with Q-Ensembles
Initialize parameters for TΦ. Tg. {Q Q3m J.M m==1" for each iteration do
Sample mini-batch {(8;, a, T2, 8(+1)} from D. Ensemble-Q learning
Generate 0,+1 Tg(0 8.) with and
Update Q-networks {Qum JMM by 16
Diffusion policy learning
Sample {at he(o,r] from a, with (4)
Predict noise and approximate a. with (8)
Update policy Th by (L5) using QLCB from
# Update target networks
à + 770 + (1 1))
3/th +- m76 + (1 th E (I,...,,M M)
end for
Implementation Details Following Diffusion-QL [44, we keep the network structure the same for all tasks with three MLP layers (hidden size 256, Mish activation (34),, and train models for 2000 epochs for Gym and 1000 epochs for others. Each epoch consists of 1000 training steps with a batch size of 256. We use Adam (24) to optimize both SDE and the Qensembless. Each model is evaluated by 10 trajectories for Gym tasks and 100 trajectories for others. In addition, our model is trained an A 100 GPU with 40GB memory for about 8 hours per task, and results are averaged over five random seeds.
Hyperparameters We keep key hyperparameters consistent: Q-ensemble size 64, LCB coefficient 4.0. The entropy temperature and Kitchen tasks. The SDE sampling step is set to T = 5 for Gym and Antmaze tasks, T = 10 for Adroid and Kitchen tasks, For 'medium' and 'large' datasets of AntMaze, we use max Q-backup following Wang et al. (44) and Kumar et al. (28). We also introduce the maximum likelihood loss for SDE training as proposed by Luo al. [32 More details are Appendix B.2
0.01 for Gym and AntMaze tasks and automated for Adroit
4.2 Comparison with other Methods
We compare our method with extensive baselines for each domain to provide a thorough evaluation and to understand the contributions of different components in our approach. The most fundamental among these are the behavior cloning (BC) method, BCQ [10 and BEAR (27 which restrict the policy dataset behavior, highlighting the need for policy regularization and exploration. We also assess against Diffusion-QQ. by Q-values. This comparison isolates the benefits of our enhanced sampling process and Q-ensemble integration. Our comparison includes CQL (28) and IQL (26), known for conservative Q-value updates. Additionally, we consider EDP (2 a variant of IQL with an efficient diffusion policy, and IDQL [17], which combines IQL as a critic with behavior cloning diffusion policy reweighted by learned Q-values. These comparisons evaluate the effectiveness of integrating diffusion policies with conservative value estimation. Finally, we include MSG [1I] which combines independent Q-ensembles with CQL, and DT 4, treating offline RL as sequence-to-sequeece translation problem. These baselines help assess the robustness and generalizability of our method across different approaches. The performance comparison between baselines and ours is reported in Table (Gym, Adroit and Kitchen) and Table 2 (AntMaze). Detailed results are discussed below.
which integrates a diffusion model for policy regularization guided

---

Table 1: Average normalized scores on D4RL benchmark tasks. Results BC, CQL, IQL, and IQL+EDP are taken directly from Kang et al. [2I] and all other results are taken from their original papers. Our results are reported by averaging 5 random seeds.
Gym Tasks
BC
DT
CQL.
IQI.
IQQL-A
IQL+EDP
Diff-QI. QI.
Ours
madfcheedaa-mmciiu--222
42.6
42.6
44.0
47.4
51.0
48.1
51.1
54.9
Hopper-mediimm-22
52.9
67.6
58.5
66.3
65.4
63.1
90.5
94.2
Walker2d walker2d-medium->2
75.3
74.0
72.5
78.3
82.5
85.4
87.0
92.5
Halfcheetah--ne wwweeeaa-mmii---eppppeee -eplaayy-rre
36.6
36.6
45.5
44.2
45.9
43.8
$7.0
sroperrmcciuun-rcplaay--2
18.1
82.7
95.0
94.7
92.1
99.1
101.3
102.7
Walker2d meee-d--mmiiimleeeeeeae
26.0
66.6
77.2
73.9
85.1
84.0
95.5
94.2
Halfcheetah-meedeeee mediim-cxxpert-22 expert-v2
35.2
86.8
91.6
86.7
95.9
86.7
96.8
90.3
52.5
107.6
105.4
91.5
108.6
99.6
111.1
111.9
107.5
108.1
108.8
109.6
112.7
109.0
110.1
111.2
Average
51.9
74.7
77.0
82.1
89.9
Admit Tasks
BC
BCQ
BEAR
CQI.
IQ1.
IQL+EDP
Diff-QI.
Pen-human-vl
63.9
68.9
-1.0
37,5
71.5
72.7
72.8
70.0
Pen-cloned-vl
37.0
44.0
26.5
39.2
37.3
70.0
57.3
68.4
Average
50.5
56.5
12.8
38.4
54.4
71.4
65.1
69.2
Kitchen Tasks
BC
BCQ
BEAR
CQL
IQL
IQL+EDP
Diff-QL
Ours
Sichhn--ompllee--00
65.0
8.1
0.0
43.8
62.5
75.5
92.7
kitchen-partial
38.0
18.9
13.1
49.8
46.3
46.3
60.5
66.3
kitchen-mixed-v00
51.5
8.1
47.2
51
51
56.5
62.6
68.0
Average
51.5
11.7
20.1
48.2
53.3
59.4
69.0
75.7
Table Average normalized scores on D4RL AntMaze tasks. Results of BC, DT, CQL, IQL, and IQL+EDP are taken directly from Kang et al. [21]. and all other results are taken from their original papers. Our results are reported by averaging 5 random seeds.
AntMaze Tasks
BC
DT
CQL
IQL
MSG
IDQL-A
IQL+EDP
Diff -L-
Ours
Antmaazeummzz--O0
54.6
59.2
74
87.5
97.8
94.0
87.5
93.4
100
Antmaze- uaaizzeee-azz-eeeee diverss--iieee
45.6
53.0
84.0
62.2
81.8
80.2
62.2
66.2
79.8
Antmaaz--mede mediren--ed------e-e play-v00
61.2
71.2
71.2
Antinaze miviun-d
0.0
0.0
53.7
70.0
88.6
84.8
70.0
78.6
91.6
Antmaze parge-payy
0.0
0.0
15.8
39.6
72.6
63.5
39.6
46.4
81.2
Antmaze diver-iivers--O
0.0
0.0
14.9
47.5
71.4
67.9
47.6
56.6
76.4
Average
16.7
18.7
50.6
63.0
83.6
79.1
63.0
86.7
Gym tasks Most approaches perform well on Gym 'medium- exxpert' and 'medium-replay" tasks with high-quality data but drop severely on 'medium' tasks with suboptimal trajectories. Diffusion-
QL 44 achieves a better performance through highly expressive diffusion policy. Our method further improves performance across all three 'medium' tasks. The results illustrate the efficacy of combining diffusion policy with entropy regularization and Q-ensembles in preventing overfitting to suboptimal behaviors. By maintaining policy stochasticity, our algorithm encourages the exploration of action spaces, potentially discovering better strategies than those in the dataset.
Adroit and Kitchen Most offline approaches cannot achieve expert performance on these tasks due the narrowness of human demonstrations in Adroit and the indirect, multitask data in Kitchen [44). Our method outperforms all other approaches in the Kitchen tasks which suggests its ability to addition,
"stitching" the dataset and generalization. same as other tasks for a robust setting. Even our method still achieves competitive performanc in Adroit tasks. This fixed a leads the agent to continuously explore the action space throughout the entire training process, even when encountering unseen states. While exploration is generally advantageous, it can be detrimental in environments with limited data variability. Additionally, unlike in antmaze tasks, random actions are more likely to negatively impact performance in tasks where precise control essential like Adroit. Moreover, it's worth noting that slightly tuning a leads to SOTA performance, illustrated in Table 3
fix the entropy coefficient to be the

---

AntMaze AntMaze tasks are more challenging, requiring point-to toanttto-oiitt navigation with sparse rewards from sub-optimal trajectories 8 As shown in Table traditional behavior cloning methods (BC and DT) get ( rewards on AntMaze medium and large environments. Our method shows excellent performance on all the tasks in AntMaze even with large complex maze settings and outperforms other methods by margin. The result is not surprising because the entropy regularization incentivizes the policy to explore various sub-optimal trajectories within the dataset and stitch them to find a path toward the goal. In tasks with sparse rewards, this can be crucial because it prevents premature convergence to suboptimal deterministic policies. Additionally, employing the LCB of Q-ensembles effectively reduces the risk of taking low-value actions, enabling the development of robust policies.
In general, employing consistent hyperparameters for each domain, along with fixed entropy tem- perature a, LCB coefficient 3, and ensemble size M across all tasks, our method not only achieves substantial overall performance but also outperforms prior works in the challenging AntMaze tasks. By the comparison with MSG[[]] (Q-ensemble alone) and Diffusion-QL. C method further improves results demonstrating its effectiveness in handling complex environments with sparse rewards by effectively combining suboptimal trajectories to find better solutions via action space exploration.
(Diffusion alone), our
4.3 Analysis and Discussion
We first study the core components of our method: entropy regularization and Q-ensemble. Then we show that adding both significantly improves the training robustness of diffusion-based policies.
Entropy Regularization The core idea of applying entropy regularization in offline RL is to increase the exploration of new actions such that the estimation of Q-ffnctions more accurate, especially for datasets with unbal- anced action distribution such as the toy ex- ample in Figure TT. Here we report the results of training the diffusion policy with different entropy temperatures in Table 3 It is observed that our method with positive entropy coeffi- cients performs better than that without the entropy term. In addition, we can extend our model with automatic entropy adjustment
Table 3: Ablation study on entropy temperatures.
Entmoy temperature
0
0.01
0.05
0.1
Antmaze-mediun-plays
85.7
91.4
91
88.3
92.0
semauzz-mddiumddiveerr
89.0
91.6
90.7
93.5
90.8
Antmaze large play-vQ v0
77.7
81.2
78.3
82
82.0
Antmaze-large largen--rrrg-diieee-----
73.7
76.4
71.3
78.3
76.0
HallChestah Medium
53.7
54.9
54.0
55.3
Hopper- Sepper-Meiuu--222 Walker2D- -MMeeiu---2
94.8
94.2
93.3
97.7
94.0
89.6
92.5
91.6
89.9
91.9
human-v1
60.9
67.2
63.6
69.8
78.5
Pen conn-lloee-vv
57.9
66.3
61.8
56.5
79.8
Kitchen complete-v0
80.6
82.3
77.6
54.4
84.4
Kitchen-Mixed-x0)
57.0
60.2
50.8
56.5
60.4
similar to the work in [16]. This approach is marked as "auto" in Table 3 The results show that auto-tuning the entropy temperature further improves the performance in the Adroit and Kitchen domains. Please refer to Appendix B.3 for more details.
Q-Ensembles We evaluate our method un- der different numbers of Q networks M E {2, 4, 64] in the AntMaze environment to ex- plore the effectiveness of Q-ensembles. The results with average performance within 5 dif- ferent seeds are provided in Table The key ob- servations are 1) As the M increases. the model gets better performance and the training process becomes more stable; 2) The standard deviation in the results decreases M increases, suggest- ing larger ensembles not only perform better on average but also provide more reliable and consistent results. 3) While increasing M from 2 to shows a substantial improvement, the performance gains decrease with an even larger size. It is worth noting that other offline RL approaches like Diffusion-QL. (4] also adopt two networks for training robustness. See Appendix B for more detailed results.
Table 4: Ablation experiments of our entropy- based diffusion policy with different ensemble sizes on selected AntMaze tasks.
Ensemble Size
4
64
Antmaze- play-v0
84.0
87.2
91.4
nummare-medium--iivess---d
71.8
87.2
91.6
Antmaze-large-play-ve
54.2
52.4
81.2
aatmare-lrgee-iversee-w
43.2
69.0
76.4
Average
63.3
74.0
85.2
LCB coefficients We evaluate our method with 8 values of 2, and on AntMaze-medium environments Figure demonstrates that adjusting the LCB coefficient improves performance, partic-
8

---

antmaze--eeduum-laayoo
antmaaz-neddummddieeee--w
2
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Normalized Score
Normalized Score
Figure 3: Ablation experiments of our method with different values of LCB coefficient ,3 1, on AntMaze-Medium environments over different random seeds.
Table 5:
Computational time comparison with different settings on Autmaze-meddium-play--dd Training time is for 1 epoch (1000 training steps) and eval time is for 1000 RL steps.
Policy
Diffusion Step T
# Critics M
Training Time
Eval Time
Gaussian
2
5m 35s
Is 450ms
Gaussian
64
7m 20s
Is 450ms
Diffusion
5
2
9m 30s
4s 800ms
Diffusion
5
64
11m
4s 800ms
Diffusion
10
2
12m 23s
8s
Diffusion
10
64
13m 55s
8s
ularly for higher values, which helps in managing the exploration-explotaiio trade-off effectively. addition, the numerical results are provided in Appendix Table S
Training Stability and Computational Time Empirically we observe that the training of diffusion policies always unstable, particularly for sparse-reward environments such AntMaze medium and large tasks. Our method alleviates this problem by incorporating the entropy regularization and Q-ensembles stated in the introduction. Here, we further show the comparison of training Diffusion-QL. and our method on four AntMaze tasks as illustrated in Figure 4 maintaining the same number of diffusion steps 7 = for both. It is observed that the performance of Diffusion-QI. even drops down as the training step increases, while our method is substantially more stable and achieves higher results throughout all the training processes. We also included detailed comparison of training and evaluation times for Gaussian and diffusion policies with Q-ensembles in Table [ Increasing M from 2 to 64 almost does influence the evaluation time. The diffusion step T has more impact on both training and evaluation time which is a common problem in diffusion models.
Related Work
antmaze-mediuu-playyyo
antmaaze-eedium-ddiveee-w
Generative Diffusion Models and Mean- reverting SDEs Recent advancements
Normalized Score
Normalized Score
have integrated diffusion models
[8]
42
40) and SDEs [43 32
45
39 for real-
istic generative modeling. The develop- ment of Denoising Diffusion Probabilis- tic Models [8] showcases the ability of diffusion models generate high-fidelity images through iterative reverse diffusion processes guided by deep neural networks, achieving state-of-the-an performance in generative tasks. In
antmaze-large-plaayvo
antrmaze-arge-diversevvo
Normalized Score
Normalized Score
6.4
45 39) mean-
reverting SDEs are applied to speech pro- cessing and image restoration tasks. These SDEs, similar to (T) but with different pa- rameters, ensure our policy adapts across various distributions without bias. The gen- eral applicability of our method is demonstrated D4RL. benchmark domains. The comparison between our SDE and 43 provided in Appendix A.6
Training Progress (#50k steps) Training Progress 50k steps)
Figure Learning curves of the Diffusion Q and our method on selected Antmaze tasks over 5 random seeds.

---

Diffusion Models in Offline RL Diffusion models in offline RL have gained growing attention for their potent modeling capabilities. In Janner al. [9]. diffusion models are introduced as trajectory planners trained with offline datasets for guided sampling, significantly mitigating compounding errors in model-based planning [6). Diffusion models are also used data synthesizers [5] 48], generating augmented training data to enhance offline RL. robustness. Additionally, diffusion models approximate behavior policies [44] [7], integrating Q-learning for policy improvement, though this can lead to overly conservative policies.
online RI., maximum entropy strategies encourage exploration by maximizing rewards while maintaining high entropy [14 [5]. This approach develops diverse skills 7 and adapts to unseen goals (38]. However, its application offline RL is challenging due to the multi-modal nature of datasets from various policies and expert demonstrations.
Entropy Regularization
Uncertainty Measurement Balancing exploration and exploitation is crucial when data is limited. Online RL methods like bootstrapped DQN [37 and Thompson sampling [30] estimate uncertainty for exploration guidance. offline RL, handling uncertainty is critical due to the lack environment interaction. Model-based methods like MOPO [47] and MORel (23 measure and penalize uncertain
model dynamics. Similarly, model-free methods like EDAC [ and MSG [11] use Q-network ensembles to obtain pessimistic value estimations for policy guidance.
6 Conclusion
In this work, we present an entropy-reee -errupp-regulriized diffusion policy for offline RL, introducing mean- reverting SDEs as the base framework to provide tractable entropy. Our theoretical contributions include deriving an approximated entropy for a diffusion model, enabling its integration as an entropy regularization component within the policy loss function. We also propose an optimal sampling process, ensuring the fast convergence of action generation from diffusion policy. Additionally, we enhance our method by incorporating Qeensembles to handle the data uncertainty. Our experimental results show that combining entropy regularization with the LCB approach leads to a more robust policy, achieving state-of-the-a- performance across offline RL benchmarks, particularly AntMaze tasks with sparse rewards and suboptimal trajectories.
Future Work While the proposed method performs well on most D4RL tasks, the diffusion policy requires longer time when executed on compute- and power-constrained devices. Our future work will investigate real-tin time policy distillation under time and compute constraints to address this challenge.
Acknowledgements
This research was financially supported Kjell och Märla Beijer Foundation and by the project Deep probabilistic regression new models and learning algorithms (contract number: 2021-04301) as well as contract number 2023-04546, funded by the Swedish Research Council. The work was also partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. The computations were enabled by the supercomputing resource Berzelius provided by National Supercomputer Centre at Linköping University and the Knut and Alice Wallenberg foundation.
10

---

References
[1] Gaon An, Seungyong Moon, Jang- Mlyn Kim, and Hyun Oh Song. Uncertainty lunertaantt-bbase based offline reinforcement lcarning with diversified q-ensemble. Advances neural information processing systems, 34:7436-7447, 2021.
[2] Brian DO Anderson. Reverse- time diffusion equation models. Stochastic Processes and their Applications. 12(3):313- 326, 1982.
[3] Chenjia Bai, Lingxiao Wang, Zhuoran Yang. Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. arXiv preprint MiXiv:2202.11566, 2022.
[4] Lili Chen, Kevin La, Aravind Rajeswaran, Kimin Lec, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances neural information processing systems. $4:15084-15097, 2021.
[5] Zoey Chen, Sho Kiami, Abhishek Gupta, and Vikash Kumar. Genaug: Retargeting behaviors to unseen situations via generative augmentation. arXiv preprint arXiv:2302.06671, 2023.
[6] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-877 8794, 2021.
[7] Benjamin Eysenbach, Abhishek Cupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv: 1802.06070, 2018.
[8] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.022// 07211, 2020.
[9] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
[10] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without explo- ration. International conference on machine learning, pages 2052-2062. PMLR, 2019.
[11] Kamyar Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. Advances Neural Information Processing Systems, 35: 15:18877-1828. 2022.
[12] Daniel T Gillespie. Exact numerical simulation of the omstein-uhlenbedd process and its integral. Physical review E, 54(2):2084, 1996.
[13] Tuomas Haarnoja, Haoran Tang, Picter Abbeel, and Sergey Levine. Reinforcement learning with deep casrgy-based policies. In International conference on machine learning, pages 1361-1361. PMLR, 2017.
[14] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[15] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[16] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, el al. Soft actor-critic algorithms and applications. arXiv preprint arv: 1 812.05905, 2018.
[17] Philippe Hansen- -sttuchh,, Ilya Kostrikov, Michael Janner, Jakuh Grudzien Kuha, and Sergey Levine. Idgl: Implicit g-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.1057 2023.
[18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851 2020.
[19] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flcxible behavior synthesis. arXiv preprint vrXXvv2205.09991, 2022.
[20] Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning, pages 5084-5096 PMLR, 2021.

---

[21] Bingyi Kang. Xiao Ma. Chao Du, Tianyu Pang. and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. arXiv preprint urXiv:2305.2008. 2023.
[22] Bingyi Kang. Xiao Ma. Chao Du, Tianyu Pang. and Shuicheng Yan. Efficient diftusion policies for offline reinforcement learning. arXiv preprint arXiv:2305.20w:
2023.
[23] Rahul Kidambi, Aravind Rajeswaran, Pranecth Netrapalli, and Thorsten Joachims. Morel: Modcl-bascd offline reinforcement learning. Advances in neural information processing systems, 33:21810-21888 2020.
[24] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.668 2014.
[25] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[26] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint
2021.
[27] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping emor reduction. Advances in Neural Information Processing Syxtems. 32, 2019.
[28] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 1111119999111,,1 2020.
[29] Sascha Lange, Thomas Gabel, and Martin Riedmiller, Batch reinforcement learning. In Reinforcement learning: Sfate-of-the pages 45-73. Springer, 2012.
[30] Tor Lattimore and Csaba Szepesvári. Bandit algorithums. Cambridge University Press, 2020.
[31] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint Si0i:::000000645: 2020.
[32] Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Image restoration with mean-reverting stochastic differential equations. International Conference on Machine Learning, 2023.
[33] Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Refusion: Enabling large-size realistic image restoration with latent-space diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2023.
[34] Diganta Misra. Mish: A self regularized non-monotonic activation function. arXiv preprint
(9Xiv:9908.08000 2019.
[35] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pages 1928 1937. PMLR, 2016.
[36] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMIR, 2021.
[37] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. Advances in neural information processing systems, 29, 2016.
[38] Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22( 13:5:3345-1399 2009.
[39] Julius Richter, Simon Welker, Maan-Marie Lemercier, Bunlong Lay, and Timo Gerkmann. Speech enhance- ment and dereverberation with diffusion uuffusiio-bbasd generative models. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
[40] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 10684-10695. 2022.
[41] Chitwan Saharia, William Chan, Saurabh Saxena. Lala Li, Jay Whang. Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to image difffusion models with deep language understanding, Advances in neural information processing systems, 0..36679-34994 2022.
12

---

[42] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, arXiv preprint arXiv:2010.02502. 2020.
[43] Yang Song, Jascha Sohl-Dickstein. Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. 2020.
[44] Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies expressive policy class offline reinforcement learning. arXiv preprint arXiv:2208, 06193, 2022.
[45] Simon Welker, Julius Richter. and Timo Gerkmann. Speech enhancement with score-based generative models in complex STFT domain. Proc. Interspeech 2022, pages 2928-2932, 2022. doi: 10.21437/ Interspeech.2222 10653.
[46] Chenjun Xiao, Yifan Chen Ma. Dale Schuurmans, and Martin Müller. Learning to combat conpounding-erve error in model-based reinforcement leamiing arXiv preprint arXiv:191211126,, 2019.
[47] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33: 33:14129-1444
2020.
[48] Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Jodilyn Peralta, Brian Ichter, et al. Scaling robot learning with semantically imagined experience. arXiv preprint 2.3vv:230 /1500, 2023.
[49] Zhengbang Zhu, Hanye Zhao. Haoran He, Yichao Zhong, Shenyu Zhang, Yong Yu, and Weinan Zhang Diffusion models for reinforcement learning: A survey, arXiv preprint arXiv:2311.01223, 2023.
[50] Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.
13

---

Appendix
A
Proof
A.1 Solution to the Forward SDE
Given the forward Stochastic Differential Equation (SDE) represented by
da -8,a dt + oz der, a po(a),
(18)
where 02 and o, are time-dependent positive functions, and w denotes a standard Wiener process. We consider the special case where o² = 20, for all t. The solution for the transition probability from time T to (T < () is given by
pla'la") N -N(a'' (a'la"e Brs (1 (1 -e-e" 20.., {1)
(19)
Proof. The proof is in general similar to that in IR-SDE 32 To solve Equation 18 we introduce the transformation
(baat) ac
(20)
and apply Itô's formula obtain
da(a)))) ore deo(1).
(21)
Integrating from r to I, we get
v ((a [ 0 dwl(),
(22)
we can analytically compute the two integrals as Og and or are scalars and then obtain
a(Ie - a=e - --eaa= o2e dw((z.
(23)
Rearranging terms and dividing by cº, we obtain
a (t) trie [: o dz(
(24)
The integral term actually a Gaussian random variable with mean zero and variance
ofe- 28.. dz = x² - 20...))
(25)
under the condition of = 20,. Thus, the transition probability is
20.:.))) I)) p(a aa)) = Naa'la' e (1 e1-e-20
(26)
This completes the proof.
Loss function From 19
the marginal distribution of p(a(t)) can be written as p(a(t)) p(a(t) a(0))
(27)
= (a(((()) (0)) 0 e
where we substitute dor with 0, for clear notation. During training, the initial diffusion state a is given and thus we can obtain the ground truth score Va log (a) based on the marginal distribution:
a -
Va log pr (a an)
(28)
1-
which can be approximated using neural network and optimized with score-matching loss. Morcover, the marginal distribution (27) gives the reparameterization of the state:
a' ale b + V1 e- 20. Ex.
(29)
14

---

where E, is standard Gaussian noise ~ (0, I). By substituting 29 into 28 can be re-written in terms of the noise
the score function
C₁
Va log 6o)
(30)
vi 11---2.
Then we follow the practical settings in diffusion models [8 estimate the noise with time- dependent neural network Es and optimize it with a simplified noise matching loss:
( = 20 (a 8: + -1-e 26, E,, -
(31)
where é is randomly sampled timestep and {= denotes the discretization of the diffusion process. And this loss (31 the same as in the main paper.
A.2 Sampling from the Posterior
Proposition 3.1. Given an initial variable a", for any diffusion state a' at time t E [1,T], the posterior of the mean-reverting SDE (1) conditioned on a is
p(a' a' .a) N'aa' I a)), 3,1), which is Gaussian with mean and variance given by:
(32)
(a a²) = - e 20,-1
e-2 28,
0r a
+
e
02-1 a
201
20,
(33)
(1 20.-11 [[1
28%
and
e- 20,
where 0, = 1i-1 O,dl and 0, is to substitute Oo.t. for clear notation.
Proof.
The posterior of SDE can be derived from Bayes' rule,
p(a a., a²) p(a" a'-1, a. p(a' aa)) p(a' a²)
(34)
Recall that the transition distribution p(a" a'~1) and p(a' aa) can be known with the solution to the forward SDE. Since all the distributions are Gaussian, the posterior will also be Gaussian.
p(a'
aa la'aa) a²)
1
(a' a-- e0-12-
(a'-1 ale-01-17
(a' aee-0e92)
2
- 20
+
28__1
1 1 e-2 261
20'
2e - 6
2e 8
exp
2
e
e-
20
e-
1a
1-2²
20
a'
1- 20,
a )a)aaa +Ola', a²)
(35)
where C(a', a²) is some function not involving (a* the mean and the variance can be computed:
122 With the standard Gaussian density function,
(a' ad)
e-
8 a'
1 e
28'
8 a
20,
26,
c
(36)
(1 e
20
[1 e 20'
and
3,
26, 22,
Thus complete the proof.
15

---

A.3 Optimal Reverse Path
In addition, we can prove the distribution mean Pe(a', a²) is the optimal reverse path from a' to
a
Proof. As stated in Proposition 3.1, the posterior is Gaussian distribution and can be derived by Bayes' rule. Thus it is natural to find the optimal reverse path by minimizing the negative oogllikelhood according to
arg min log p(a a-= a ad)
(37)
From 34
we have
-
log loggp(a a", aa) x - logp(a' aa"1,a0) log 1ogp(a"'1 a²)
(38)
Then we can directly solve 37 by computing the gradient of the negative log- log-likelihood and setting
V ao "d'd""aaaaa) -Va - a
aa))
-V lo-=logp(a,"" aa")
% (a'a' a e %)
a ale 0-1
2-
1 e e-20 201-1
a
e 20,
f-1
a'e-0,
0--1 age -
+
c 20
1 p
28-1111
2-2
20
a 1-1 (1 c c2d:)
a'c-op
0 age 0
0.
(1 c 20 20,) (1 c 0-20--1
202
20
(39)
Since 39 is linear, we get
a - x
0-2
- -aaa"
+
1 - e-2-
0r a
(40)
1 20,
1 e 2-2
20,
This completes the proof. Note that the second-order derivative is a positive constant, and thus a is the optimal point. And we find that this optimal reverse path is the same as our posterior distribution mean shown in Proposition]3 3.1
A.4 Entropy Approximation
Proposition
3.1 further shows that the conditional posterior from at to af is Gaussian, meaning that
loo (a s;) log (a s;) + C,
(41)
where C is constant and a: can be approximated using
p(a' a²) = NNaaa |a
Rra a (2.a-)
(42)
Proof. Let's consider sequentially sampled action states a,, af from our optimal sampling strategy. Then we have
π (af]ss)) = Taallat, a.) (a)(ss))
(43)
where a is the approximated action's initial state from Eq. (8). Proposition 3.1 shows that the conditional posterior from at to at is a Gaussian with certain mean and variance, meaning that the term T7 (aa [a]]o], aQ) is computable constant and thus we can write
log (a.]s:)
log क (a (a. ag) log (aa [ss) = log ( (s.) + C.
(44)
16

---

A.5 Visualization
For more intuitive explanation of our approach, Figure outlines the forward and reverse processes of the mean-reverting SDE used for action prediction.
Forward: Mean-reverting SDE
Data
Noise
( 1a
a
a²
a'
a
p(a
|a')
Reverse: optimal sampling for action prediction
Figure 5: Visualization of the workings of the mean-reverting SDE for action prediction. The SDE models the degradation process from the action from the dataset to noise. By guiding the policy with corresponding reverse-time SDE and the LCB of Q, a new action is generated conditioned on RL state.
A.6 Comparison VP SDE
Our mean-reverting SDE is derived from the well-known Orastein-Uhlenbect (OU) process 12 which has the following form:
dæ = 0(p r)dt + adw.
As 00, its marginal distribution (x) converges to a stationary Gaussian with mean value ja which explains the name: "mean-reverting". We assume that there is no prior knowledge of the actions and thus set μ mean-reverting SDE has the same form as VP SDE. However, in [43]. no solution of the continuous- time SDE was given. The authors start from perturbing data with multiple noise scales and generalize this idea with an infinite number of noise scales which makes the perturbed data distributions evolve according to SDE. They keep using the solution of DDPM while we use Itô's formula to solve the continuous- time SDE. Compared to the original VP SDE, our mean-reverting SDE is analytically tractable, see and thus its score V, log Pr (z) is easier learn. More importantly, the solution of the mean-reverting SDE can be used for entropy approximation.
generate actions from standard Gaussian noise. Then, with μe the
B Additional Experiments Details
B.1 More experiments on proposed optimal sampling with different sample step
We added the figures of data generation with fewer steps (T' = 1 and T = 2) for the toy task in Section 3.1. The results show that the optimal sampling strategy significantly outperforms the reverse-time SDE in all steps, further demonstrating the efficiency and effectiveness of our method.
B.2 Hyperparameters
As stated in Section 4.1 keep our key hyperparameters, entropy weight 0.01, ensemble size M = 64, LCB coefficient 3 and diffusion steps T for tasks in different domains. As for others related to algorithm, and whether to use max Q backup. For implementation details, we consider the gradient clip norm, diffusion loss type, and whether to clip action at every diffusion step. We keep the hyperparameter same for tasks in the same domain except for the AntMaze domain. We use max Q backup (88 for complete tasks. The hyperparameter settings are shown in Table
consider the policy learning rate, Q-learning weight ?,
B.3 Automating Entropy Adjustment
possible to consider automating entropy adjustment similar to 15 but a depends on the state since the offline dataset is pre-collected and may be imbalanced across different states shown in
17

---

Data Distribution
Gaussian Noise
Reconstruction (T-1)
Reconstruction (T-2)
Reconstruction
Reconstruction (T-10)
Reconstruction (T- 30)
Reconstruction (T=50)
Reverse time SDE Optimal Sampling
Figure 6: The proposed optimal sampling with different sample steps.
Table 6: Hyperparameter settings of all selected tasks. means all the AntMaze tasks use max Q-backup trick (28) except the "antmaze-umaze-vo" task the same that in other papers. The 'likelihood' loss is proposed in IR-SDE 32 which forces the model to learn optimal reverse paths from a' a
Tasks domain
learning rate
max Q-backup
gradient norm
loss type
action clip
T
Gym
3e-4
1.0
False
4.0
Likelihood
False
AntMaze
3c-4
2.0
True*
4.0
Noise
True
5
Adroit
36-5
0.1
False
8.0
Noisc
True
10
Kitchen
3e-4
0.005
False
10.0
Likelihood
False
10
Figure
One way compute the gradients for state-depend is
Jaa E,,,D,a,saal (-a(s; log To (agls))) (Z))
(45)
where (<s;) is implemented as a neural network with single hidden layer consisting of units and H is desired minimum expected entropy which represents the desired level of exploration which can set as function action space dimension without the need for extensive hyperparameter tuning across different tasks.
B.4 More Analysis for Q-Ensembles
Here, provide more detailed experiments for analyzing the effect of ensemble sizes M as discussed in Section 4.3 More specifically, the results of different ensemble sizes are reported in and
Table and Figure 8 which we also provide the variance that further shows the robustness of our method.
Table 7: Ablation study ensemble size M on selected AntMaze tasks.
Ensemble Size
M
M
M
M 16
M 64
antmaze- metiizeeeediummmaeee plapiiuummplly-Oo
50.2 : 26.4
84.0 = 5.0
87.2111
83.6 7.7
91.4 + 1.5
antmaze-medium--dvveeee-ww
67.2 + 7.6
71.8 14.1
87.2+3.8
88.0 2.2
91.6 2.3
untmazc-large-play-xd
48.2 + 10.8
54.2 10.0
52.4 - 13.0
71.8 + 5.8
81,2 3.0
ummaaz--lnge-ddiereee-dd
58.8 + 11.4
43.2 + 16.1
69.0 + 8.3
76.4 + 8.47
76.4 2.1
Average
56.1
63.3
74.0
80
85.2
18

---

pen-human-vi
per- cloneleed
Return (Quailty)
kitchen-complete--0
kitcher--artin partial-O
Kitchen-mized-va
Return (Quality)
0ntmeazemmedimmmplay--0
antnaae-medium-diieee--00
antmaze-large-llay-x
antmaze-large-diivrsewww
Retur (Qualtt)
3
Figure 7: A I-SNE visualization of randomly selected 1000 states from Antmaze, Adroit and Kitchen domain. The color coding represents the return of the trajectory associated with each state.
artmazs-mediummpplag. play.v0
antmaze- www.em-iidun----eeee.....
antmaae--are play-u0
antmazz-large diverssggeedvveeevv0
M-1
M-4
6.2-
lormalted Store
M-54
6d-
Nomelized Score & E.
Nomaltedd
6.3-
622
Training Progress steps)
Thaining Progress steps)
Thaining Progress steps)
Triiing Progress
Figure 8: Abaltion study of Q-ensemble size M on selected AntMaze tasks. We consider M E {1, 4, 64]and found size 64 is the best overall tasks.
B.5 Ablation study on LCB coefficients |3
To explore the impact of different LCB coefficients 3. We add an experiment our method with 3 values of 1, 2, and on AntMaze-medium environments. the LCB coefficient improves performance, particularly for higher values, which helps managing the exploration-exploittionn trade-off effectively. In addition, the numerical results are provided in Table g
that adjusting
B.6 Ablation study on Diffusion step T
We evaluated the impact of varying the number of diffusion steps range of tasks, including AntMaze, Gym, and Kitchen in Table Our findings indicate that while increasing the number of steps generally improves performance, five steps provide the best balance across different tasks and
19

---

Table 8: Ablation study of LCB coefficients B.
ICB Coefficient /
1
2
4
Antmaze-medt medium-play--e play-v0
82.4 = 4.9
88.6 = 1.5
91.612.3
Autmaze-modium--iivree---
74.6 + 3.7
84.0 - 7.8
91.4 11.5
Average
78.5
86.3
91.5
between performance and computational time in Gym and Antmaze tasks. For more complex tasks as in Kitchen and Pen, we choose T = 10.
Table 9: Ablation study of diffusion step T.
Diffusion Step 7
3
10
Halicheetah-mccuuu--plaaa---
43.4
57.0
49.5
Hopper-mediuum--eppay-222
39.4
102.7
101.7
Walker2d-meedium-repaaayy-
51.2
94.2
98.1
Antmaze-mmedium-pllay-wW
96.6
91.6
90.2
Autmazz-medduum-ddeers--00
95.8
91.4
83.8
Autmaze-large-play-v00
67.6
81.2
63.2
Antmaze-largee-divers--00
81.0
76.4
70.0
pen-human-vl
65.4
67.2
70.0
pen-cloned-vl
67.3
66.3
68.4
Kitchen-copplete-00
7.5
82.3
92.7
Kitchen-partial-v0
10.9
60.3
66.3
Kitchen-mixed-v0
4.8
60.2
68.0
Average
52.6
77.6
76.8
B.7 Ablation study on Max Q-back
We conducted experiments with and without max Q-backup on AntMaze tasks in Table I0 The inclusion of max Q-backup significantly enhances performance, particularly in more complex envi- ronments (e.g., Antmaze-large).
Table 10: Ablation study of "Max Q trick".
Max Q-backup
True
False
Antmaze-mediim-plaa--dd
91.6+2.3
89.2 = 2.9
Antmaze- mulmmzze-maiuum-diieee-----
91.4+1.55
87.6 1.8
Antmaze-large-play-vd
81.2 +3.0
22.3 7.1
Autmazzelarge-diverse---
76.4 12.1
26.5 6.1
B.8 Offline vs Online Model Selection
We use the online experience to evaluation our model during training. Table presents comparison of our method with Diffusion-QU including both online and offline results. Additionally, we include method's performance based on offline selection using the BC Loss criterion, selecting the step where the difference between consecutive steps was less than 4e-3.
20

---

Table 11: Performance comparison with online model selection and offline model selection.
AntMaze Tasks
Diffusion- QL (Offline)
DIffusion- QL (Online)
Ours (Offline)
Ours (Online)
untmaze-umuze-vl
93.4
96,0
99.0
100.0
mntnaaze-umaze-divee diverse-vO
66.2
84,0
67.5
79.8
antmaze-meddun--lee play-v0
77.6
79.8
84.0
91.4
siimazzemediimm-iieeee---
78.6
82.0
85.4
91.6
untmaze-lange-play-x00
46.6
49.0
72.6
81.2
untmazelange--divvrseww
56.6
61.7
65.9
76.4
Average
69.6
75.4
79.2
86.7
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly describe the introduction of mean- reverting SDEs for entropy-regularized diffusion policies, the integration of Q-ensembles for robust policy improvement, and the empirical validation through performance on D4RL benchmarks, which thoroughly discussed and validated the subsequent sections of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section
6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (c.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
21

---

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
See Appendix for more details.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The code is provided in supplementary materials and additional experiments Details can be found in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
22

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should descrihe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
The code with README file is provided.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//iips.cc/publl wttpcccppppcccceeeeee..ee..ccltt htt.c//ppcc.cguudeeeeeeeeeeeee/e guwdes/ http///oodubissiiniil. sosoooooccsss for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experiment setting and main hyperparameters are shown in Section 4 and additional experiments Details can be found in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
23

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our results reported averaging by 5 random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computer resources are shown in Section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The dataset used in the paper is public without privacy-related data.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
25

---

We use the dataset D4RL for training offline RL agents and cite the original paper in Section
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pappessiithteeee thcode condsatsssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
26

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
27


Reviews:
[{"review_id": "pNUiA3cJGF", "reviewer": "Reviewer_jdmN", "summary": "The paper proposes that adding entropy regularization to offline RL is beneficial, and using pessimistic Q-value estimation through ensemble methods can provide a better estimate of the Q-value. Figure 1 explicitly shows the benefit of the ensemble Q method. The methods show impressive performance in D4RL.", "questions": "1. A typo in Eq(5). There is an unnecessary right parenthesis.\n\n2. I found Mean-Reverting SDE is a mathematical rewrite of VP SDE, see [42]. VP SDE has form \n\\begin{align*}\n    dx=-\\frac{1}{2}\\beta(t)xdt+\\sqrt{\\beta(t)}dw\n\\end{align*}\nMean-Reverting SDE has form \n\\begin{align*}\n    dx=-\\theta_txdt+\\sqrt{2\\theta_t}dw\n\\end{align*}\n\n3. These two SDEs are the same in my mind. Then why is it called Mean-Reverting SDE instead of VP SDE in [42], and why is [42] not cited in this context? Can the authors point out any differences between them?\n\n4. Optimal Sampling seems to use the same trick as DDIM[41], using $p(x_{t-1}|x_t,x_0)$ for reverse sampling instead of $p(x_{t-1}|x_t)$ as in DDPM. If that's the case, I believe the authors should point out and properly cite DDIM in the relevant context.\n\n5. It is amazing to see that in Figure 2, optimal sampling can have such great performance when $N=5$. Can the authors also provide its performance in this toy task when $N=1$ and $N=2$? I am curious about their performance with fewer steps.\n\n6. It is hard for me to prove Eq(14). Can the authors elaborate on more details about its proof?\n\n7. Can the authors elaborate more on how Eq(15) is derived from Eq(12) and Eq(13)? I guess the last term in Eq(15) should be $\\log p(\\hat a_i^1|s_i)$, not $\\log p(\\hat a_i^1|a_i^T,s_i)$. (There is a typo in Eq(15). It should be $a_i^T$ not $a_t^T$).\n\n8. The paper proposes using automatic entropy tuning as in SAC in line 216. In line 551, the authors mention that the entropy target is the dimension of the action space. I guess it should be the negative dimension of the action space (as SAC does)? For the alpha loss in Eq (45), $\\alpha$ depends on $s_i$, while SAC doesn't. Why is the alpha loss designed this way? Is there any difference in making $\\alpha$ depend on $s_i$?\n\n9. I found your method's performance in Antmaze-medium-play-v0, Antmaze-medium-diverse-v0, Antmaze-large-play-v0, and Antmaze-large-diverse-v0 in Table 1 to be inconsistent with the training curve in Figure 3. For example, the score of Antmaze-medium-diverse-v0 in the table is 91.6, while the training curve in Figure 2 shows the training curve score is around 40. Why do they differ so much? The same phenomenon occurs in the other three Antmaze environments. Can the authors explain the mismatch? Can the authors also provide training curves for other environments?\n\n10. In Table 1, how is the score recorded? Is it the final round score, online model selection, offline model selection, or moving average?\n\n11. In Eq (34) and (35), should $t$, $t-1$, and $0$ be subscript or superscript? Why is $t-1$ subscript and $t$ superscript?\n\n12. Can the authors elaborate on why Eq(43) holds?\n\n13. In Table 5, the loss type is Likelihood or Noise. What are their mathematical forms?\n\n\n[41] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.\n\n[42] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.", "limitations": "The paper has some gaps in the mathematical proofs. Additionally, more details of the experiments need to be added. I am more than willing to increase my score if my questions are well addressed by the authors.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-organized and easy to follow. I found the pessimistic ensemble Q trick, which increases estimation accuracy (Figure 1), to be interesting. It is intriguing to see that main entropy to the agent can also benefit offline RL in some tasks.", "weaknesses": "I have some trouble overcoming some gaps in the mathematical proofs and found that some mathematical equations may not hold.\n\nThe empirical performance in Table 1 is not consistent with the training curve in Figure 3.\n\nMore detailed questions about these two aspects are in the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and insightful feedback. Your comments have been invaluable in guiding our efforts to refine and clarify our work. Below, we address your main concerns in detail:\n\n1. *Mean-reverting SDE and VP SDE.*\n\n   Our mean-reverting SDE is derived from the famous Ornstein-Uhlenbeck (OU) process [1] which has the following form:\n   $$\n   \\mathrm{d} x = \\theta (\\mu - x) \\mathrm{d} t + \\sigma \\mathrm{d}w.\n   $$\n   As $t \\to \\infty$, its marginal distribution $p_t(x)$ converges to a stationary Gaussian towards the mean value $\\mu$, which gives the informative name: “mean-reverting”. We assume there is no prior knowledge of the actions and thus set $\\mu = 0$ to generate actions from standard Gaussian noise. Then, with  $\\mu = 0$, the mean-reverting SDE has the same form as VP SDE. However, in [42], no solution of the continuous time SDE was given.  The authors start from pertubing data with multiple noise scales and generalize this idea with an infinite number of noise scales which makes the perturbed data distributions evolve according to an SDE.  They keep using the solution of DDPM while we use Itô's formula to solve the continuous SDE. Compared to the original VP SDE, our mean-reverting SDE is analytically tractable (Eq.(2)) and thus its score $\\nabla_{x} \\log p_t(x)$ is easier to learn. More importantly, the solution of the mean-reverting SDE can be used for entropy approximation. We will clarify these points and cite [42] appropriately in the revised manuscript.\n\n2. *Optimal Sampling and DDIM.*\n\n   Yes, our optimal sampling uses the same conditional posterior distribution$p(x_{t-1} | x_t, x_0)$ as DDIM for reverse sampling and we will cite it in Section 3.1 in the revised manuscript.\n\n3. *Performance of Optimal Sampling with $N=1$ and $N=2$.*\n\n   We provide new samples generated with fewer steps of the toy task in **General Response Figure 1**.\n\n4. *Proof for Eq.(14).*\n\n   Eq.(14) is a multiple conditions Bayes' rule given by\n   $$\n       p_\\phi(a_i^1 \\mid a_i^T, a_i^0, s_i) = \\frac{ p_\\phi(a_i^T \\mid a_i^1, s_i) \\; p_\\phi(a_i^1 \\mid a_i^0, s_i)}{p_\\phi(a_i^T \\mid a_i^0, s_i)}.\n   $$\n   Here we use $p_\\phi(a_i^1 \\mid a_i^T, s_i)$ instead of $p_\\phi(a_i^1 \\mid a_i^T, a_i^0, s_i)$ since both $a_i^0$ and $a_i^1$ are computed from the diffusion sampling process and the conditional posterior from $a_i^1$ and $a_i^0$ is a certain Gaussian (with known mean and variance, see Appendix A.4). We will clarify it in the revised manuscript to make the notation and proof clearer.\n\n5. *Clarification on Eq.(15).*\n\n   Substituting Eq.(13) to Eq.(12) gives the new policy objective and its last term should be $\\log (p(\\hat{a}_i^1 \\mid s_i))$. However, we assume that the terminal state $a_i^T$ must be a standard Gaussian and thus we add the condition $a_i^T$ as $\\log (p(\\hat{a}_i^1 \\mid a_i^T, s_i))$ which can be obtained from Eq.(14) and the results won't be affected. (And thank you for pointing out the typo.)\n\n6. *Entropy target should be the negative dimension of the action space.*\n\n   Yes, our code uses the negative dimension of the action space as the entropy target, the same as SAC. We will correct it in the paper.\n\n7. *The $\\alpha$ depends on $s_i$ in Eq.(45) while SAC doesn't. Why is the alpha loss designed this way?*\n\n   SAC is an online algorithm that can interact with the environment and collect new samples with various state actions. However, the offline dataset is pre-collected and may be *imbalanced* across different states. Thus our idea for alpha loss is to assign different entropy values to each state based on available data.\n\n8. *Inconsistent between Table 1 and Figure 3.*\n\n   We made a mistake in Figure 3, Antmaze-medium-diverse-v0, where we used the wrong data source for the curve, resulting in a major inconsistency. The updated figure is provided in **General Response Figure 2**. The reason for other minor inconsistencies is that we report the mean of the *best* results in tables while showing the average values of each step in figures. Moreover, the error bar is a default value in *seaborn.lineplot* which takes 95% confidence.\n\n9. *Training curves for other environments?*\n\n   We show some training curves in **General Response Figure 3** and will also add other curves to the appendix in the revised manuscript.\n\n10. *How is the score recorded?*\n\n    Consider that the behavior-cloning (BC) loss is not suitable for exploring OOD samples thus we choose to use online evaluation to select the model. It is worth noting that Diffusion-QL has proven the online-selected model has a similar performance as the offline-selected model. This will be clarified in the revised manuscript.\n\n11. *Subscript or superscript \"$t$\" in Eq.(34) and Eq.(35).*\n\n    All actions should use superscript sample steps (i.e., $t$, $t-1$, and $0$). We have now fixed them in the manuscript.\n\n12. *Why Eq.(43) holds?*\n\n    Eq.(43) holds when action states $a_i^1$ and $a_i^0$ are sequentially sampled from a diffusion process. The term $\\pi(a_i^0 \\mid a_i^1, \\hat{a}_i^0)$ is the conditional posterior sampling from Proposition 3.1. We will elaborate more details in the revised manuscript.\n\n13. *Likelihood loss and Noise loss in Table 5.*\n\n    The \"Noise\" loss is similar to the simplified loss in DDPM as shown in Eq.(5). In contrast, the \"Likelihood\" loss is proposed in IR-SDE [2], which forces the model to learn optimal reverse paths from $x_t$ to $x_{t-1}$. Generally, the Likelihood loss is more stable while the Noise loss tends to generate more stochastic samples. This will be clarified in the revised manuscript.\n\n14. *Typos*\n\n    Thank you for pointing out these typos and we will fix them in the revised manuscript.\n\n*[1] Exact numerical simulation of the Ornstein-Uhlenbeck process and its integral. Physical review E, 1996.*\n\n*[2] Image restoration with mean-reverting stochastic differential equations. ICML, 2023.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jSu9JcVimA", "reviewer": "Reviewer_zXRk", "summary": "The paper proposes to use reverse-time SDE as the policy in an actor-critic algorithm. To make it work, entropy regularization is added, for which an entropy approximation scheme is suggested. Furthermore, to improve stability, an ensemble of Q-networks is employed, and the pessimistic lower-confidence bound (LCB) is taken as the value, i.e., $\\mathbb{E}[Q] - \\beta \\sqrt{\\mathbb{V}[Q]}$.\n\nEvaluations on D4RL show improved performance compared to baselines.", "questions": "In the abstract, the authors write \"we show that the entropy of such a policy is tractable\". This sounds like it is analytically tractable. But in fact an approximation is used (Sec. 3.2). I would suggest to reformulate the statement in the abstract to avoid such misunderstanding.", "limitations": "Computational limitation is briefly mentioned in the conclusion. It would be nice to have a more extended discussion of limitations and providing some numbers on the timings.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality: good. The paper proposes a few novel ideas to make diffusion policy work in offline RL.\nQuality: good. Overall presentation and evaluation are good. Tested only on D4RL, but this is common in offline RL.\nClarity: excellent.\nSignificance: excellent. Potentially a new baseline on D4RL, especially AntMaze.", "weaknesses": "1) Influence of multi-modality of the policy not sufficiently explored. Is this in the end what makes it better compared to using a Gaussian? Or is it rather the LCB and Q-ensemble? How would SAC with Q-ensemble and LCB-value perform (i.e., not using the diffusion policy)?\n2) No comparison to other methods that use multi-modal policies, e.g., Gaussian mixture model policy ([Wasserstein Gradient Flows for Optimizing Gaussian Mixture Policies](https://proceedings.neurips.cc/paper_files/paper/2023/hash/429b5216a4d08850c586fbf809e17877-Abstract-Conference.html)) or energy-based policies ([Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow](https://arxiv.org/abs/2405.13629))", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and for recognizing the novelty and potential impact of our work. We appreciate your positive feedback and are happy to provide our point-to-point response below:\n\n1. *Influence of multi-modality of the policy...SAC with Q-ensemble and LCB*\n\n   In offline RL, the pre-collected datasets are always imbalanced and lead to a complex multi-modal policy rather than a Gaussian. We provided a comparison in Table 2 of the paper, which shows that our model outperforms MSG (Gaussian policy with Q-ensembles) on all AntMaze tasks, meaning that the multi-modal policy is important and improves the Gaussian in offline RL. Also, the ablation study in Table 4 of the paper shows that increasing the ensemble size further improves the performance. Moreover, unfortunately, we can't directly compare our method with SAC on offline RL tasks since SAC and its variants are all online algorithms that need to interact with the environment.\n\n2. *Comparison to Other Multi-Modal Policies*\n\n   Please allow us to clarify that we have compared the behavior cloning (BC), decision transformer (DT), and Diffusion-QL, that are all non-Gaussian multi-modal policies. We appreciate the reviewer's suggestion and will be happy to implement the mentioned Gaussian mixture and energy-based policies in our future work.\n\n3. *Reformulation of the Abstract*\n\n   Thank you for pointing out the unclear formulation in the abstract. We will revise the abstract to clarify that the entropy of the policy can be approximated in a tractable way.\n\n4. *Computational Limitations and Timings*\n\n   We acknowledge that the computational time was only briefly mentioned. We add experiments to compare the training time and evaluation time between different policy types, diffusion steps, and the number of critics. The results are shown in **General Response Table 4**. These results indicate that while the diffusion policy requires longer training times compared to the Gaussian policy, the increase is manageable and justified by the significant performance gains. The evaluation times, however, remain relatively comparable, suggesting that the practical deployment of the trained models is feasible.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mterv34iB6", "reviewer": "Reviewer_Jzch", "summary": "The paper presents Entropy-Regularized Diffusion Policy with Q-Ensembles for offline reinforcement learning. This method addresses Q-value overestimation on out-of-distribution (OOD) data by using a mean-reverting stochastic differential equation (SDE) to transform action distributions into a Gaussian form, combined with entropy regularization for enhanced exploration and Q-ensembles for pessimistic Q-value estimation. The approach achieves state-of-the-art performance across D4RL benchmarks especially on AntMaze.", "questions": "1. Could you please give a more thorough explanation on why the choice of hyperparameters, especially the entropy temperature plays a critical role in the performance. As we know that the adroit and kitchen tasks are more complex than Gym and AntMaze, I suspect on the explanation about narrowness of human demonstrations.\n\n2. What are the potential challenges or limitations in extending this method to real-world offline RL tasks outside the D4RL benchmarks?\n\n3. Could the authors provide more intuitive explanations or visualizations to illustrate the workings of the mean-reverting SDE and its impact on action distribution sampling?", "limitations": "The authors have figured out the limitation of the proposed method on high computational cost and large inference time.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper introduces a method that combines entropy regularization with Q-ensembles within the framework of diffusion policies for offline RL, offering a novel solution to tackle Q-value overestimation and limited exploration of OOD samples. The use of a mean-reverting SDE to model action distributions is theoretically robust and aligns well with the goal of transforming actions into a tractable Gaussian form for efficient sampling. The authors provide detailed theoretical contributions, including the tractability of entropy regularization in diffusion policies and the benefits of using Q-ensembles for robust value estimation.", "weaknesses": "1. Computational Cost: High computational resources are required, including extensive training epochs on high-capacity GPUs, which limits accessibility and applicability in real-world scenarios with constrained resources.\n\n2. Hyperparameter Sensitivity: Although some experiments with different hyperparameters are provided, a more thorough exploration of hyperparameters like entropy temperature and Q-ensemble size is needed to further validate the method's robustness.\n\n3. Generality: The evaluation is robust but focused on D4RL benchmarks. More experiments on diverse or real-world offline RL tasks would strengthen the claims about the method's generalizability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and for recognizing the strengths of our work. We appreciate your positive feedback on the novelty and theoretical robustness of our proposed method.\n\n1. *Computational cost.*\n\n   We acknowledge that diffusion policies require longer training and inference time due to the multi-step sampling process. However, the increased time is manageable and justified by the significant performance gains. A detailed comparison is provided in **General Response Table 4**. Moreover, techniques like the SDE/ODE solver and diffusion distillation have been proven useful for sample accelerating. We will regard it as our future work and try to apply the faster version to some real-world scenarios.\n   \n\n2. *Hyperparameter sensitivity...exploration of hyperparameters like entropy temperature and Q-ensemble size is needed.*\n\n   Please let us clarify that the ablation studies on entropy temperature and Q-ensemble size are already provided in Table 3 and Table 4 of the paper. Moreover, we further add two ablation experiments on LCB coefficient $\\beta$ and diffusion steps $T$ in **General Response Table 1 and Table 3**.\n\n3. *Challenges and limitations in real-world applications.*\n\n   As discussed in the Conclusion section, the main challenges and limitations of our work are the action sample time and the computation cost. And the network is also too simple (only 3 dense layers) for complex real-world applications. Our future work will investigate real-time policy distillation under time and compute constraints and explore other efficient network architectures to address real-world datasets and scenarios.\n\n4. *Could the authors provide more intuitive explanations or visualizations to illustrate the workings of the mean-reverting SDE and its impact on action distribution sampling.*\n\n   Intuitively, the mean-reverting SDE has a forward process (action to noise) and a reverse process (noise to action). The forward process is only used in training to provide an analytically tractable solution for score network learning. In inference, the reverse process (based on the trained score network) is used as the policy to generate actions conditioned on RL states. A rough visualization of the workings of the mean-reverting SDE is provided in **General Response Figure 4**. This will also be clarified in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YIhl4uSIVj", "reviewer": "Reviewer_T2t7", "summary": "This work introduces a new (fast) SDE-based sampling technique to derive actions from a diffusion based policy.", "questions": "- it's a bit hard to see variance of green line in Fig 1, Perhaps extracting it / using IQM plots with error bars would be helpful. (I know it is a toy example, but it does a good job illustrating the utility)\n- may just be me, but I believe using $t$ index in RL terms is more standard than $i$, similarly for $T$ vs. $L$. Perhaps consider swapping this notation?\n- on a similar notation front, the use of $a$ seems like a poor choice since starting at L76 it is very general (not yet discussing actions). Maybe switch to $x$?\n- I'm curious why different gradient norms were required; do you have any insight here? Was there parameter/grad divergence in some cases?\n- Missing reference to SUNRISE? (I know it is UCB instead of LCB, but perhaps still useful to comment on the relationship): https://arxiv.org/pdf/2007.04938\n- In algorithm 1, can you please include Q^LCB_psi rather than Q_psi alone (if it is correct)? Otherwise it is confusing where the previously discussed value estimate comes in.\n- Can you explain a bit more about the sense of \"optimality\" regarding the reverse time SDE?\n- Use of Mish activation function seems a bit non-standard. How important is it?\n- L356: can you give a rough time comparison of the diffusion policy vs. standard approaches?\n- How does entropy regularization interact with the SDE sampling? Is MaxEnt required for diffusion to be useful?\n- Can you increase linewidth in Fig 1a/c and Fig 3? Adding symbol/markers would be helpful too for accessibility\n\nminor typographical:\n- L10: \"is tractable and that **it** can...\"\n- L27: rephrase \"work introduce\"\n- L95-100, maybe point to Fig 1 to see multi-modality of proposed approach", "limitations": "Beyond the weaknesses mentioned above, I believe the limitations were discussed adequately.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Compares favorably against most relevant benchmark (Diff-QL)\n- Good ablation studies; this was helpful to understand the importance of each component.\n- Strong connection to standard tools in SDEs from diffusion literature", "weaknesses": "I'm not sure how big a weakness this is, but there appears to be some disconnect in the theory vs. experiments for discounting, that is infinite-time horizons. Why do you introduce a discount factor but in all equations use a finite time horizon?\n\n- High memory requirement: What is the memory cost (i.e. in VRAM) for an ensemble of 64 models?\n- Inconsistent use of \"max Q trick\" from [27]. Can you explain why this is used only in a limited set of environments? What happens if it is removed? A performance comparison would be useful.\n\n\n**Experiments**\n- Given that this is a (mostly) experimental paper, it would be good to have more experiments, in particular some studies on the sensitivity of $\\beta$ (is it OK to be equal for all envs? it seems like the magnitude of Q-value variance for the LCB should be MDP-dependent.)\n- Use of $T=5$ steps: can you show any (simple) experiments illustrating the use of various $T$ values? It seems implicitly dependent on the environment since it somehow controls the expressivity of the policy (please correct me if wrong)\n- L153, choice of $\\eta$ (see above comments)\n- In figure 3, can the training time be extended? Especially regarding the decrease in performance for antmaze-med-play; if there is always a decrease over time in training this is important to note. (on the flip side, it can be noted as a useful limitation for future work to improve on)\n- Using non-scalar comparisons would be beneficial to gain a better idea of the statistical comparison of all algorithms. (cf. https://github.com/google-research/rliable in case data is still available, it should be straightforward to generate the plots)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback and the opportunity to improve our work. Your insights are invaluable, and we look forward to incorporating these revisions to strengthen our submission.\n\n**Weakness**\n\n1. *Why do you introduce a discount factor but in all equations use a finite time horizon?*\n\n   Note that all considered environments are continuous tasks, and the return is computed using an infinite time horizon. Hence, a discount factor is used. On the other hand, a finite time horizon is used for the SDE steps. Note that the superscript $t$ of $a_i^t$ is the SDE step, and the subscript $i$ is the RL step.\n\n2. *Memory Requirement of 64 Q-ensembles.*\n\n   For the Antmaze task, the model requires ~3GB and ~5.5GB for $M=4$ and $M=64$, respectively. We will report more details in the revised manuscript.\n\n3. *The sensitivity of $\\beta$.*\n\n   The additional experiments of $\\beta=1,2,4$ are provided in **General Response Figure 5 and Table 1**. These results show that $\\beta=4$ performs the best in both *antmaze-medium-diverse* and *antmaze-medium-play* environments. The value of $\\beta$ determines the amount of pessimism, and of course, it can be tuned for different datasets.\n\n4. *Use of \"max Q trick\".*\n\n   We follow the Diffusion-QL to only use max Q-backup in Antmaze tasks. The ablation study of max Q-backup on selected Antmaze tasks is provided in **General Response Table 2**. One can see that the max Q-backup trick improves the Antmaze results.\n\n5. *Experiments on various $T$ values.*\n\n   The experiment of $T=3,5,10$ is shown in **General Response Table 3**. The step $T=5$ gives the best overall performance. Moreover, as shown in **General Response Figure 1**, when $T=2$, the diffusion model already achieves a good approximation of the original data distribution.\n\n6. *Choice of $\\eta$.*\n\n   We select the value of $\\eta$ by checking the most used value for each task group. Compared to Diffusion-QL which carefully tunes parameters for each environment, our settings are more general. Here, $\\eta$ is mainly for balancing the two losses but we did not explore different values in the paper.\n\n7. *Extend training time.*\n\n   We found that extending the training time decreases the performance of the antmaze task in most offline RL methods such as Diffusion-QL. The Q-ensemble technique can alleviate this problem but is still unstable and challenging. We are happy to note and list it as our future work.\n\n8. *Non-scalar comparisons.*\n\n   We appreciate the suggestion to use non-scalar comparisons for a more comprehensive statistical analysis. We have starred the project and will be happy to illustrate our non-scalar comparisons in the future.\n\n**Questions**\n\n1. *Figures improvement.*\n\n   Thank you for the suggestions on figure improvements. Due to the page limitation of the rebuttal PDF, we will try to update the figures (e.g., use IQM plots with error bars, increase the linewidth, and add symbols/markers) directly in the revised manuscript.\n\n2. *Notation problems e.g. \"$i$\" vs. \"$t$\" and \"$a$\" vs. \"$x$\".*\n\n   We will consider swapping the notations to align more closely with standard RL terminology, making the presentation clearer.\n\n3. *Why different gradient norms.*\n\n   Compared to Diffusion-QL which carefully tunes hyperparameters (e.g., the norm value) for each env, our settings are more general: the values are taken close to the average values as in Diffusion-QL and we use the same gradient norm value for the same type of tasks.\n\n4. *Missing Reference to SUNRISE.*\n\n   Thank you for the suggestion. We will include a discussion on SUNRISE and its relationship to our method, particularly how it compares to using LCB.\n\n5. *$Q^\\text{LCB}_\\psi$ in Algorithm 1.*\n\n   Thanks for pointing this out, we will change it to \"Update policy $\\pi_\\phi$ by (15) using $Q^\\text{LCB}_\\psi$\".\n\n6. *The sense of \"optimality\" regarding the reverse time SDE?*\n\n   The path for reverse-time SDE can be diverse and tortuous due to the stochasticity of the diffusion term $\\mathrm{d}w$, which requires long timesteps to research the real data distribution. Our optimal sampling utilizes the condition posterior trick (see Proposition 3.1) and can generate new samples in very few steps.\n\n7. *Mish activation function.*\n\n   In this work, we didn't focus on architecture tuning and our network simply follows the Diffusion-QL with three dense layers and the Mish activation function.\n\n8. *Time comparison of diffusion policy vs standard approaches.*\n\n   We have conducted additional experiments to provide training time and evaluation time comparison between our method and our method with Gaussian policy between different settings. The results are shown in \\textbf{General Response Table 4}. While our diffusion policy requires more computational resources during training compared to standard Gaussian policies, the performance gains justify the additional time investment. The evaluation times remain comparable, ensuring practical applicability in real-world scenarios. These results provide a clearer understanding of the computational trade-offs and reinforce the robustness and effectiveness of our proposed method.\n\n9. *How does entropy regularization interact with the SDE sampling?*\n\n   During training, we first sample action from the SDE and then learn to maximize the approximation of the entropy to increase exploration of the environment in offline settings. In this stage, the MaxEnt is helpful for policy learning. Then, in inference, we can directly sample actions from the SDE and the entropy won't affect the sampling.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for their detailed reviews and constructive comments. We have conducted additional experiments to address the raised concerns and further validate our approach. All figures can be found in the attached PDF file. Below, we summarize the key results and discussions related to these experiments.

**Figure 1 (for Reviewer jdmN):** *The proposed optimal sampling with different sample steps.*
We added the figures of data generation with fewer steps ($T=1$ and $T=2$) for the toy task in Section 3.1. The results show that the optimal sampling strategy significantly outperforms reverse-time SDE in all steps, further demonstrating the efficiency and effectiveness of our method. 

**Figure 2 (for Reviewer jdmN):** *Corrected learning curves of the Diffusion-QL and our method on selected Antmaze tasks.*
We updated the learning curves of our method and Diffusion-QL on selected AntMaze tasks, showing that our method consistently outperforms Diffusion-QL across various environments (the subfigure for ant-medium-diverse-v0 env in the draft uses the wrong data but now it is fixed).

**Figure 3 (for Reviewer jdmN):** *Additional learning curves of our method on different environments over 5 random seeds.*
We validate the robustness of our method with additional learning curves. However, due to the page limitation, we only provide the learning curves of Gym-medium-replay environments, and we will add the rest directly to the appendix of the revised manuscript.

**Figure 4 (for Reviewer Jzch):** *Visualization of the workings of the mean-reverting SDE for action prediction.*
For a more intuitive explanation of our approach, Figure 4 outlines the forward and reverse processes of the mean-reverting SDE used for action prediction.

**Figure 5 and Table 1 (for Reviewer T2t7):** *Ablation experiments of our method with different LCB coefficients.*
To explore the impact of different LCB coefficients $\beta$. We add an experiment of our method with $\beta$ values of 1, 2, and 4 on AntMaze-medium environments. Figure 5 demonstrates that adjusting the LCB coefficient improves performance, particularly for higher values, which helps in managing the exploration-exploitation trade-off effectively. In addition, the numerical results are provided in Table 1 below.

### Table 1: Experiment on LCB coefficients $\beta$

| LCB Coefficient $\beta$ | 1          | 2          | 4              |
| ------------------------- | ---------- | ---------- | -------------- |
| Antmaze-medium-play-v0    | 82.4 ± 4.9 | 88.6 ± 1.5 | **91.6 ± 2.3** |
| Antmaze-medium-diverse-v0 | 74.6 ± 3.7 | 84.0 ± 7.8 | **91.4 ± 1.5** |
| **Average**   | 78.5       | 86.3       | **91.5**       |



**Table 2 (for Reviewer T2t7):** *Ablation experiments with max Q-backup trick on AntMaze tasks.*
We conducted experiments with and without max Q-backup on AntMaze tasks in Table 2. The inclusion of max Q-backup significantly enhances performance, particularly in more complex environments (e.g., Antmaze-large).



### Table 2: Experiment on "Max Q trick"
| Max Q-backup              | True           | False      |
| ------------------------- | -------------- | ---------- |
| Antmaze-medium-play-v0    | **91.6 ± 2.3** | 89.2 ± 2.9 |
| Antmaze-medium-diverse-v0 | **91.4 ± 1.5** | 87.6 ± 1.8 |
| Antmaze-large-play-v0     | **81.2 ± 3.0** | 22.3 ± 7.1 |
| Antmaze-large-diverse-v0  | **76.4 ± 2.1** | 26.5 ± 6.1 |

**Table 3 (for Reviewer T2t7):** *Ablation experiments with different diffusion steps on selected AntMaze tasks.*
We evaluated the impact of varying the number of diffusion steps on a range of tasks, including AntMaze, Gym, and Kitchen in **Table 3**. Our findings indicate that while increasing the number of steps generally improves performance, five steps provide the best balance across different tasks and between performance and computational time. Thus we choose $T=5$ for all tasks in the paper.


### Table 3: Experiment on diffusion step $T$

| Diffusion Step $T$         | 3        | 5         | 10       |
| ---------------------------- | -------- | --------- | -------- |
| Halfcheetah-medium-replay-v2 | 43.4     | **57.0**  | 49.5     |
| Hopper-medium-replay-v2      | 39.4  | **102.7** | 101.7    |
| Walker2d-medium-replay-v2    | 51.2   | 94.2 | **98.1** |
| Antmaze-medium-play-v0       | **96.6** | 91.6| 90.2   |
| Antmaze-medium-diverse-v0    | **95.8** | 91.4   | 83.8   |
| Antmaze-large-play-v0        | 67.6     | **81.2**  | 63.2   |
| Antmaze-large-diverse-v0     | **81.0** | 76.4   | 70.0    |
| pen-human-v1    | 65.4     | 67.2      | **70.0** |
| pen-cloned-v1    | 67.3     | 66.3      | **68.4** |
| Kitchen-complete-v0        | 7.5      | 82.3      | **92.7** |
| Kitchen-partial-v0| 10.9     | 60.3    | **66.3** |
| Kitchen-mixed-v0   | 4.8      | 60.2    | **68.0** |
| **Average**| 52.6     | **77.6**  | 76.8     |

**Table 4 (for all Reviewers):** *Computational time comparison.*
We included a detailed comparison of training and evaluation times for Gaussian/Diffusion policies and Q-ensembles in Table 4 below.
Increasing $M$from 2 to 64 almost does not influence the evaluation time.
The diffusion step $T$ has more impact on both training and evaluation time but it's a common problem in diffusion models. We regard the sample acceleration as our future work and will try to address it with reliable SDE/ODE solvers or diffusion distillation techniques.


### Table 4: Computational time comparison with different settings on Antmaze-medium-play-v0

| Policy    | Diffusion Step $T$ | # Critics $M$ | Training Time (1 Epoch) | Eval Time (1k steps) |
| --------- | -------------------- | --------------- | ----------------------- | -------------------- |
| Gaussian  | 1    | 2 | 5m 35s | 1s 450ms   |
| Gaussian  | 1  | 64 | 7m 20s | 1s 450ms |
| Diffusion | 5  | 2 | 9m 30s | 4s 800ms |
| Diffusion | 5   | 64  | 11m  | 4s 800ms |
| Diffusion | 10  | 2  | 12m 23s  | 8s    |
| Diffusion | 10| 64  | 13m 55s| 8s|

Meta-review: The reviewers and myself are all in agreement that this paper is clearly written, provides a well-founded method with adequate empirical evidence showcasing its merits. While some concerns were raised in the reviews, the authors have addressed most of them during the rebuttal. As such, I am recommending an acceptance.
Decision: Accept (poster)