Diffusion Policies Creating a Trust Region for Offline Reinforcement Learning
Authors: Tianyu Chen
Zhendong Wang
Mingyuan Zhou
Venue: NeurIPS 2024 poster (2024)
Keywords: offline RL
diffusion models
behavior cloning
policy regularization
Q-learning
Primary area: reinforcement_learning
TL;DR: efficient diffusion q learning in both training and inference.

Abstract: Offline reinforcement learning (RL) leverages pre-collected datasets to train optimal policies. Diffusion Q-Learning (DQL), introducing diffusion models as a powerful and expressive policy class, significantly boosts the performance of offline RL. However, its reliance on iterative denoising sampling to generate actions slows down both training and inference. While several recent attempts have tried to accelerate diffusion-QL, the improvement in training and/or inference speed often results in degraded performance. In this paper, we introduce a dual policy approach, Diffusion Trusted Q-Learning (DTQL), which comprises a diffusion policy for pure behavior cloning and a practical one-step policy. We bridge the two polices by a newly introduced diffusion trust region loss. The diffusion policy maintains expressiveness, while the trust region loss directs the one-step policy to explore freely and seek modes within the region defined by the diffusion policy. DTQL eliminates the need for iterative denoising sampling during both training and inference, making it remarkably computationally efficient. We evaluate its effectiveness and algorithmic characteristics against popular Kullback-Leibler (KL) based distillation methods in 2D bandit scenarios and gym tasks. We then show that DTQL could not only outperform other methods on the majority of the D4RL benchmark tasks but also demonstrate efficiency in training and inference speeds. The PyTorch implementation is available at https://github.com/TianyuCodings/Diffusion_Trusted_Q_Learning.

Diffusion Policies Creating a Trust Region for Offline Reinforcement Learning
Tianyu Chen
Zhendong Wang
Mingyuan Zhou
The University of Texas at Austin
ftianyuchen, zhendong. wang) @utexas. edu mingyuan... zhou@mccooosseee wut.c/oow............e..oee..... wtw.ts.eee... edu
Abstract
Offline reinforcement learning (RL) leverages pre-collected datasets to train op- timal policies. Diffusion Q-Learning (DQL), introducing diffusion models as powerful and expressive policy class, significantly boosts the performance of offline RL. However, its reliance on iterative denoising sampling to generate ac- tions slows down both training and inference. While several recent attempts have tried to accelerate diffusion-QI, the improvement in training and/or inference speed often results in degraded performance. In this paper, we introduce dual policy approach, Diffusion Trusted Q-I-nnning (DTQL., which comprises a diffu- sion policy for pure behavior cloning and a practical one-step policy. We bridge the two polices by a newly introduced diffusion trust region loss. The diffusion policy maintains expressiveness, while the trust region loss directs the one-step policy to explore freely and seek modes within the region defined by the diffu- sion policy. DTQL eliminates the need for iterative denoising sampling during both training and inference, making it remarkably computationally efficient. We evaluate its effectiveness and algorithmic characteristics against popular Kullback- Leibler divergence- baaeed distillation methods in 21) bandit scenarios and gym tasks. We then show that DTQQL could not only outperform other methods on the majority of the D4RL benchmark tasks but also demonstrate efficiency in training and inference speeds. The PyTorch implementation available lhtts htg.ta//....//aaaaaoo/////////// htw.rruuuiioooruuueeeeuueeoo
Trusted Bruusted___Lerniin
1 Introduction
Reinforcement learning (RL) centers on developing policy to make sequential decisions by inter- acting with an environment, aiming to maximize the total rewards accumulated over a trajectory [Wiering and Van Otterlo 2012 Li 2017]. Offline RI. addresses these challenges by enabling the training of an RL policy from fixed datasets of previously collected data, without further interactions with the environment [Lange al. 2012 Fu et al.) 2020). This approach leverages large-scale historical data, mitigating the risks and costs associated with live environment exploration. However, offline RL introduces its set of challenges, primarily related the distribution shift between the data on which the policy was trained and the data it encounters during evaluation [Fujimoto et al.. 2019). Additionally, the limited expressive power of policies that may not adequately capture the multimodal nature of action behaviors is also a concern.
To mitigate distribution shifts, popular approaches include weighted regression, such as IQL (Kostrikov et al. [02T] and AWAC [Nair et al.] 2020), aimed at extracting viable policies from historical data. Alternatively, behaaior-rguularized policy optimization techniques employed to constrain the divergence between the learned and in-sample policies during training [Wu et al. 2019) Notable examples of this strategy include TD3-BC (Fujimoto and Gul 202T], CQL Kumar et al.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

2020J, and BEAR [Kumar et al. 2019]. These methods primarily utilize either Gaussian or determin- sstc policies, which have faced criticism for their limited expressiveness. Recent advancements have incorporated generative models to enhance policy representation. Variational Autoencoders (VAFs) Kingma and Welling 2013] and Generative Adversarial Networks (GANs) [Gooodfellow et al. 2020] have been introduced into the offline RL domain, leading to the development of algorithms such as BCQ [Fujimoto et al. 2019] and GAN-Joint (Yang et al.) 2022). Moreover, diffusion models have
recently emerged the most prevalent tools for achieving expressive policy frameworks [Janner
ct al. 2022 Wang et al. 2022a| Chen et al. 2023 Lalassn--Estrree et al. 2023 Chen et al. 2022].
demonstrating state-ofthhe oh-t---rrtt-- performance on the D4RL benchmarks. Diffusion Q-Learning (DQL)
[Wang et al. 2022a] applies these policies for behavior regularization, while algorithms such as IDQL
[Hansen-Estrus Estruchtreuchhh et al. 2023 leverage diffusion-baat based policies for policy extraction.
However, optimizing diffusion policies for rewards in RL computationally expensive due to the need for iteratively denoising to generate actions during both training and inference. Recently, distillation has become a popular technique for reducing the computational costs of diffusion models, e.g., score distillation sampling (SDS) (Poole et al. 2022) and variational score distillation (VSD) [Wang et al. [2024] in 3D generation, and Diff-Instruct [Luo et al.] 2024], Distribution Matching Distillation Yin et al., 2023], and Score identity Distillation (SiD) [Zhou et al. 2024) in 2D. These
advancements distill the iterative denoising process of diffusion models into a one-step generator.
SRPO [Chen et al.
2023 employs SDS (Poole et al.) (222) in the offline RL field by incorporating
Kullback- (KL) divergence-basee behavior regularization loss to reduce training and inference
costs. Another related work. IDQL [Hansen- Estrunc et al. 2023], , selects action candidates from a diffusion behavior-cloning policy and requires a 5-step iterative denoising process to generate multiple candidate actions (ranging from 32 to 128) during inference, which remains computationally demanding. Unlike previous approaches, our paper introduces a diffusion trust region loss that moves away from focusing on distribution matching; instead, emphasizes establishing a safe, in-sample behavior region. We then simultancously train dual policies: a diffusion policy for pure behavior cloning and a one-step policy for actual deployment. The one-step policy is optimized based on two objectives: the diffusion trust region loss, which ensures safe policy exploration, and the maximization of the Q-value function, guiding the policy to generate actions in high-reward regions. We elucidate the differences between our diffusion trust region loss and KL-based behavior distillation in Section 3 empirically and theoretically. Our method consistently outperforms KL-based behavior distillation approaches. We provide more discussions on related work in Appendix
In summary, we propose DTQL with diffusion trust region loss. DTQL achieves new state-of- the-art results in majority of D4RL [Fu et al.) [020] benchmark tasks and demonstrates significant improvements in training and inference time efficiency over DQL [Wang al. 2022a] and related diffusion- methods.
2 Diffusion Trusted Q-Learning
Below, we first introduce the preliminaries of offline RL and basics of diffusion policies for our modeling. We then propose new diffusion trust region loss which inherently avoids exploring dustr--distritution actions and hence enables safe and free policy exploration. Finally, we introduce our algorithm Diffusion Trusted Q-Learning (DTQL), which is efficient and well-performed.
2.1 Preliminaries
In RL, the environment is typically defined within the context of a Markov Decision Process (MDP). An MDP is characterized by the tuple M = A, (s()), p)sspssssss r(s, ((s.a))))) a), where S denotes the state space, AA represents the action space, (s(s) is the initial state distribution, p(s') [s. a) is the transition kernel, r(s, a) is the reward function, and y is the discount factor. The objective is to learn a policy my (a) fals), parameterized by 0, that maximizes the cumulative discounted reward E [> 7'r(s1, a.)]. In the offline setting, instead of interacting with the environment, the agent relies solely on a static dataset = (s, a, r, 8'} collected by a behavior policy ne (als). This dataset is the only source of information for the agents.
2

---

2.2 Diffusion Policy
Diffusion models are powerful generative tools that operate by defining forward diffusion process to gradually perturb a data distribution into a noise distribution. This model is then employed to reverse the diffusion process, generating data samples from pure noise. While training diffusion models is computationally inexpensive, inference is often costly due to the need for iterative refinement- -asedd sequential denoising. In this paper, we only train a diffusion model and avoid using for inference, thus significantly reducing both training and inference times.
The forward process involves initially sampling 2o from an unknown data distribution p( followed by the addition of Gaussian noise to Zo. denoted by æt. The transition kernel qr (2 (2)) is given by 2t 0,330 + o,E, where a and oN are predefined, and e represents random Gaussian noise.
The objective function of the diffusion model aims to train a predictor for denoising noisy samples back to clean samples, represented by the optimization problem:
min E., 1) (wo (a () xo
where w(() weighted function dependent only on offline RL, since our training data is state-action pairs, we train a diffusion policy using a conditional diffusion model as follows:
( =
(or(() (t)ll)) f|s) a.l1
where ag; are the action and state samples from offline datasets DD and a; 04 + o,e. Following
previous work [Chen et al. 2023 Hansen-Estruch et al. 2023 Wang et al. 2022a pLa, f|s) can be
considered an effective behavior-cloning policy.
The ELBO Objective The diffusion denoising loss intrinsically connected with the evidence
lower bound (ELBO) It has been demonstrated in prior studies [Ho et al. [020 [ong et al. 2021
Kingma et al. 2021| Kingma and Gao 2024) that the ELBO for continuous- diffusion models can be simplified the following expression (adopted in our setting):
logp(ag|s) 2 ELBO( ()) = E. F=..,..1),.-.00.... (r(())lm 48) a + e,
(3)
where ag = 0,00 w(()) = aSNR(+) and the signal-to-noise ratio SNR(() o c constant
dt
relevant @. Since we always assume that the SNR(t) is strictly monotonically decreasing in t, thus (t) > 0. The validity of the ELBO is maintained regardless of the schedule of a and oz.
Kingma and Gao 2024 generalized this theorem stating that if the weighting funetion w(t)
dS) -v(() dt where vl)) monotonic increasing function of I, then this weighted diffusion denoising
loss is equivalent to the ELBO as defined in Equation 3 The details of how to train the diffusion
policy, including the weight and noise schedules, will be discussed in Section 4.3
2.3 Diffusion Trust Region Loss
We found that optimizing diffusion denoising loss from the data perspective with a fixed diffusion model can intrinsically disencourage out-of-distribution sampling and lead mode seeking, For any given 8 and a fixed diffusion model Psg the loss is to find the optimal generation function me(- () that minimize the diffusion-based trust region (TR) loss:
(i) = (D (s1s) (t) + t. t|s) - a0l1
(4)
where Te (a]s) is a one-step generation policy, such as a Gaussian policy.
Theorem 1. policy ps satisfies the ELBO condition of Equation 3 then the Diffusioon Trust Region
Loss aims to maximize the lower bound of the distribution mode max log p(ao {)r any given 8.
Proof. For any given state 8 max log pl (a (s) 2 max maxEao [log((a)ss) 2 maxEao (11)) [ELBO(a)|s]] do
min F
[w(tt]] + o.e, t|s) a.11 [| +
Then, during training, we consider states D. Thus, by taking the expectation over 8 D on both sides and setting t ~ U(0, 1). derive the loss described in Equation 4
3

---

By definition of the mode of probability distribution, we know minimizing the loss given by Equation aims maximize the lower bound of the mode of a probability. Unlike other diffusion models that generate various modalities by optimizing Φ to learn the data distribution, our method specifically aims to generate actions (data) that reside the high-density region of the data manifold specified by js through optimizing 6. Thus, the loss effectively creates trust region defined by the diffusion-bain based behavior-cloning policy, within which the one-step policy TTe can move freely. the generated action deviates significantly from this trust region, it will be heavily penalized.
Remark 1. For any given s, assuming that our training set consists of a finite number of samples (a....., a. ). this implies that p(æ) s) is represented by a mixture of Dirac delta distributions:
1 -=55
p(
8(x -ac-aa ab)
This indicates that all actions a appearing in the training set have a uniform probability mass. Therefore, the generated action ag can be any of the actions in {a)...., 03 minimize LTR(0) in Equation 4 since all of them are modes of the data distribution.
Remark 2. This loss is also closely connected with Diffusion-GAr (Wang al. 2022b| and EB-GAN IZhao et al 2016), where the discriminator loss is considered as:
D(aals) = (Dec(En((aa))s)) aoll
our model, the process of adding noise, 0g + otC, functions as an encoder, and P. acts as decoder. Thus, this loss can also be considered the generated action ag resembles the training dataset.
discriminator loss, which determines whether
Remark 3. By Theorem 7 the trust region can be defined using the conditional lilieeliihodd. Specifically, for given state the trust region for an action is defined as the set {a log p(a s) 2 threshold), where the conditional logliiilidood is approximated by the diffusion loss. The threshold can be adjusted by tuning the hyperparameter during the optimization of the final loss (Eq. 5).
This approach makes the generated action ag appear similar to in-sample actions and penalizes those that differ, thereby effectuating behavior regularization. Thus, visualization of the toy examples (Figure I can help better understand how this loss behaves. The generated action ag will incur small diffusion loss when it resembles a true sampa action and high diffusion loss if it deviates significantly from the true in-sample action.
Dala
1 0.02
0.1
1=1
<=
2- 80
High
Action Dimensonn
Loss
Actinn
Action Dimensioon
Action Dimonsiio
Action Dimension 1
Action Dimension
Action Dimendion
1
Figure I: Diffusion trust region loss. The first column shows how the training behavior dataset looks. Columns 2-6 display the diffusion trust region loss on different actions with varying magnitudes of Gaussian noise. We can observe that the trust regions captured by the diffusion model at given f are consistent with the nigh-density regions of the noisy data at that specific t. For example, when is small, the diffusion loss is minimal where the true action lies, and high in all other locations.
2.4 Diffusion Trusted Q-Learnning
We motivate our final algorithm from DQL Wang et al. 2022a which utilizes diffusion model an expressive policy to facilitate accurate policy regularization, ensuring that exploration remains within safe region. Qeearnnng implemented by maximizing the valuue function at actions sampled from

---

the diffusion policy. However, sampling actions from diffusion models can be time-connse consuming, and computing gradients of the Q-value function while backpropagating through all diffusion timesteps may result in a vanishing gradient problem, especially when the number of timesteps is substantial,
Building this, introduce dual-policy approach, Diffusion Trusted Q-Learning (DTQL): diffusion policy for pure behavior cloning and a one-step policy for actual depolyment. We bridge the two policies through our newly introduced diffusion trust region loss, detailed in Section 2.3 The diffusion policy ensures that behavior cloning remains expressive, while the trust region loss enables the one-step policy to explore freely and seek modes within the region designated by the diffusion policy. The trust region loss is optimized efficiently through each diffusion timestep without requiring the inference of the diffusion policy. DTQL not only maintains an expressive exploration region but also facilitates efficient optimization. We further discuss the mode-seeking behavior of the diffusion trust region loss Section Next, we delve into the specifics of our algorithm.
Policy Learning. Diffusion inference is not required during training or evaluation in our algorithm; therefore, we utilize unlimited number of timesteps and construct the diffusion policy (e in
continuous-time setting. based on the schedule outlined in EDM Karras et | 2022]. Further details provided in Section 4.3 The diffusion policy Po can be efficiently optimized by minimizing (())
as described in Equation ( Furthermore, we can instantiate one typical one-step policy To (a)a) 8) in
two cases, Gaussian Te (a) 8)) (s) (g())))))))))))))))))))))))))) (8)) Implicit ag ( e), E NNO I). Then,
optimize Te by minimizing the introduced diffusion trust region loss and typical Q-value function maximization, as follows.
L (0) = a (0) BayDaay [0, aæ)],
(5)
where LTR (0) serves primarily as a behavior-regullrizatio term, and maximizing the Q-value
function enables the model preferentially sample actions associated with higher values. Here
use the double Q-learning trick (Hasselt 2010) where Q,,(s, a.) = min(Q (s, as). Qn (8, aa)). If Gaussian policy is used, an additional negative log likelihood (NLL) term, a [log TT [a]s)] should be introduced to preserve the policy's 's entropy and encourage exploration during training. This aspect is particularly crucial for diverse and sparse reward RL tasks. The empirical results of the NLL
term will be discussed in Section 4.4
Q-Learning. We utilize Implicit Q-Learning (IQL) to train Q function by maintaining two funetions (Qn, Qn and one value function Ve, following the methodology outlined IQL
Kostrikov et al. 2021].
The loss function for the value function Vi is defined as:
(v()) (.. [Tg] (min(Q (s, a); Ong a)) - Vs(())]
(6)
1(w < where T is quantile in [o, 1]. and L (u) = =|r 11w
0)/u². When 7 = 0.5, L3 simplifies to the
L2 loss. When 7 > 0.5, Ly encourages the learning of the quantile values of Q.
The loss function for updating the Q-functions, Q given by:
(%) = ((....''<)) [[]r(s, a) + Vs((') Qm (s, a)l
(7)
where y denotes the discount factor. This setup aims to minimize the error between the predicted Q-values and the target values derived from the value function Vi and the rewards. We summarize our algorithm in Algorithm
3 Comparison of Different Mode-Seeking Behavior Regularizations
Another approach to accelerate training and inference in diffusion- based policy learning involves utilizing distillation techniques. Methods such as SDS [Poole et al. [002].. VSD [Wang et al. 2024). Diff-Instruet [Luo et al. [ and DMD (Yin al.) 2023 illustrate this strategy. These papers share a common theme: using a trained diffusion model alongside another diffusion network to minimize the KL divergence between the models. In our experimental setup, this strategy employed for behavior regularization by
Ptake (as (asls)
CKL (0)
log Preat (agls)
(8)
where (8, instantiated one-step implicit policy.
5

---

Algorithm Diffusion Trusted Q-Llearning
Initialize policy network To, Ma, critic networks Qn and Q12 and target networks Org and Ong. value function Ve for each iteration do
Sample transition mini-batch B {(s ae, rt, 8[+1]] D
1. Q-value function learning: Update Qn Qn and v4 by Lo and Lv (Eqs. 6 and 7).
2. Diffusion Policy learning: Update Ma by [()) (Eq. 2
3. Diffusion Trust Region Policy learning: 0g ~ To ( s). Update wo by 2 (6) (Eq.
4. Update target networks: r// = pT/ (1 p)7 for: (1,2].
end for
As do not have access to the log densities of the fake and true conditional distributions of actions, the loss itself cannot be calculated directly. However, we are able to compute the gradients. The gradient of log Preat aa|s) can be estimated by the diffusion model ( (), and the gradi- ent of log Ptake (aal)) can also be estimated by a diffusion model trained from fake action data ag. For more details, please refer to Appendix C
Training Data
Generation
KL divergence is employed in this context with the goal of capturing multiple modalities of the data distribution. We evaluated this loss function using 2D toy task to gain deeper understanding of its capability to capture multiple modalities of the dataset, illustrated in Figure 2
Figure 2: datasets trained Red points generated by ny. trained using L This demonstrates that the KI. loss encour- ages the generation process to cover mul- tiple modalities of the dataset.
Green points represent the
We further investigate the differences between our trust region loss, CTR. and the KL-based behavior distillation loss within the context of policy improvement. As illustrated in Figure D CTR ensures that the generated action as remains within the action manifold of the in-sample dataset. Coupled with the gradient the Q-function, this allows actions to move freely within the in-sample data manifold while gravitating toward high-reward regions, which correspond the single modality present in the dataset.
Conversely, Lx1. (0) seeks to align the distribution of (T(-) with that of Map(118) (-)s), thereby encouraging coverage multiple modalities, unlike LTR. Covering a wide range of modalities particularly beneficial in image generation, where diversity among generated images is essential. However, this characteristic is less advantageous in reinforcement learning (RL) contexts, where typically a single, highest-reward action is optimal for given state. Additionally, maximizing the Q function often results in a more deterministic policy by favoring the highest-reward paths, potentially discarding alternative actions. From this perspective, LIR demonstrates stronger mode-seeking capability compared to LKL-
To visualize how these two different behavior losses work with policy improvement, we use 2D bandit scenarios. We designed scenario shown in Figure for additional settings, please refer to Appendix F.T In the designed 25 Gaussian setting, all four corners have the same high reward. LTR encourages the policy to randomly select one high reward mode without promoting covering all of them. In contrast, LKL tries cover all high-density and high-reward regions and, byproduct, introduces artifacts that appear as data connecting these high-density regions. This could partially be due to the smoothness constraint of neural networks. The same situation occurs in Swiss roll dataset where the high reward region is the center the data; LyR adheres closely to the high reward region, while LXL includes some suboptimal reward regions.
In addition to testing on 2D bandit scenarios, we also evaluated the performance of two losses on the Mujoco Gym Medium task. Consistent with our previous findings, the hehavior-regularization loss LTR(( (0) consistently outperformed Lx.. (0) in terms of achieving higher rewards. The results are presented in Table 5 and the training curves are depicted in Figure in Appendix F2

---

Trust Region
KL
Region
KL
High
Reward Value
Low
(a) Guassian example with four corner have equally highest reward.
(b) Swiss roll example with center have highest reward.
Figure 3: We tested the differential impact of LTR and LxL. behavior regularization, using trained Q-function for policy improvement. Red points represent actions generated from the one-step policy no.
Connection and Difference with SDS and SRPO SDS was first proposed in DreamFusion (Poole et al. 2022 for 3D generation, using the gradient of the loss form (adopted in our setting):
Ozz
VeLSDS Er, ((t))(s((z,
(9)
00
where 2; = 0,40 0,5 and &g is the noise-prediction diffusion model. This loss utilized by SRPO Chen
2023 in offline RL.
Considering the gradient of (TR(0) in Equation 4 and acknowledging the equivalence between noise-prediction and data-prediction diffusion models with only modification in the weight function w()), we can reformulate the loss noise- coise-prediction form by:
LI (6) (t))e: = Os (z,, 11s)
(10)
0z: 00
VOLTR (0) 22r,, w (1())))(())))))) E)
(11)
Oz.
The primary distinction between the gradient of method, as shown in Equation TT and that of SDS/SRPO, detailed Equation lies in the inclusion
Jacobian term, des(si,t)s) This Jacobian 8z,
term, identified as the score gradient in SiD by Zhou et al. (2024), is notably absent from most theoretical discussions and was deliberately omitted in previous works, with DreamFusion (Poole
ct al. 2022 and SiD being the sole exceptions.
DreamFusion reported that the gradient depicted in Equation 11 fails to produce realistic 3D samples. Similarly, SiD observed inadequacy in generating realistic images. These findings align with our Theorem which demonstrates that this gradient primarily targets the mode and does not sufficiently account for diversity- an essential factor in both 3D and image generation.
In high-dimensional generative models, modes often differ significantly from typical image samples. discussed by Nalisnick et al. (2018). DreamFusion observed that the gradient from Equation 9 which is based on KL loss, effectively promotes diversity. However, while diversity is crucial in image and 3D generation, might be lesser importance in offline RL. Consequently. SRPO's use of the SDS gradient, which is tailored for diverse generation, may result in suboptimal performance compared our diffusion trust region loss. This assertion is supported by empirical results on the D4RL datasets, discussed in Section 4.1
Experiments
In this section, we evaluate our method using the popular D4RL benchmark (Fu al. 2020). We further compare our training and inference efficiency against other baseline methods. Additionally, an ablation study on the negative log likelihood (NLL.) term and one-step policy choice is presented. Details regarding the training of the diffusion model and its structural components also discussed.
7

---

Table 1: The performance of Our methods and SOTA baselines on D4RL Gym, AntMaze, Adroit, and Kitchen tasks. Results for our methods correspond to the mean and standard errors of normalized scores over random rollouts (5 independently trained models and 10 trajectories per model) for Gym tasks, which generally exhibit low variance in performance, and over 500 random rollouts (5 independently trained models and 100 trajectories per model) for the other tasks. Our method outperforms all prior methods by clear margin most of domains. The normalized scores is recorded by the end of training phase. Numbers within 5 % of the maximum in every individual task are highlighted.
Gym
BC
Onestep RL
TD3-BC
DT
CQL
IQL.
DQL
IDQL
SRPO
Ours
halfcheetab medium--2 v2
48.4
48.3
44.0
47.4
0.13
happer-madium-.
52.9
59.6
$9.3
67.6
58.5
66.3
90.5
65.4
95.5
99.6--0.87
walker2d medium-vy v2
75.6
81.8
83.7
74.0
72.5
78.3
87.0
82.5
84.4
89.4.0.133
halfchectah- medium- repllyy--222
38.1
45.2
44.2
45.9
50.9-0011
hopper medium replay- v2
18.1
97.5
60.9
82.7
95.0
94.7
101.3
92.1
101.2
100.010.13
49.5
77.2
73.9
95.5
85.1
88.5= 2.16
halfcheetahh medium cereeiiim--epppete expert-v2
55.2
93.4
90.7
86.8
91.5
86.7
96.8
95,9
92.2
92.7 + 0.2
aopor-mddiun-exeerte
52.5
103,3
98.0
107.6
105.8
91.5
111.1
108.6
100.1
109.3 1,49
walker2d-medinne expert-v2
107.1
109.4
109.6
0.07
Gym Average
51.9
76.1
75.3
74.7
77.6
77.0
88.0
82.1
87.1
88.7
Antmaze
BC
Onestep RI.
TD3-BC
DT
CQL.
IQL.
DQL
IDQL
SRPO
Ours
umaze-vD
54.6
64.3
78.6
59.2
74.0
87.5
93.4
94.0
90.8
94,8 1
unimaze umaze- umaze--ivvrsse- diverse-vi
45.6
60.7
71.4
53.0
84.0
62.2
66.2
80.2
59.0
78.8=1.83
untmaze- medium- play-vo
0.0
10.6
0.0
0.0
61.2
71.2
76.6
84.5
73.0
79.6 + 1.8
animaae medium- -iveess----
53.7
70.0
1.71
untmaze-larg large play-s0
0.0
0.0
0.0
0.0
15.8
39.6
46.4
63.5
38.8
52.0+ 2.23
untmazze- large diverse-vO
0.0
0.0
0.0
14.9
47.5
56.6
67.9
33.8
54.0
2.23
Antmaze Averuge
16.7
20.9
27.3
18.7
50.6
63.0
69.6
79.1
30.1
73.6
Adroit Tasks
BCQ
BRAC-P
BRAC-V
REM
CQL
IQL
DQL
-unahuman--l pon-elonad-vl
25.8
68.9
-1.0
8.1
0.6
5.4
35.2
71.5
72.8
64.14+2.97
44.0
-2.5
-1.0
37.3
81.3 3.04
Adrolt Average
32.1
56,3
12.8
4.9
-1.0
2.2
31.2
54.4
65.1
72.7
Kitchen Tasks
BC
BCQ
BEAR
BRAC-P
BRAC-
AWR
CQL
IQL
DQL
Ours
kitchen sombbn--ommlote-vt
33.8
8.1
0.0
0.0
0.0
0.0
43.8
62.5
84.0
80.8::1.00
kitchen purtial-vo
15.4
74.4
kitchen mixed-vil
47.5
8.1
47.2
0.0
0.0
10.6
51.0
51.0
62.6
60.2:10.59
Kitchen Average
11.7
8.7
71.8
In D4RL benchmarks, for all Antmaze tasks, we incorporate an NLL term, while for other tasks, this term is omitted. Additionally, we adjust the parameter a for different tasks. Details on hyperparameters and implementation are provided in Appendices D and
Hyperparameters
4.1 D4RL Performance
In Table
we evaluate the D4RL performance of our method against other offline algorithms. Our
selected benchmarks include conventional methods such TD3+BC [Fujimoto and Gu 202T and
IQL [Kostrikov et al. 2021]. along with newer diffusion-basee models like Diffusion QL (DQL)
Wang et 2022al IDQL Blansen-Estruct et al., 2023]. and SRPO (Chen et al. 2023
In the D4RL datasets, our method (DTQL) outperformed all conventional and other diffusion- based offline RL methods, including DQL and SRPO, across all tasks. Moreover, it is 10 times more efficient in inference than DQL and IDQL; and 5 times more efficient in total training wall time compared with IDQL (see Section 4.2)
Remark We would like to highlight that the SRPO method [Chen et al. 2023 reported results on Antmaze using the "-v2" version, which differs from the "-v0" version employed by prior methods such as DQL [Wang el al. [222aa] and IDQL Crunsen-Esttuce et al. 2023), to which it was compared. This version discrepancy, not explicitly stated in their puper, is evident upon inspection of SRPO's 's official codebase 1 The variation between the -v2" and -v0" datasets significantly impacts algorithm performance. To ensure a fair comparison, utilize the "-v0" environments consistent with established baselines. We employed the official SRPO code on Antemze-vO and maintained identical hyperparameters used for Antmaze-v2. Additionally, we conducted experiments with our
Refer to 7 httppp////gtitiioooo...aaaaaa//a
commit b006412

---

Table Training and Inference time required for different algorithms in D4RL antmaze-umazee tasks. Every single experiment is conducted with the same Py Torch backend and run on a single RTX-A5000 GPU.
antmazee-umaze-v0
DQL
IDQL
SRPO
Ours
Training time (s per 1k steps)
24.13
17.57
24.71
21.83
Inference time (s per trajectory)
3.03
3.04
0.22
0.35
Training epochs
1000
3000
1000
500
Total training time (hours)
6.70
14.64
9.42
3.33
algorithm on the Antmaze-v2 environment using the same hyperparameters as in the Antmaze-v0 setup but extended the training epochs, as detailed Table 6 in Appendix
4.2 Computational Efficiency
We further examine the training and inference performance relative other diffusion-based offline RL methods. An overview of this performance, using antmaaz-umaaze-v as benchmark, is presented in Table 2 Our method requires less training time per epoch than DQL and SRPO, yet more than IDQL. However, while IDQL necessitates 3000 epochs, DTQL operates efficiently with only 500 epochs, considerably reducing the overall training duration.
As depicted in Figure the extended training time per epoch for our method results from the requirement to train additional one-step policy, step needed by IDQI. Although SRPO also incorporates a one-step policy, our method achieves greater efficiency in training the diffusion policy. Unlike SRPO, which requires several ResNet blocks for effective performance, our approach utilizes only 4-layer MIP, further curtailing the training time. Additional details on total training wall time are provided in Appendix F.4
For inference time, our method performs compa- rably to SRPO, both utilize one-step policy. The slightly higher inference time results from use of a stochastic policy, which requires re- sampling after each forward pass of the neural network. Additionally, we employ the stochastic max Q trick, similar to that used in DQL Wang et al. [2022a] However, our method achieves a tenfold increase in inference speed over DQL and IDQL, which require 5-step iterative denois- ing to generate actions. All experiments were performed on server equipped with eight RTX- A5000 GPUs, each with 24GB of memory.
Q Loaming Diflusion Pollcy One Step Policy
Informm Iffrrncc Time
20
Traniing Time SK gredien
25
24.13
24.72
3.03
3.04
21.83
0.22
0.35
D0L
10OL
SAPO
brQL
Remark 5. For total training time, SRPO trains 1000 epochs for the one-step policy while train- ing 1500 epochs for the diffusion policy and Q function. DTQL requires 50 epochs of pretrain- ing. Implement details are in Appendix
Figure 4: Training time required for different al- gorithms in D4RL antmaze- -umzzzzz-mmzzeeO tasks. All experiments are conducted with the same Py PoTch backend and the same computing hardware setup.
4.3 Diffusion Training Schedule
For training the diffusion policy as described in Equation 2 and the diffusion trust region loss in Equation 4 we utilize the diffusion weight and noise schedule outlined in EDM Karras et al. 2022 Although EDM does not satisfy the ELBO condition stipulated in Equation
fact established
in Kingma and Gao [2024] we adopted it due to its demonstrated enhancements in perceptual generation quality, as evidenced by metrics such as the Fréchet Inception Distance (FID) and Inception Score in the field of image generation. |Kingma and Gao (024] also attempted to modify weight schedule to be monotonically increasing, but this did not lead to better FID for image generation. Thus, we retain EDM as our continuous training schedule. For completeness, the details of the EDM schedule are discussed in Appendix B
EDM

---

Avoiage Edooib Roaert
Saussan Palcy Entroy 11
Ausnage CaNoee Neltewers
Caus P4 Eniagy
Trained Epocts
Traned Epocns
(a) Reward Curve
(b) Entropy Curve
(c) Reward Curve
(d) Entropy Curve
Figure 5: Rewards and Gaussian policy entropy during training recorded and illustrated in the
figures. The blue line represents training without the addition of an NLL term, while the orange line indicates training with the NLL term included.
4.4 Ablation Studies
One-step Policy Choice We chose to use Gaussian policy for all our experiments instead of an implicit or deterministic policy because the Gaussian policy flexible and provides convenient way control entropy when needed. When there is no need to maintain entropy, the Gaussian policy quickly degenerates to deterministic policy, where the variance approaches zero, as indicated in Figures 5b and 5d
we incorporate an NLL term - -E,, -a, llog i (a|s)) into the loss function in Equation to maintain exploration and policy entropy during training when using a Gaussian policy. We conducted an ablation study assess its impact on the final rewards and the of the Gaussian policy, taking antmaze- -uaiizzeemmazeo and antmaze-large lareuzz--arr--iieeess
Negative Log Likelihood Term As mentioned in Section 2.4
vO as examples. As observed Figure 5| for the less complex task antmuze-umaze-ue adding the NLL term does not significantly enhance the final score but does stabilize the training process (see Figure 5a). However, for more complex tasks like animaze- antmaze-large-diiee large-diverse--O, the addition of the NLL term markedly increases the final score. We attribute this improvement to the ability of the NLL term to maintain high entropy during training, thus preserving exploration capabilities, as shown in
Figures 5b and [d
5 Conclusion and Limitation
In this work, we present DTQL, which comprises diffusion policy for pure behavior cloning and practical one-step policy. The diffusion policy maintains expressiveness, while the diffusion trust region loss introduced in this paper directs the one-step policy to explore freely and seek modes within the safe region defined by the diffusion policy. This training pipeline eliminates the need for iterative denoising sampling during both training and inference, making remarkably computationally efficient. Moreover, DTQL achieves fatte-of-the-as performance across the majority of tasks in the D4RL benchmark. Some limitations of DTQL include the potential for improvement in its benchmark performance. Additionally, some design aspects of the one-step policy could benefit from further investigation. Currently, our experiments are primarily conducted in an offline setting. It would be interesting explore how this method can be extended to online setting or adapted to handle more complex inputs, such as images. Additionally, rather than focusing solcly on point estimation of rewards, it would be beneficial to estimate the distribution of rewards, as recommended by previous studies in distributional reinforcement learning [Bellemare et al. 2017 Barth-Marow et al. 2018
Yuc
al. 2020
Acknowledgments
The authors acknowledge the support of NSF-IIS 2212418 and NIH-R37 CA271186.
References
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic
10

---

policy gradients. arXiv preprint 0rXiv:1804.086/7 2018.
Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement
learning. In International conference on machine learning, pages 449-458. PMLR, 2017.
Huayu Chen, Cheng Lu, Chengyang Ying. Hang Su, and Jun Zhu, Offline reinforcement learning via
high-fidelity generative behavior modeling. arXiv preprint (u.Xiv:2299.44355 2022.
Huayu Chen, Cheng Lu, Zhengyi Wang. Hang Su, and Jun Zhu. Score regularized policy optimization through diffusion behavior. arXiv preprint arXiv:2310.07297, 2023.
Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference Robot Learning, pages 158-168. PMLR, 2022.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiw:2004007219 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-ppolicy deep reinforcement learning without exploration. International conference on machine learning, pages
PMLR, 2019.
Seyed Kamyar Seyed Chasemipour, Dale Schuurmans, and Shixiang Shane Gu. EMAQ: Expected- max Q-learning operator for simple yet effective offline and online RL. In International Conference on Machine Learning. pages 3682-3691. PMLR, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, (3(11):139-144, 2020.
Philippe Mansen-Esttuch lya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: Implicit Q-learning as an actor-critic method with diffusion policies. arXiv preprins arXiv:2304. 10573, 2023.
Hado Hasselt. Double Q-leaming. Advances in neural information processing systems, 23, 2010.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-683 2020.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.099%) 2022.
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances Neural Information Processing Systems, 36, 2024.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 35:26565-266 26577, 2022.
Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the ELBO with simple data augmentation. Advances in Neural Information Processing Systems, 36, 2024.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21666-21707, 2021.
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. arXiv preprint 1rXiv:1312.61 14, 2013.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learning, arXiv preprint arXiv:2110.06166 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances neural information processing systems, 32, 2019.
11

---

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. Advances Neural Information Processing Systems. 33:1179-1191 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learnin In Reinforcement learning: State-of-the-ari, pages 45-73. Springer, 2012.
Yuxi Li. Deep reinforcement learning: An overview. arXiv preprint 1xXiv 2017.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A fast ODE solver for diffusion probabilistic model sampling in around steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
Weijian Luo, Tianyang Hu, Shifeng Zhang. Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff- Instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006. 003590000.099, 09359, 2020.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayaman. Do deep generative models know what they don't know? arXiv preprint arXiv: 1fXiv:1810009366 2018.. 2018.
Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating human behaviour with diffusion models. arXiv preprint arXiv:2301.10677 2023.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D diffusion. arXiv preprint vrXiv:2209.149966 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. beore-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020b.
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-hased diffusion models. Advances in neural information processing systems, 34:1415-1428, 2021.
Zhendong Wang. Jonathan Hunt, and Mingyuan Zhou. Diffusson policies as an expressive policy class for offline reinforcement learning. arXiv preprint 120.i:::208.069990 2022a.
Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion- GAN: Training GANs with diffusion. arXiv preprint arXiv:2206. 022662, 2022b.
Zhengyi Wang, Cheng Lu. Yikai Wang. Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifieDreamer: High-fidelity and diverse text-to-3D generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024.
Marco A Wiering and Martijn Van Otterlo. Reinforcement learning. Adaptation, learning, and optimization, 12(3):729, 2012.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1361, 2019.
Shentao Yang, Zhendong Wang, Huangjie Zheng, Yihao Feng, and Mingyuan Zhou. A behavior regularized implicit policy for offline reinforcement learning. arXiv preprint arXiv:2202.09673, 2022.
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. arXiv preprint arXiv:2311.18828, 2023,
12

---

Yuguang Yue, Zhendong Wang, and Mingyuan Zhou. Implicit distributional reinforcement learning. Advances in Neural Information Processing Systems. 33:7135-7147, 2020.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. arXiv preprint arXiv:1609.033 2016.
Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang. Score identity
distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In Forty-first International Conference Machine Learning, 2024. URL htt.ttttt//:ppreetttie net:fforum!dddhhqggeeekgx
13

---

Supplementary Material: Diffusion Policies Creating a Trust Region for Offline Reinforcement Learning
A Related Work
Expressive Generative Models for Behavior Cloning Behavior cloning refers to the task of learning the behavior policy that was used to collect static datasets. Generative models often employed for behavior cloning due to their expressive power. For instance, EMaQ (Ghasemipour
al. [021]] uses auto-regressive model for behavior cloning. BCQ [Fujimoto al. 2019 utilizes
a Conditional Variational Autoencoder (VAE), while Florence et al. [2022] employ energy- basedd
models. GAN-Joint Yang al.)
2022
leverages GANs, and several studies (Wang et al. 2022a
Janner et al., 2022 Pearce et al.
2023 utilize diffusion models for behavior cloning. Diffusion
models have demonstrated strong performance due their ability to capture multimodal distributions. However, they may suffer from increased training and inference times because of the iterative denoising process required for sampling.
Efficiency Improvement in Inffusion-Based RL Methods. Several studies aim to accelerate the training of diffusion models in offline RL settings. One approach involves using specialized diffusion ODE solvers, such as the DDIM solver [Song et al. 2020a] or the DPM-solver (Lu et al. 2022], speed up iterative sampling. Another strategy is avoid iterative denoising during training or inference. EDP [Kang et al., 2024) and IDQI. [Hansen-Estes Estruch et al. 2023] both focus on avoiding iterative sampling during training. EDP adopts approximate diffusion sampling scheme to minimize the required sampling steps, although it still requires iterative denoising during inference. IDQL accelerates the training process by only training a behavior cloning policy without denoising sampling. However, it requires iterative sampling during inference by selecting from a batch of candidate generated actions. SRPO [Chen et al., 2023 employs score distillation methods avoid iterative denoising in both training and inference.
Distillation Methods. Distillation methods for diffusion models have been proposed enable one-step generation of images or 3D objects. Examples of such methods include SDS [Poole et al.) 2022). VSD [Wang al. 2024). Diff' Instruct [Luo et al. 2024], and DMD (Yin et 2023 The core idea of these methods is to minimize the KL divergence between pre-trained diffusion model and target one-step generation model. SiD [Zhou al. 2024 uses a different divergence metric but shares the same goal of mimicking the distribution learned by pre-trained diffusion model. The distillation strategy can also be applied in the offline RL field to accelerate training and inference. However, directly adopting these methods may result in suboptimal performance.
B Diffusion Schedule
This diffusion training schedule is the same for training the behavior-cloning policy in Equation and the diffusion trust region loss in Equation
Noise Schedule We illustrate the EDM diffusion training schedule in setting. First, need to define some prespecified parameters: daata = 0.5, miin = 0.002, Fmax = 80. The noise schedule is defined by a; a;a o,E, where ~ No, I). We set a 1 and a, t. The variable log(t) follows logistic distribution with location parameter log dana and scale parameter 0.5. The original EDM paper samples log(t) from (12.,2, 1.2?), but this difference does not significantly affect our algorithm.
Denoiser
The denoiser p. is defined as:
Ma (a,, t|s) cakip(a)at + cour Fou(())F(Cnn( (o) Choiss (o))s)
14

---

where a = = and F represents the raw neural network layer. We also define:
0 (ou
data
Co)p(()
0² + dana"
0
data
(n((r = 0² + Jana
Caoise (0) = log(or).
Weight Schedule The final loss is given by:
2
Ba (an) Fo((m(() (a ( +), Caoise (o)(s) Cou( (a - Cskip(a) (a + E))
where \(o) "(((a))
C Details in KL Behavior Regularization
Here we introduce how we implement KL divergence regularization. The idea similar to previous
KL-based distillation methods [Wang et al. 2024 Luo et al. 2024 Yin et al. 2023) but adapted to our setting. Our loss function is defined as:
Plake (agls) log
(KL(O) DxL (=)_((((1s))lla(((s)) Re
(12)
Preal aa (s)
The gradient of (0) ) given by:
ooCKI. = (... (asls) - Srea(aul))) VoTo]
where Steal (real (a Score-ODE given in [Song et al. 2020b), we can estimate &real Srea( (a.) 8) and (aol)) with diffusion model. Let a0,, ag dre, the real score can be estimated by:
as log Preal a. s) and $take ()) = V Vasheta log Prake (agle(aglee ())s)) By using the
Sreal (10,t; (/s) 0g,t Creps (aa,, ((8)
of
where Hs the pre-trained diffusion behavior cloning model that learns the true data distribution. Similarly, we can estimate the fake score by:
a0.t aate t|s)
Stake (ag . (|s)
of
where μe is trained using fake data:
C(E) = (as,; t|s) ag 112
which is trained with generated fake action data.
Thus, the gradient of (f) can be expressed as: VoLki = E 00.090 [i (Sane t|ss Sreal (as.1. t|s)) Vomol
where W. [I)((a..))= and A is the dimension of the action space.
The algorithm for KL regularization is shown below:
D Implementation Details
Diffusion Policy We build our policy as an MLP-based conditional diffusion model. The model itself is an action prediction model. We model μą and bl as 4-layer MLPs with Mish activations, using 256 hidden units for all networks. The input to Má and luş is the concatenation of the noisy action vector, the current state vector, and the sinusoidal positional embedding of timestep L. The output of μs and μS is the predicted action at diffusion timestep t.
15

---

Algorithm 2 KL Regularization
Initialize policy network Ta. Mig
for each iteration do
Sample transition mini-batch B = [(8,), a, rt, 8(41)) ~ D
Diffusion Policy Learning: Update ue by (
end for
Initialize policy and fake score network: 0 for each iteration do
{ +
Sample transition mini-batch
{(1,, ai rt. 8(+1]} D, generate ag
Random timestep and add noise: Choose t, as a,00 0rE
with_ sii_no_gge gro_gr))():
cudd_real_actione ( (
Seighlling_fcllnn = abs(ag pred_ meddrrel_aaee reaa_aalllon) (eaeeeeiin grad fate_
_cction
meightiing_aaetn
loss 0.5 x mse_loss(an) stopgrad( - grad))
Update by loss
Diffusion Fake Policy Learning: Update μe by L(E)
end for
Q and V Networks We build two networks and V network with the same MLP setting diffusion policy, Each network comprises 4-layer MLPs with Mish activations and 256 hidden units.
Stochastic Max Q Trick Similar DQL Wang et al.) [2022a), during inference, generate N candidate actions and then randomly select an action according to exp(Q(a,, s)). Here, is fixed at 1024 and remains unchanged across different tasks.
One-Step Policy We build Gaussian policy using 3-layer MLPs with ReLU activations, utilizing 256 hidden units. After sampling action, we apply tanh activation ensure the action lies between [-1, 1]. an implicit policy is instantiated, structure is the same as that the diffusion policy.
pretrain the diffusion policy μs and the Q function Q, for 50 epochs to ensure they can better guide uy- Then, Hex Q,, and my are concurrently trained for the epochs specified in Table We found that introducing a pretrain schedule does significantly influence the final performance. Our ablation study on the Gym Medium Task revealed that while pretraining yields slightly better results, the final rewards are largely similar. Therefore, maintain a 50-epoch pretrain for all our tasks. The results are shown in Table 3
Pretrain In our implementation,
Table 3: The performance with and without pretraining D4RL Gym tasks. Environment
Pretrain
No Pretrain
malfcheetah-meedinne 57.9
57.5
hopper-medium-v2
99.6
87.6
89.4
88.7
16

---

E
Hyperparamaters
Table Hyperparameters for D4RL benchmarks. One epoch represents 1k steps, and the optimizer
used is Adam.
Gym
Term Term
Pretrain Epochs
'Training Epochs
Learning Rate
Lr decay
auffchectah-mediimeg
0.7
False
50
1000
3 10-
False
5
0.7
False
50
1000
3 10
False
malfchectal-mediinee mediuu--xxpert-2
False
1000
3 10
False
hopper-mediuee meneeer-edium-222
Fulse
1000
10
True
hopper-medium-repleee
0.7
False
50
1000
3 x 10
False
20
0.7
False
50
1000
3 x 10
False
wdlker2d medium-Ve v2
False
1000
10
True
walker2d-mediume -ellueel-mm
False
1000
1
True
0.7
False
50
1000
3 x 10
True
umaze-vl
True
50
500
x 10
False
untmaze-umizze-dieee -iverssee--
0.9
True
50
500
3 10
True
antmaze medium--diimee medium-plaa---o play-vO
True
400
3 10-4
False
antmave ceduuu--mdiium-iiieee -ivees-s-
10-4
False
latmaze-lrgge-paa-xx)) antmaz-large--divree--000
True
350
10
False
0.5
0.9
True
50
300
3 x 10
False
untmaaze-umizz-e22
0.9
True
50
500
3 10-1
False
antmaaz--mazzediivveeee.
0.9
True
500
3 x 10-6
True
autmaaz-maadduum-paay-222
True
3 10
False
cummaaz--cdiin--diieee-----
True
500
10
False
antmaze-lrge--llyy-222
0.9
True
50
500
x 10
False
sutmaze-laree-divrrs--
0.5
0.9
True
50
500
3 x 10
False
pen huhan-vl
1500
0.9
False
50
300
3 x 10-5
True
pen-cloned-vl
1500
False
200
1
False
kitchen-con -orelen-ommleee--
200
False
500
10
Thue
kitchen
sitchen-partal--x partial-vO
1x100"
kitchen-mixed-vl
200
0.7
False
500
1
True
17

---

Additional Experiments
F1 Complete 2D Toy Experiments
We also conducted some 2D bandit experiments with different reward scenarios. In Figure red points are generated by the one-step policy TT.-
In the first column, where the four corners have the same high reward, LKL tends to encourage exploration of all these high-reward regions, resulting in some suboptimal reward actions. In contrast, LTR generates actions that randomly select one of the high-reward regions, thereby avoiding suboptimal actions. The same situation occurs in the fourth and fifth columns of Figure ( where LKL. covers some suboptimal regions while LIR adheres closely to the highest reward regions.
However, when the data have only one mode with the highest reward, such in the second and third columns Figure both LKI. and LTR guide the policy to generate high-reward actions.
Trust Rogion
Poneee Coreen Valua
Low
Figure 6: 2D Bandit toy examples, where the behavior regularization is conducted by LTR and LRL. in different behavior data and reward scenarios. The first row uses behavior regularization by LTR. and the second row uses LKL. Yellow indicates the highest reward, and dark blue indicates the lowest reward.
F.2 Comparison with KL behavior Regularization in Gym Tasks
addition to testing on 2D bandit scenarios, we also evaluated the performance of two losses LKI. and LIR on the Mujoco Gym Medium task. The behavior regularization loss LIR (0) consistently outperformed CkL (0) in terms of achieving higher rewards. The results are presented in Table and the training curves are depicted in Figure
Table 5: The performance of LTR (f) and (()(0) D4RL Gym tasks. Results correspond to the mean of normalized scores over 50 random rollouts (5 independently trained models and 10 trajectories per model).
Environment
(0)
Lx)(O)
halfcheetah mediuut--a--miiu--222
57.9
24.1
hopper-medium--22 99.6
15.0
walker2d-medium-wy
89.4
3.4
F.3 Comparison with SRPO on Antmaze-v2 Datasets
Since SRPO uses Antmaze-v2 their D4RL benchmarks, we also conducted experiments on Antmaze-v2 using our algorithm, with the same hyperparameters as those used in Antmaze-v0 but with more training epochs. Hyperparameters details can found in Table The results for Antmaze-v2 v2 from SRPO are taken directly from their paper.
18

---

halffcheetah-mediuma..
hopper-medium--2
walkker2d-meduu--22
Average Egisodic N-Rwward
Avaraga Epizodic N-Roward
Avaraga Epioodic N-Roward
Method
DTQL
Trained Epochs
Trained Epochs
Trained Epochs
Figure 8: Training curves comparing policy learning with diffusion trust region loss and KL loss across three Gym medium tasks demonstrate that diffusion trust region regularization (DTQL) consistently outperforms KL-based behavior regularization policy learning.
The results for Antmaze-v2 are shown in Table 6
Our observations indicate that, on average, our method achieves higher score and exhibits significant performance improvements in complex Antmaze tasks, such as antmaze-medium diveen- antmaze-large-ploe and autmace-large-divess
Table 6: The performance of Our methods and SOTA baselines on D4RL AntMaze-v2 tasks. Results for DTQL correspond to the mean and standard errors of normalized scores over 500 random rollouts.
Antmaze
SRPO
Ours
antmaze-umaze-v2
97.1
92.6 1.24
antmaaz-uumre--liveees--->
82.1
74.4+
autmaze-mediium-plae
80.7
7611.91
mannazze-meiium-diveeesee
75.0
80.611.77
antmaze-Harge-play-22
53.6
59.2+ 19
autmaze-arge--dieeree->
53.6
62:12.17
Average
73.6
74.1
19

---

F.4 Overall Training and Inference Time
In Table we show the total training and inference wall time recorded on RTX-A5000 GPU servers, which include all training epochs specified in Table and the entire evaluation process. For evaluation, we test 10 trajectories for gym tasks and 100 trajectories for all other tasks.
Table 7: Total training and inference wall time for D4RL benchmarks
Tasks
Overall Training and Inference Time
Training Epochs
halfcheetah- medium--y2
medium-replay-v22
5.1h
1000
halfchestah- medium-expert-v2
5.5h
1000
hopper-medium--v2
5.0h
1000
pepper-mddium-eplay-y
5.4h
1000
seoper--edium-experee
malker2d-mediun--
4.9h
1000
walker2d
malker2--edduu--opllaaa medium-replay--2
4.9h
1000
walker2d medium expert--v2
4.9h
1000
untmize-umizo-v0
3.3h
500
aatmaaz-umazz-divereeeee
4.0h
500
untmize-medium-plly--o play--00ppppp-
3.1h
400
antmaz-mndduum-diveeeee
3,2h
400
aatmaze-laree-paa-x00
2.3h
350
mamuzz-large-divers--000
2,6h
300
untmize-umaze-v2
3,3h
500
antmaze diverse-v2
3.1h
500
uatmaz--mddun--diereeyy
3.1h
500
latmaze-large-play-22
3.3h
500
untmuzelarrg-ddivres--22
3.3h
500
pen-luman-vl
300
cloned-v1
0.6h
200
kitchen-complete-x0
3.0h
500
kicchen-patial-v0
1000
kicchen-mixed-v0
3.0h
500
20

---

G
Training Curves
hallchhotaa--mdddum-22
noopper-medium-22
walker2ddmaddium22
Avarage Epioddo N N-Ravard
Avaragg Epioode E- N-Roerr N-Raward
Averrage Epoodd N Revard
Trained Epochs
Trained Epochs
Trained Epochs
haffcheetah-meddin--reeee
hopper-medium--ellay--
walleer2d-meddiu--repayyyy
Avarage Eppodd N-Rwwad
Avaraga Eppodd N-Roward
Average Eppcdd N Reward
Trained Epochs
Trained Epochs
Trained Epochs
aufichesta-mmedium-exppere
hogpermmedin--xxpeet//
walker/z-meddum-experwt
Averag Eptoodic M-Rwward
Averge Eptoodic N-Roward
NuRowwrdd
Trained Epochs
Trained Epochs
Trained Epochs
iilimazz-umaaz-divvereee
anlmaze-umazo-vo
arimmaze-medium-play--
Avenge Epledd M-Rowardd
Avenga Eptode N-Raward
Average Eploade N-Rowardd
Trained Epochs
Trained Epochs
Trained Epochs
mntmazze-medium--iivess-vv
anlmaze-largo-plaa--0
anrmaaze-argg-diivrre--0
Average Eptosii No Rovoordd
Aveeage Episoric N- Saeward
Average Epioocioo RuRowwerdd
Trained Epochs
Trained Epochs
Trained Epochs
Figure 9: Training curves. Rewards evaluated after every 50 epochs.
21

---

antmazze-uaaze-22
antraaze-umaze-divvrssv/
ariimaze-mmedium-plaa--
Avarage Epsoote N. H-Reward
Average (--pwwrrd
Average Epccoic N Rewarr
Trained
Trained Epoche
Trained
anttnaze-meddumddirre----
antmaze-large-pless
antmaze-large-divereew
Avorage Episdds N.owaad
Howward
Avsrace Epoodde NRwward
Trained
kitchen-complete-v0
kitchen-partial-v0
kitchen-mixed-v00
Averge Eppede N.oward
Average Epkedoo N Roward
Average 6
Epochs
Trained Epochs
100
Trained Epochs
pen-human--v1
pen-cloned-vv
Average Eptoci N- Rewardd
Eptacil No
Trained Epochs
Trained Epochs
Figure 10: Training curves. Rewards evaluated after every 50 epochs.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our contribution is accurately reflected in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: has been discuss Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

Justification:
All theorems are proven in main text or by reference.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The algorithms are given in Algorithm and Appendix Implement details
are discuss in Appendix D and hyperparameters discussed in
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification: The code has already been published in https://githut com/
Tlanyucoding//Diffuue wtt.g////ifffuiooonnoooo Ttt.uusi.ooruuudddd_eee
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
wttpgg/dddd//oeuuuunnnnn ssicnpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Expriments details are all contained Appendices to Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Standard error is provided in Table ! and also plotted in training curves, Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computer resources have been discussed in Section 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http::
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
No societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We are using simulation dataset D4RL which has no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We using D4RL datasets and it is explicitly cited.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: The primary new asset is the code of our algorithm. It is not published yet, but will be made public once the paper is accepted.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No human subjects involved in this research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "bL3QNuZ8iK", "reviewer": "Reviewer_R2od", "summary": "This paper introduces DTQL, a novel offline reinforcement learning (offline RL) method. With a newly introduced diffusion trust region loss, DTQL constrained policy update within a predefined trust region near a diffusion policy trained by behavior cloning (BC). Through the empirical experiments, DTQL demonstrated improved performance against BC and offline RL baselines, including recent baselines that utilize diffusion models in offline RL settings.", "questions": "1. The reverse KL objective should also be mode-seeking instead of mode-averaging. I do not quite understand why the KL regularization does not show this property in the example provided in Figure 3. As in the SRPO[1], a similar illustration was shown in Figure 5. Could the author provide more insights on what caused this difference in behavior?\n\n2. When DTQL only utilizes four layers of MLP and SRPO requires several resnet blocks, why is SRPO inference faster than DTQL?\n\n[1] Chen H, Lu C, Wang Z, Su H, Zhu J. Score regularized policy optimization through diffusion behavior. arXiv preprint arXiv:2310.07297. 2023 Oct 11.", "limitations": "The limitations were discussed in the conclusion sections, along with possible future works.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The diffusion-based trust region loss looks novel and interesting.\n\n2. The visualization of trust region loss and the toy-tasks provide an intuitive explanation of the proposed algorithm.\n\n3. The empirical experiments show improved performance compared to standard offline RL methods and faster inference time than most offline RL with diffusions. The reviewer appreciates that the author also shows the results where DTQL performs worse than IDQL (among the Antmaze tasks), which provides a more comprehensive view of the algorithm's performance.\n\n4. The paper is easy to follow, and the connections and differences to related works are clearly outlined.", "weaknesses": "1. The performance improvements compared with DQL, IDQL, and SRPO are not very significant; in some experiments (Antmaze), IDQL outperformed the proposed algorithms.\n\n2. The ablation studies only show one seed result, which reduces the results' plausibility, especially for Figure 5 (a), where the learning curves are quite noisy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's recognition of novelty, presentation and performance improvement of our paper. Below, we provide detailed clarifications and answers to solve the remaining concerns.\n\n### Responses to Weakness 1.\n\nWe first want to emphasize that one major contribution of DTQL is for speeding up Diffusion-QL. We proposed a dual policy training approach and achieved dramatically improvement in terms of both training and inference efficiency. \n\nFor D4RL benchmark performance, we provide more clarifications below. \n1. Compared with DQL, our major baseline, our method demonstrates dramtically speeding up, superior average score and greater stability. We expand the explanation for stability here. As shown in Figure 8 of [2], the training curve of DQL is less stable due to the challenges in computing gradients of the Q-value function while backpropagating through all diffusion timesteps, which may result in a vanishing gradient problem. In contrast, we propose a dual policy trainig approach, and train a one-step Gaussian policy with trust region defined by a diffusion policy. This avoids the gradient vanishing issue and mitigates training instability, as shown in the learning curve in Appendix H. \n2. DTQL outperforms IDQL in Gym and slightly underperforms it in Antmaze. However, IDQL is **10x** slower than us in inference. \n3. Regarding SRPO, our method not only scores higher on Gym tasks but also surpasses SRPO on Antmaze-v0 as shown in Table 1 (from 30.1 to 73.6) and Antmaze-v2 as shown in Appendix G.3.\n\n[2] Lu, C., Chen, H., Chen, J., Su, H., Li, C., \\& Zhu, J. (2023, July). Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning (pp. 22825-22855). PMLR.\n\n### Responses to Weakness 2.\n\nThank you for your suggestion. We have updated the ablation study to include results across 5 random seeds in the PDF of global response, along with the standard error. We would like to make the following clarifications:\n1. The blue line in Figure 5 (a), only used for ablation study, is quite noisy because it does not include the entropy term. After adding the entropy term, the orange line for the reward curve of the proposed algorithm becomes much more stable.\n2. We acknowledge that the orange line in the antmaze-large-diverse-v0 appears noisy. However, this level of variations is common in the offline RL setting and is, in fact, found to be more stable than DQL.\n\n### Responses to Question 1.\n\nThanks for giving us opportunity to explain about this, and we will also make more discussion about it in the revision. In sum, KL regularization in Figure 3 is not the same as reverse KL used in SRPO. In Figure 3, we are using the loss from Diff-Instruct [3], Variational Score Distillation [4]  and Distribution Score Matching [5]. The detailed algorithm is described in Appendix D. An analysis of the gradients shows distinct differences between the Score Distillation Sampling (SDS) loss in SRPO [1] and our loss:\n\n- As discussed beginning at line 220, for SRPO [1], we have:\n$$\n\\nabla_\\theta L_{\\text{SDS}} =E_{t,s,\\epsilon}\\left[w(t)(\\epsilon_\\phi(z_t,t|s) - \\epsilon)\\frac{\\partial z_t}{\\partial \\theta}\\right]\n$$\n - In contrast, the loss we used in Figure 3 is:\n$$\n\\nabla_\\theta L_{\\text{KL}} = E_{t,s,\\epsilon}\\left[w(t)(\\epsilon_\\phi(z_t,t|s) - \\epsilon_{\\text{fake}}(z_t,t|s))\\frac{\\partial z_t}{\\partial \\theta}\\right]\n$$\nwhere $\\epsilon_{\\text{fake}}$ is the score function we learned from the current policy. Thus, the loss we used in Figure 3 provides an updating direction that can be rather “fine” and “sharp” due to the difference between pretrained $\\epsilon_\\phi$ and $\\epsilon_{\\text{fake}}$. This approach is encouraging for covering more modes (as shown in Figure 2) and differs from the reverse KL loss (SDS loss) in SRPO. A more detailed comparison has been discussed in [4].\n\n[3] Luo, W., Hu, T., Zhang, S., Sun, J., Li, Z., \\& Zhang, Z. (2024). Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36.\n\n[4] Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., \\& Zhu, J. (2024). Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36.\n\n[5] Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., \\& Park, T. (2024). One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6613-6623).\n\n\n\n### Responses to Question 2.\n\nThank you for your observation. We will clarify this point in the revision. \n1. SRPO uses a deterministic policy, whereas our method employs a stochastic policy. Consequently, after the network forward pass, we need to resample to generate an action, which adds to the inference time.\n2. Additionally, we utilize the (stochastic) max Q trick in inference, as in DQL, which involves generating $N$ candidate actions for a given state and then selecting an action randomly with a weight proportional to $\\exp{Q(s,a)}$. \n\nTherefore, these two factors—stochastic resampling and max Q —contribute to making our inference slightly slower than the deterministic policy used in SRPO. We will provide a more detailed explanation of this point in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e9jW6wiHnY", "reviewer": "Reviewer_Px6z", "summary": "The paper introduces Diffusion Trusted Q-Learning (DTQL), a novel approach for offline reinforcement learning that enhances performance and efficiency. DTQL utilizes a dual policy framework to bridge the gap between diffusion policies and one-step policies, improving expressiveness without the need for iterative denoising sampling. By incorporating a diffusion trust region loss, DTQL ensures stability and effectiveness in training. The method outperforms traditional distillation methods in various scenarios, offering a promising solution for offline RL challenges.", "questions": "* How should the term \"trust region\" in the article's title and method name be understood?\n\n* How is the weight function $w(t)$ in Equation 4 set in this paper, and why is it configured in this manner? A detailed discussion is needed.\n\n* How does DTQL eliminate the need for iterative denoising sampling during both training and inference, making it computationally efficient?", "limitations": "* The writing of this paper has issues—there is a lack of coherence between the motivation, method, and experiments. Section 2 provides insufficient detail on the proposed method and does not effectively explain why it improves computational efficiency. Additionally, Section 3 dedicates too much space to the comparison experiments replacing $\\mathcal{L}_{\\mathcal{TR}}$, lacking a comparative analysis with the baseline method DQL.\n\n* The method presented in this paper is merely a simple combination of previous works [1] and [2], which is lack of novelty. \n\n[1] Wang, Zhendong, Jonathan J. Hunt, and Mingyuan Zhou. \"Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning.\" The Eleventh International Conference on Learning Representations.\n\n[2 ] Kingma, Diederik, and Ruiqi Gao. \"Understanding diffusion objectives as the elbo with simple data augmentation.\" Advances in Neural Information Processing Systems 36 (2024).", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* To my knowledge this paper presents a new approach to make Diffusion policy computationally efficient compared with previous method.\n\n* Based on the experimental results, it is evident that DTQL outperforms the previous method in terms of performance.", "weaknesses": "* This is an incremental work that relies on Diffusion Q-learning (DQL) and adds a weighted loss to the diffusion model objectives(Equation 4) to accelerate model inference, without clearly explaining the necessity of this combination.\n\n* The explanation of the proposed method in Section 2 is too brief, with details only provided in subsection 2.3. Additionally, subsection 2.3 does not clearly explain the meaning of Trust Region in Equation 4. The subsequent Theorem and Remark do not effectively justify the necessity of using this update formula. The overall logic lacks coherence.\n\n* The introduction of other methods for accelerating training and inference in diffusion-based policy learning in Section 3 is overly lengthy and lacks relevance to the methods used in this paper. It is recommended to move most of this content to the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We appreciate the time and effort you have dedicated to evaluating our work. It seems there may have been some misunderstandings regarding the main points of our paper. We will do our best to clarify these issues and hope you will consider re-evaluating our work.\n\n### Responses to Weakness 1,2,3\n\nWe try to address the reviewer's concerns in the **Global Response**, where we introduce the main goal, contribution, and presentation logic of our paper. We believe this global response adequately addresses the reviewer's concerns regarding the weaknesses. If the reviewer still has any questions about these aspects, we are happy to discuss them further.\n\n### Response to Question 1\n\n**1.From Toy Example:** Let us illustrate this with toy examples. In Figure 2, the first column shows the offline action data distribution, and the second column shows the trust region loss for different actions. We can observe that actions lying in the high-density region of the raw data manifold have less trust region loss. This means that actions similar to in-sample data are trusted and have lower loss. In contrast, actions outside the in-sample data manifold, as shown in the second and third columns, have large trust region loss. This visualization demonstrates that for a given fixed diffusion model $\\mu_\\phi$, the loss $E[||\\mu_\\phi(a_\\theta+\\sigma_t\\epsilon,t|s)-a_\\theta||^2]$ can be used to check whether the action $a_\\theta$ generated by the one-step policy $\\pi_\\theta$ is similar to or different from in-sample actions. If the generated data is similar to in-sample data, it is trusted and has a lower trust region loss.\n\n**2.From Theoretical Perspective:** From Theorem 1, the trust region can be defined by the conditional log likelihood. Specifically, for any given state $s$, a trust region of action is a set {$a|\\log p(a|s)\\geq\\text{threshold}$} where the conditional log likelihood is approximated by conditional diffusion loss (Eq 4). The magnitude of the threshold can be tuned by the hyperparameter $\\alpha$ during the optimization of the final loss (Eq. 5).\n\n**3.Further Discussion:** The discussion about the \"trust region\" is also covered from Line 117 to Line 122. If the reviewer has any further questions about understanding this loss, we are happy to discuss it in more detail.\n\n\n\n### Response to Question 2\n\nAs our goal is not to design a new weight schedule for the diffusion model, we are using the state-of-the-art diffusion weight schedule, EDM [3], which is discussed in Section 4.3 and Appendix C for completeness. However, other diffusion training schedules, such as VP and VE [4], can also be accommodated within our algorithm framework.\n\n[3] Karras, T., Aittala, M., Aila, T., \\& Laine, S. (2022). Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35, 26565-26577.\n\n[4] Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., \\& Poole, B. (2020). Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456.\n\n### Response to Question 3\n\nIn brief, we do not use a diffusion policy to generate actions, thereby avoiding iterative denoising. A comprehensive explanation of this point is discussed in the **global response**. If the reviewer has any further questions about this point, we are happy to discuss them.\n\n### Response to Limitation 1\n\nWe are confident that by considering the **global response** and gaining a more comprehensive understanding of our paper, the logic and presentation will become clearer and more accessible for readers. We kindly request that the reviewer re-evaluate our paper in light of these clarifications.\n\n### Response to Limitation 2\n\nAs discussed in the **global response**, our paper is not the same as DQL [1]. On the contrary, we introduce a completely different training scheme to address the drawbacks in DQL. Additionally, [2] is used to support Theorem 1, and we believe that our use of this reference does not diminish the novelty of our new offline RL method design in any way.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m7zcdpySew", "reviewer": "Reviewer_Rh1M", "summary": "This paper introduces Diffusion Trusted Q-Learning (DTQL) for offline reinforcement learning. DTQL employs a dual-policy representation, a diffusion policy trained by behaviour cloning, and a one-step Gaussian policy trained by RL and distilling the diffusion model. Specifically, DTQL introduces a diffusion trust region loss for the distillation. Such a scheme allows both one-step generation, and achieves strong performance on the D4RL benchmark.", "questions": "1/ It seems to me from Fig. 2 that this approach encourages interpolation of the distribution. Have you tried to design tasks to validate such behaviours?\n\n2/ It is a bit unclear how to guarantee the distilled Gaussian policy captures all relevant information of the diffusion process, as its policy class naturally limits the expressiveness. Could you provide more explanations for this?", "limitations": "The experiments are done only on simple D4RL benchmarks. The policy distribution is quite simple and tasks are relatively short horizon. As a result, I’m not fully convinced when applied to more challenging scenarios, distilling the diffusion policy into a Gaussian with trust-region loss can still work given more complex action distributions.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1/ The proposed method is interesting.\n\n2/ The proposed DTQL is well-motivated. It addresses one of the big issues of diffusion policy, i.e., the multi-step inference, through policy distillation.\n\n3/ The paper has conducted solid experiments, for both intuitive understanding and empirical results on benchmarks.", "weaknesses": "Overall I don’t see many weak points of this work.\n\n1/ The distilled policy captures some certain modes, but is actually not multi-modal. It also missed certain modes as in Fig. 2, which is inconsistent with what the authors have claimed in the caption. I would suggest the authors to be careful and avoid over-claiming.\n\n2/ The writing has certain room for improvement. The introduction could be reorganised for better clarity. There are some typos, e.g., in Line 183, “instantiates”, and inconsistent wording, e.g., “cooperative policy” and “dual policy”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's recognition of our method's novelty, motivation, and empirical performance. Thank you for the careful review. \n\n### Responses to Weakness 1.\n\nThank you for the good suggestion. You are absolutely right that some modes are also missed by $\\mathcal{L}_{\\text{KL}}$. We will modify the caption accordingly in the revision to accurately reflect this.\n\n### Responses to Weakness 2.\n\nThank you for pointing this out and for your valuable suggestions. We will continue polishing our paper to enhance clarity. We apologize for the oversight in Line 183 with the word \"instantiates.\" We will correct this and thoroughly review the entire paper to eliminate any other typographical errors. Additionally, we will correct the inconsistent wording and change \"cooperative policy\" to \"dual policy\" to provide clearer communication to the readers.\n\n\n### Responses to Question 1.\n\nThank you for the insightful observation. We hypothesize that this behavior is due to neural networks' tendency to learn continuous functions, which can lead to the connection of some modes with artificial dots. This phenomenon also occurs when using diffusion models to mimic distributions with isolated density modes. For example, in Figure 2 of DQL [1], there are similar artifacts in the interpolation of modes when we use pure diffusion policy to do behaviour cloning.\n\nYes, we have also designed an RL algorithm based on $\\mathcal{L}_{\\text{KL}}$, as shown in Figure 3 and Appendix G.2. We suspect that interpolation may cause some out-of-distribution action generation, which could explain the performance drop observed in the learning curve in Figure 8.\n\n[1] Wang, Z., Hunt, J. J., \\& Zhou, M. (2022). Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193.\n\n### Responses to Question 2.\n\nThank you for the question. We are happy to make some clarification on this point.\n\n1. We don't expect the Gaussian policy to capture all the information from the diffusion model due to the unimodal nature of Gaussian policies. However, the Gaussian policy is sufficient to capture the optimal mode, which has the highest Q value and is usually unimodal. For example, in Figure 2 of DQL [1], even though the diffusion policy initially captures all behavior modes, it eventually focuses only on one optimal mode after policy improvement (guided by Q value). Thus, we believe that in an offline RL setting, the Gaussian policy is able to capture one optimal behavior mode when constrained by the diffusion policy.\n\n2. We chose the Gaussian policy because it is widely used in RL settings, such as in IQL [2] and SAC [3], making it a good starting point to demonstrate the performance of our algorithm. However, our algorithm can accommodate any one-step policy. One possible extension is to use an implicit policy parameterized by a neural network to enhance the expressiveness of the policy and cover multimodal distributions if necessary.\n\n[2] Kostrikov, I., Nair, A., \\& Levine, S. (2021). Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169.\n\n[3] Haarnoja, T., Zhou, A., Abbeel, P., \\& Levine, S. (2018, July). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning (pp. 1861-1870). PMLR.\n\n### Responses to Limitations.\n\nThanks for the discussion. We evaluate our algorithm using widely used benchmarks for fair comparison with other offline RL algorithms. We acknowledge that the Gaussian policy may not have sufficient expressiveness for complex tasks. However, our algorithm framework can accommodate any one-step policy. Therefore, any policy, such as an implicit policy defined by passing random noise through a deep neural network, can potentially be used in our algorithm to handle more complex tasks. This represents an interesting future direction to explore how these diffusion-based RL algorithms perform in more complex settings.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mbYtyciNAl", "reviewer": "Reviewer_BozX", "summary": "The paper proposes a new diffusion-based training loss for offline reinforcement learning, in which the empirical actions from the offline datasets are replaced with the actions sampled from a parameterized policy. The paper calls this new loss “trust region loss” and uses it as one of the objectives to minimize in the diffusion policy learning. The paper points out the mode seeking behavior induced by the minimization of such “trust region loss” and compares it with forward KL-regularization behavior cloning. The paper also evaluates the methods on D4RL datasets with some offline RL methods.", "questions": "1. Can the authors explain which empirical results support that the assertion “diversity, while valued in image and 3D generation, is not essential in offline RL”? as speculated in Line 234 - 235. \n2. Can the authors explain “discourage out-of-distribution sampling” in line 107? \n3. What does it mean by “policy $\\pi_\\theta$ can move freely” in line 121?", "limitations": "n/a", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "### One-step policy\nIntroducing a one-step policy seems a promising way to address the challenges in the diffusion sampling. This idea is nicely motivated and well-justified. The discussion on the mode-seeking behaviors is interesting, especially with the empirical results as an extra evidence to support.", "weaknesses": "I found the paper shares great similarities to the SRPO paper, in terms of the idea, the formulation, and the presentations. The empirical results of the proposed method are also close to SRPO method. \nBeyond that, I have several other concerns:\n### **Unknown trust region**\n1. A bit surprisingly, **the paper gives no explicit definition of the crucial trust region used in Eq. (4)**, the diffusion-based trust region loss, even though trust region is the key for the whole paper. Particularly, the loss function defined by Eq. (4) comes directly from a simple modification of Eq. (3) by replacing the empirical action $a_0$ with the action generated by the policy $\\pi_\\theta$.  It is unclear what the trust region this simple modification induces. \n\n2. Regarding “the loss effectively creates a trust region defined by the diffusion-based behavior cloning policy, within which the one-step policy $\\pi_\\theta$ can move freely”, the paper does not explain how “to generate actions (data) that reside in the high-density region of the data manifold specified by $\\mu_\\theta$ through optimizing $\\theta$”. Note that $a_\\theta$ generated by $\\pi_\\theta$ will be fed into $\\mu_\\phi$. **So a trivial solution of this optimization will be $\\pi_\\theta(\\cdot|s) = a_0$, i.e., $\\pi_\\theta$ just returns $a_0$ for every $(a_0, s)\\sim D$ as in Eq. (2). Then loss in Eq. (4) is exactly the same as Eq. (3)**. So how does the optimization of Eq. (4) avoid such trivial solutions?  Especially when the paper says Eq. (4) is to encourage mode seeking behaviors. \n\n3. **the “one-step policy”, another important factor in the trust region, is introduced a bit inconsistently section 2.3 and 2.4**. I understand that Eq. (4) is to introduce a policy that seeks to model only one mode, i.e., one action only, from the training samples. But then line 133-137, “the generated action $a_\\theta$ appear similar to in-sample actions and penalizes those that differ”, which means the action should be similar to any of the training samples to avoid penalties. Also, this “mode-seeking” contradicts with the statement “explore freely” in line 149: how does this policy move freely? And how the penalization is triggered? In addition, the paper mentioned an implicit parameterization of $\\pi_\\theta$: $a_\\theta=\\pi_\\theta(s, \\epsilon$. For this implicit parameterization, what’s the different between optimization Eq. (4) and the simple behavior cloning? One even more inconsistent description is given in line 183 “one-step Implicit policy”: why $\\pi_\\theta$ here is instantiated implicitly? \n\n\n### A bit trivial discussion on mode seeking behaviors\n**I found the discussion on mode seeking behaviors in Section 3 a bit trivial and pretty incremental as these have been pointed by SRPO already in its section 2.2**. \n\n1. **the notations are not quite clear**: $p_{fake}$ and $p_{real}$ are not defined explicitly with respect to $\\pi_\\theta$ and $\\mu_\\phi$. \n\n2. the forward KL divergence is mode-covering in that sense it learns a distribution that tries to cover all modes, while reverse KL divergence is mode-seeking, behaving similarly to the optimization of $L_{TR}$. This discussion has been provided by SRPO paper its equations (2) and (3). So what’s new? More importantly, **since the paper focuses more on the mode seeking behavior, why does it not compare against the reverse KL divergence minimization?** \n\n3. **Theorem 1 tells very little about the mode seeking behavior of $\\pi_\\theta$ except the simple substitution of mode actions**. I don’t its importance in the training, especially the mode seeking behaviors. \n\n### Hard to understand\n**There are quite many sentences and descriptions with language issues or typos, which make the whole paper rather hard to understand**. Examples of language issues and typos: \n* Line 192-193: “KL divergence is employed in this context, it is designed to recover all modalities of the data distribution”\n* Line 199: “the loss ensures that the generated action .$a_\\theta$ lies within the in-sample datasets”\n* Line 288: “a Jacobian term” is different from what in Eq. (11). \n* Line 126, “minimize $L_{PB}$ in Equation 4”. There is no such $L_{PB}$ term. \n* Line 183, “is instantiates as an”\n\n**Importantly, the hyperparameters used in the experiments are a bit hard to understand**. Particularly, in the experiments, the paper mentioned that “the variation between datasets significantly impacts the algorithm performance”. Then why did the paper still “employed the official SRPO code on Antemze-v0 and maintained identical hyperparameters used for Antmaze-v2”? Why wouldn’t the paper just report the performance on both versions, separating DQL, IDQL from SRPO?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We believe there may have been some misunderstandings regarding our paper. We will strive to address your questions thoroughly and hope you will consider re-evaluating our work.\n\n### Similarities\n\nFirstly, we have discussed some differences between our method and SRPO in the **Global Response**. Here we are addressing the reviewer's concerns about \"presentation\" and \"empirical result\" similarity.\n\n**The presentation.** We believe every excellent work, like DQL, SRPO, and IDQL, shares a similar logic in presenting the story in the offline RL setting. The presentation of our work humbly learns from these papers. Thus our presentation flow is not only similar to SRPO but also to DQL and IDQL, as these papers serve as excellent examples for us.\n\n**For empirical results.** Our method slightly outperforms SRPO on Gym tasks, achieving different SOTA results on various sub-tasks. For instance, SRPO scores higher on walker2d-m-e (114 vs. 110) but lower on hopper-m-e (100 vs. 109). Additionally, our method surpasses SRPO on antmaze-v0 tasks and excels on some antmaze-v2 sub-tasks like antmaze-l-p and antmaze-m-d (**Table 6**).\n\n### Unknown trust region\n \n**1. Definition.** Formally, by Theorem 1, given any state $s$, we can define a trust region for action by the set {$a | \\log p(a|s) \\ge \\text{threshold}$}, where $\\log p(a|s)$ is approximated by the diffusion loss (Eq 4). The magnitude of the threshold can be tuned by the hyperparameter $\\alpha$ during the optimization of the final loss (Eq. 5). Thank you for pointing this out; we will make this point clearer in the revision.\n\n **2. Trivial solution.** For the reviewer's concerns about \"move freely\" and \"how to generate actions...\", we have tried to answer them in **Global Response**. For \"trivial solution\", such a trivial solution is only possible when dealing with discrete space control questions, and the offline datasets contain information for whole state spaces. However, for continuous control questions, such a trivial solution has a probability measure of 0 since both action and state spaces are continuous, and the size of the offline dataset is finite. Thus, this trivial solution does not apply to our setting. Moreover, even in the discrete case, the trivial solution you mentioned is essentially behavior cloning. When adding the Q function for guidance to improve the policy, $\\pi$ will only return the action $a$ with the highest Q value for any given state $s$.\n\n **3. Inconsistently.** We addressed the reviewer's concerns about \"inconsistent\" presentation in our global response section. Now, we aim to clarify \"why $\\pi_\\theta$ is instantiated implicitly\" in our approach. The reason for the implicit instantiation of $\\pi_\\theta$ in line 183 is due to the KL loss (Eq 8), which encourages mode seeking (Figure 2). The Gaussian policy has limited expressiveness; therefore, we make $\\pi_\\theta$ implicit to enhance its expressiveness and ability to seek more modes. It is important to note that Eq 8 is never used in our method; it is included only as a comparison baseline.\n\n\n### Mode seeking behaviors\n\n**1. Trivial.** The discussion in Section 3 is primarily to demonstrate that our method indeed exhibits mode-seeking behaviors. While SRPO discusses mode-seeking for its own method, our focus here is to discuss the mode-seeking behavior specific to our approach. Therefore, we believe this discussion is necessary and not trivial.\n\n**2. Notations.** Constrained by the length of the paper, we do not demonstrate all details about the algorithm in Section 3 since it is not our main contribution. However, we did mention in line 191 that more details are deferred to Appendix D. For full details, please refer to Variational Score Distillation [33], Diff-Instruct [23], and Distribution Matching Distillation [37]. These reference indices are consistent with those used in our paper.\n\n**3. Reverse KL divergence minimization.**  This was addressed by SRPO, and since our paper thoroughly compares with SRPO, we believe we have sufficiently addressed the comparison with reverse KL divergence.\n\n**4. Theorem 1.** Please refer to our global response.\n\n### Hard to understand\n\n**1. Typos.** Thank you for the careful review and valuable suggestions. We will incorporate these corrections in the revised manuscript.\n\n**2. Experiments.** We have contacted the authors of SRPO by email, and they acknowledged that there is a version mismatch between SRPO and other baselines in Antmaze v0 and v2. Since SRPO does not provide hyperparameters for Antmaze v0, we used the same hyperparameters as for v2 and reported them in the main table. Additionally, as illustrated in line 262 and in **Table 6**, we also compare our method with SRPO on **Antmaze v2**, where we use same hyperparameters as those in Antmaze v0 for our method. The SRPO results are taken from the original paper for a fair comparison. And separating table is a good idea. We can separate SRPO from the main table and create an additional table to compare results in the revision, which will only include Antmaze-v2 results if the reviewer believes it is better to present in this way.\n\n### Responses to Questions\n\nFor Questions 2 and 3, we believe they have been covered in the global response. For Question 1, in offline RL, typically after training and during interaction with the environment, for any given state, we want the agent to generate the action with the highest Q value. Therefore, diversity, in the sense of generating a batch of diverse actions for a given state, is not essential. As long as the action has a high Q value, the diversity of the generated actions does not contribute to increasing the final cumulative reward. A good example is the deterministic policy, which, for a given state, generates only one action. Although it lacks diversity, it can still achieve a high Q value.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their valuable feedback. We believe some aspects of our paper may have been misunderstood, particularly by **Reviewer Px6z regarding the goal, contribution, and logic of our paper**, and by **Reviewer BozX who questioned the differences with SRPO**. Below, we focus on addressing these two concerns.

## Goal, Contribution, and Logic

The goal of our paper is to accelerate training and inference for diffusion-policy-based offline RL methods, such as DQL, while also increasing training stability. Previous baselines, like DQL, generate actions through iterative denoising during both training and inference phases. This process cannot be computed in parallel, making training and inference time-consuming. Additionally, since actions are generated by iterative denoising, gradient computation requires backpropagation through all diffusion time steps. This can result in a vanishing gradient problem, undermining training stability.

Our paper aims to address these issues with two main objectives:

1. Leverage diffusion policy and enhance performance.

2. Avoid iterative denoising for action generation, thus accelerating training and inference and enhancing stability.

To this end, we propose a dual policy method, containing one diffusion policy and one one-step Gaussian policy. The diffusion policy is not used to generate actions directly, while the Gaussian policy generates actions for both training and inference. **This one-step policy eliminates the need for iterative denoising, thus speeding up training and inference.** Additionally, since actions are not generated through iterative denoising, the vanishing gradient issue is mitigated, resulting in more stable training.

We introduce a Trust Region loss (Eq. 4) to bridge the two policies. After performing pure behavior cloning, using actions from the dataset to train diffusion policy $\mu_\phi$, we generate actions from the one-step policy. We then reuse the diffusion loss calculated by $\mu_\phi$ to evaluate how much the generated actions (from the one-step policy) deviate from the existing dataset. Figure 1 demonstrates that if the generated action is similar to in-sample actions, the Trust Region loss is small; if the generated action deviates significantly from in-sample actions, the Trust Region loss is large. Thus, the Trust Region loss uses pretrained $\mu_\phi$ to measure whether the actions generated by the one-step policy $\pi$ are similar to or different from the in-sample data.

Then adding Q function, we derive the final loss in Eq. 5. Since the one-step policy $\pi$ is the actual policy interacting with the environment, and we only use $\mu_\phi$ to calculate the diffusion loss without reverse sampling, training and inference are accelerated. The first component in Eq. 5 regularizes the generated action to not deviate far from the dataset, and the second component maximizes the Q value. 

## Difference with SRPO

Some concerns about our method is similar to SRPO. While we aim to solve the same problem of diffusion policy as introduced above, our approach differs significantly from SRPO in terms of idea, loss formulation, loss explanation, theorem, and empirical results, as elaborated below:

### 1. Idea and loss formulation difference

We believe the intuition behind our loss $L_{\text{TR}}$ (Eq. 4) fundamentally differs from the reverse KL-based SRPO loss. The starting point and motivation of our paper is to reuse the diffusion loss as a regularization, which helps us determine whether the generated action is far from the in-sample dataset. Here, the diffusion policy acts more like a **detector** to detect out-of-distribution actions. In contrast, the motivation of SRPO is to use reverse KL to distill information from diffusion policies, where the diffusion policy serves more as a **distillation target** rather than a detector. The loss formulations are also evidently different. Let $z_t = \alpha_t a_\theta + \sigma_t \epsilon$ where $a_\theta \sim \pi_\theta$. The formulations of the losses for SRPO and our method are clearly different.:

$$
L_{\text{SRPO}} = E_{s,\epsilon,a_\theta\sim\pi_\theta}[\log\pi_\theta(a_\theta|s) - \log \mu_\phi(a_\theta|s)]
$$

$$
L_{\text{TR}} = E_{s,\epsilon,a_\theta\sim\pi_\theta}[||\mu_\phi(z_t|s) - a_\theta||_2^2]
$$


### 2. Theorem explanation difference

In addition to Theorem 1, which demonstrates that our loss encourages the generated action $a_\theta$ to lie in the high-density region of the offline data, we would like to further clarify the differences: 

- During the training of the diffusion model, we aim to optimize $\phi$ to minimize the negative log likelihood $E_{a\sim D}[-\log\mu_\phi(a|s)]$. However, we use a variational upper bound, defined as the diffusion training loss in Eq. 2, for tractable optimization (Theorem 1 and [1]). During the optimization of policy $\pi_\theta$, we adapt this diffusion training loss from Eq. 4 as our trusted region loss $L_{\text{TR}}$, which acts as a variational upper bound for the negative log likelihood $E_{a\sim \pi_{\theta}}[-\log\mu_\phi(a_{\theta}|s)]$. Therefore, there is no inconsistency between the diffusion training and diffusion-based regularization in our method.

- We further observe that SRPO incorporates $E_{a\sim \pi_{\theta}}[-\log\mu_\phi(a_{\theta}|s)]$ as the second term in its KL-based theoretical loss. The other component of the SRPO KL loss is $E_{a\sim \pi_{\theta}}[\log\pi_\theta(a_\theta|s)]$, which is notably absent in our loss formulation. The theoretical loss of SRPO is intractable to optimize, necessitating an approximation that substitutes $a_\theta$ with $z_t$. This substitution further differentiates the SRPO loss from our DTQL loss.

[1] Song et al. "Maximum likelihood training of score-based diffusion models." NeurIPS 2021.

### 3. Empirical performance difference

Due to character limitations, the discussion will be provided in our response to the specific reviewer.

Meta-review: Most of the criticism regarding this submission centered on the clarity of its positioning and the interpretation of the trust region. As the Area Chair, I believe the authors have addressed these concerns satisfactorily in their rebuttal, and I therefore recommend acceptance. However, I strongly encourage the authors to undertake a significant revision of the paper to ensure it fully realizes its potential and provides maximum value to the community.
Decision: Accept (poster)