Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
Authors: Chen-Xiao Gao
Chenyang Wu
Mingjun Cao
Chenjun Xiao
Yang Yu
Zongzhang Zhang
Venue: ICML 2025 poster (2025)
Keywords: Behavior-regularized Reinforcement Learning
Diffusion Policy
Offline Reinforcement Learning
Primary area: reinforcement_learning->batchoffline
TL;DR: We introduce a principled behavior-regularized reinforcement learning framework tailored for diffusion-based policies in offline scenario.

Abstract: Behavior regularization, which constrains the policy to stay close to some behavior policy, is widely used in offline reinforcement learning (RL) to manage the risk of hazardous exploitation of unseen actions. Nevertheless, existing literature on behavior-regularized RL primarily focuses on explicit policy parameterizations, such as Gaussian policies. Consequently, it remains unclear how to extend this framework to more advanced policy parameterizations, such as diffusion models. In this paper, we introduce BDPO, a principled behavior-regularized RL framework tailored for diffusion-based policies, thereby combining the expressive power of diffusion policies and the robustness provided by regularization. The key ingredient of our method is to calculate the Kullback-Leibler (KL) regularization analytically as the accumulated discrepancies in reverse-time transition kernels along the diffusion trajectory. By integrating the regularization, we develop an efficient two-time-scale actor-critic RL algorithm that produces the optimal policy while respecting the behavior constraint. Comprehensive evaluations conducted on synthetic 2D tasks and continuous control tasks from the D4RL benchmark validate its effectiveness and superior performance.

Behavior-Regularize Diffusion Policy Optimization for Offline Reinforcement Learning
Chen-Xiao Gao
Chenyang Wu
Mingjun Cao
Chenjun Xiao
Yang Yu
Zongzhang Zhang
Abstract
Q
1
Deterministic Policy
Diffusion Policy (BDPO)
Behavior regularization, which constrains the pol- iey stay close some behavior policy, widely used in offline reinforcement learning (RL) manage the risk of hazardous exploitation of un- seen actions. Nevertheless, existing literature on behavior-regularized RL primarily focuses on explicit policy parameterizations, such as Gaus- sian policies. Consequently, remains unclear how extend this framework to more advanced policy parameterizations, such diffusion mod- cls. In this paper, introduce BDPO, princi- pled behaaior-regularized RL. framework tailored for diffusion- baseed policies, thereby combining the expressive power of diffusion policies and the robustness provided by regularization. The key ingredient of our method to calculate the Kullback-Leiibee (KI.) regularization analytically the accumulated discrepancies in reverse-time transition kernels along the diffusion trajectory. By integrating the regularization, we develop an efficient two-time-scale actor-critic RL. algorithm that produces the optimal policy while respecting the behavior constraint. Comprehensive evalua- tions conducted on synthetic 2D tasks and contin- uous control tasks from the D4RL benchmark val- idate its effectiveness and superior performance. The code and experiment results of BDPO are available on the project webpage.
Policy
Diffusion Path
Behavior Policy
Actor Difrusion Direction
Penalty
Behavior Direction Penalty
Figure I. Illustration of the behavior-regularizzd RL framework with different policy parameterizations. Unimodal policies, such as deterministic policies (left), compute the behavior the center of mass and therefore lead to mislcading regularizations; while our method (right) harnesses the flexihility of diffusion madels, and the regularization is calculated as the accumulated discrepancies in diffusion directions the actor and the behavior diffusion.
and generative model fine-tuning (Ouyang et al., 2022), re- inforcement learning (RL) algorithms typically require mil- lions of online interactions with the environment to achieve meaningful optimization (Haarnoja et al., 2018; Schulman et al., 2017). Given that online interaction can be hazardous and costly in certain scenarios, offline RL, which focuses optimizing policies with static datasets, emerges practical and promising avenue (Levine et al., 2020).
However, without access to the real environment, RL algo- rithms tend to yield unrealistic value estimations for actions that are absent from the dataset (Levine et al., 2020; Fuji- moto Gu, 2021). Since RL necessitates querying the val- ues of unseen actions further refine its policy beyond the dataset, is prone exploiting the values of those unseen actions, leading to serious overestimation in value function. To manage this risk, one predominant approach is to employ behavior regularization (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Tarasov et al., 2023), which augments the usual RL objectives by incorporating penalty terms that constrain the policy remain close the behavior policy that collects the datasets. In this way, the policy is penalized for choosing unreliable out-of-distribution (OOD) actions and thus exercises the principle of pessimism in the face of uncertainty.
1. Introduction
Despite its huge success in industrial applications such as robotics (Kumar et al., 2021), game AI (Vinyals et al., 2019),
Naaioonl Key Laboratory for Novel Software Technology, Nanjing University, China & School of Artificial Intelligence, Nanjing University, China "The Chinese University of Hong Kong, Shenzhen, China. Correspondence to: Zongzhang Zhang <zzzhang nju.edu.en>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Most preliminary works in offline RL assume explicit pol- icy distributions (Fujimoto & Gu, 2021; Haarnoja al.,

---

Behavior- Regularized Diffusson Policy Optimization for Offline Reinforeement Learning
2018; Christodoulou, 2019); for instance, the agent's policy is often modeled as a Gaussian distribution with parame- terized mean and diagonal covariance, or a deterministic distribution that directly outputs the action. Despite the com- putational efficiency, such assumptions present significant challenges within the aforementioned behaaior-regullrrieed framework (Wang et al., 2023; Gansen-Estruchs et al., 2023; Chen et al., 2023). The underlying difficulty originates from the multi- modality of the distributions and manifests in two ways: 1) the dataset may be collected from multiple policy checkpoints, meaning that using unimodal distributions to approximate the behavior policy for regularization can in- troduce substantial approximation errors (see Figure 1); 2) the optimal policy distribution in behavior-regularizee RI. framework is actually the Boltzmann distribution of the opti- mal value function (Haarnoja al., 2017: Nair et al., 2020), which is inherently motivates researchers to explore diffusion models, which generate actions through a series of denoising steps, as a viable parameterization of the policy ( (Wang et al., 2023; Hunsen-EEttruee et al., 2023; Chen et al., 2024a;b; Fang et al., 2024; Zhang et al., 2024b).
2. Related Work
Offline RL. To improve beyond the interaction experience, RL algorithms need to query the estimated values of unseen actions for optimization. In offline scenarios, such distri- bution shift tends cause serious overestimation in value functions due to the lack of corrective feedback (Kumar et al., 2020a). To mitigate this, algorithms like CQL (Kumar et al., 2020b), EDAC (An et al., 2021), and PBRL (Bai et al., 2022) focus on penalizing the Q-values of OOD actions to prevent the over-estimation issue. Behavior regularization provides another principled way to mitigate the distribution shift problem, by adding penalties for deviation from the dataset policy during the stage of policy evaluation, pol- icy improvement (Fujimoto et al., 2019; Fujimoto & Gu, 2021; Ran et al., 2023), or sometimes both (Wu et al., 2019; Tarasov et al., 2023). Another line of research, also based on the behuvior-regularized RI. framework, approaches offline RL by performing in-samlle value iteration (Kostrikov et al., 2022; Garg et al., 2023; Xu et al., 2023), thus eliminating OOD queries from the policy and directly approximating the optimal value function. Lastly, model-based offline RL methods (Yu et al., 2020; Jia et al., 2024; Sun et al., 2023; Luo et al., 2024) introduce learned dynamics models that generate synthetic experiences to alleviate data limitations, providing extra generalization compared to model- -reeffeeeeeeee al- gorithms.
well. Such limitation
Nonetheless, extending the behavior-reggularized RL frame- work to diffusion policies faces specific challenges. Al- though diffusion models are capable of generating high- fidelity actions, the log-probability of the generated actions is difficult to compute (Song et al., 2021). Consequently, the calculation of regularization terms, such as Kullback- RL
Diffusion Policies in Offline RL. There has been no- table trend towards the applicatioons of expressive generative models for policy parameterization (Janner et al., 2022), dynamics modeling (Micheli et al., 2023; Alonso et al., 2024), trajectory planning (Chen et al., 2021; Ajay et al., 2023; Gao et al., 2024), and representation learning (Shribak et al., 2024). In offline RL,, several works employ diffu- sion models to approximate the behavior policy used for dataset collection. To further improve the policy, they uti- lize in-sample value iteration to derive the Q-values, and subsequently select action candidates from the behavior diffusion model (Hlansen-Estrude et al., 2023; Chen et al., 2023) or use the gradient of Q-value functions to guide the generation (Lu et al., 2023; Mao et al., 2024). However, the performance of these methods is limited, the actions come from behavior diffusion. Alternatively, SRPO (Chen et al., 2024a) and DTQL (Chen et al., 2024b) maintain a simple one-step policy for optimization while harnessing the diffusion model or diffusion loss to implement behavior reg- ularization. This yields improved computational efficiency and performance in practice; however, the single-step policy still restricts expressiveness and fails to encompass all of the modes of the optimal policy. A wide range of works there- fore explore using diffusion models as the actor. Among them, DAC (Fang et al., 2024) formulates the optimization as a noise-regression problem and proposes to align the output from the policy with the gradient of the Q-value func-
Leibler (KL) divergence, within the framework remains ambiguous (Zhang al., 2024b; Wang et al., 2024). Besides, it is also unclear how to improve diffusion policies while simultaneously imposing effective regularization. In this paper, we introduce BDPO, which provides an efficient and effective framework tailored for diffusion policies. Specifically:
1) Framing the reverse process of diffusion models as an MDP, we propose to implement the KL divergence w.r.t. the diffusion generation path, rather than the clean action samples (Figure 1);
2) Building upon this foundation, we propose a two-time- scale actor-critic method to optimize diffusion policies. In- stead of differentiating the policy along the entire diffusion path, BDPO estimates the values at intermediate diffusion steps to amortize the optimization, offering efficient com- putation, convergence guarantee, and state-of-the-art perfor- mance:
3) Experiments conducted on synthetic 2D datasets reveal that our method effectively approximates the target distri- bution. Furthermore, when applied to continuous control tasks provided by D4RL, BDPO demonstrates superior per- formance compared to baseline offline RL algorithms.
2

---

Behavior- Regularized Diffusson Policy Optimization for Offline Reinforcement Learning
tions. Diffusion-QL. (Wang et al., 2023) optimizes the actor by back-propagating the gradient of Q-values throughout the entire diffusion path. This results in a significant memory footprint and computational overhead, and EDP (Kang et al., 2023) proposes to use action approximations to alleviate the cost, In contrast, BDPO maintains value functions for inter- mediate diffusion steps, thereby amortizing the optimization cost while also keeping the optimization precise.
improvement, the performance of the policy defined by Eq. (1) is guaranteed to improve (Haarnoja et al., 2018).
Diffusion Models. Diffusion models (Ho et al., 2020; Song et al., 2021) consist of a forward Markov process which progressively perturbs the data x. ~ go to data that approxi- mately follows the standard Gaussian distribution x. ~ qN. and a reverse Markov process that gradually recovers the original x0 from the noisy sample z²" The transition of the forward process 1/+1/n usually follows Gaussian distribu- tions:
3. Preliminaries
Behavior- Behavior-Regularized Offline RL. We formalize the task as a Markov Decision Process (MDP) A. T, R, 7), where S is the state space, A the action space, T(d'ss, a) denotes the transition function, R(s, a) is a bounded reward function, and y is the discount factor. In offline RL, an agent is expected to learn a policy : S -> ((A) to maximize the expected discounted return E, (> 7 R(st, a:)) with an offline dataset DD = {(5,, le. rz)),, }. where st, S1+1 E S, a, E A. and r. R(*,, 4x) E RR. We consider the behavior- regularized RL objective, which augments the original RL objective by regularizing the policy towards some behavior policy v:
In/n-1 (1x (1="|x"=1)) = -N(x";
(5)
where (Br) is specified according to the noise schedule and / denotes the Identity matrix. Due to the closure prop- erty of Gaussian distributions, the marginal distribution of x" given x0 can be specified as:
I 0mjo(="|2") N(=x; Vanz" -dn))),
(6)
where an = 1- - Br, an Ih=== Owy. The transition of the reverse process can be derived from Bayes' rule,
In-1/n(x" (x=) Un|n- (x"1x=" )1 pn (a
max, Ex 7 nDKL ((((s))))(()()))))
On (n(x))
(1)
However, it is usually intractable, and therefore we use parameterized neural network Ph to approximate the reverse transition, which is also a Gaussian distribution with parameterized mean:
where DKL is the Kullback-Leibler (KL) divergence and
7 > 0 controls the regularization strength. In offline RL, Eq. (1) is widely employed by setting v as the policy ap that collects the dataset prevent the exploitation of the out- of-dataset actions. Besides, when setting as the uniform distribution, Eq. (1) equates to the maximum-entron entrooy RL in online scenarios up to some constant.
(x( = =N(0, I); ph Pn- (x (xm) = N1 (((") (,
(8)
To solve Eq. (1), a well-established method is soft policy it- cration (Haarnoja et al., 2018; Wu et al., 2019). Specifically, we define the soft value functions as
where On
I-an Bn as VB. The learning ob-
jective is matching ph- (z"=) with the posterior
Vn=2)n,0(v"~11100 1,0", z:)):
oo
* (aa (8t) r+- r glog
v(ae(s)) )
V (s) E
(2)
Lai (f) =
Em a
(KKL, In (n- 1a"", 10)) (x 12t))
where the expectation is taken w.r.t. random trajectories gen- erated by ı under the initial condition so = s and ag = a. The soft Q-value function in this framework can be solved by the repeated application of the soft Bellman operator 13
where x0 x0, ~ an on (-1=2)). Upon the completion of training, it can be shown that the objective Eq. (9) yields
Ph- (x"-110") |2-) == q 1|n (xn- 12=").
(10)
T(alls)) BT as) = R[s a) + 7E (s"(s' 17 log v(aals)
Simplified training objectives can be derived by reparam- eterizing nº with noise prediction or score networks (Ho et al., 2020; Song et al., 2021). After training, the genera- tion process begins by sampling '.V -N, 7), followed by iteratively applying Pa to generate the final samples 2o. which approximately follow the target distribution go-
(3)
where s' ~ ~T((|s, a) and a' ~ (-1)"). For policy improve- ment, we can update the policy using the objective:
max Ee-x(-1e) [Qt a)] - D). ([)(s))])))))))
(4)
Note that regularization is added for both Q-value functions and the policy. By iterating between policy evaluation and
Diffusion Policy. Diffusion policies are conditional dif- fusion models that generate action a on a given state s.
3

---

Behavior-Regularizn Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
the last equation holds due PN PN NO In the following, abbreviate DKL. pa-in " v,s,a" (a").
St
st+1
T( 1g.
n
with
at
T PN-1IN
at
1
O
a
Based on this decomposition, we present the pathwise KL- regularized RL problem.
Pots
Definition 4.1. (Pathwise KL-Regularized RL) Let p" be the behavior diffusion process. The pathwise KL-regularized RL problem seeks to maximize the following objective
Tt
N-1
Figure 2. Semantic illustration of the interplay between diffusion policies and the environment. We use orange to denote the tran- sition P f--iln and the penalty f (see Section 4.1) associated with the diffusion generation process, whereas blue signifies the transition T(-[as, a and the reward from the original environ- ment MDP.
sc max E
)
=
Σ 1=
r
R( R(st, 7
C (at)
(12)
where the expectation is taken w.r.t. trajectories following p" and the dynamics T.
In this paper, we will use p" and pr to denote the dif- fusion policy and its state-conditioned version, respec- tively. Similarly, we will use Pn-1|n as shorthand for the single-step reverse transition conditioned on a", i.c., Pa-1/n = (-l,)) At timestep / of the environment MDP, the agent observes the state drives the reverse
A Although our problem calculates the accumulated discrep- ancies along the diffusion path as penalties, it actually pre- serves the same optimal solution to Eq. (1), which only regularizes the action distribution at diffusion step n = 0. The following theorem captures the equivalence.
__,,0"
diffusion process a O:N Po.N T.9t as defined in Eq. (8), and takes a as its action ae. For action af' we will use in the subscript to denote the timestep the environment MDP, while using the superscript to denote the diffusion steps. A semantic illustration of the environment MDP and the reverse diffusion is provided in Figure 2.
Theorem 4.2. (Proof in Appendix C.2) Let p' be the be- havior diffusion process. The optimal diffusion policy p of the pathwise Kl.rregularized RI. problem in Eq. (12) ) is also the optimal policy n² of the KL regularized objective in Eq. (1), in the sense that x (als) = S Po (a ON )ō(a a )d O:N Vs S. where is the Dirac delta function.
That said, we can safely optimize Eq. (12) and ultimately arrive the solution that also maximizes the original KL- regularized RI problem. As will be demonstrated in the following sections, formulation of the penalty leads to an analytical and efficient algorithm, since each single-step reverse transition is tractable.
4. Method
4.1. Pathwise KL Regularization
In behavior-regularize RL, previous methods typically reg- ularize action distribution, i.e., the marginalized distribution Ph Instead, we shift our attention the KL divergence with respect the diffusion path ao:,, which can be further decomposed thanks to the Markov property:
4.2. Actor-Critic across Two Time Scales
To solve the pathwise KL-regularized RL (Eq, (12)), we can employ an actor-critic framework. Specifically, we maintain a diffusion policy p" and a critic Q The update target of the critic Q" is specified as the standard TD target in the environment MDP:
DKL IP:N II
PO.N (a (ND:N]
J.S
E log
D.S Po.N (a (:N)
PN (a IIN.
- 1/1 (a" (am)
log
P'N (a
I1 P 1/n (an-21a")
(11) (² (ossa) + BB (B
N - Σ
ex (d"))
log
TT Na PN (a
Σ log
p Po- 1|n
(13)
v Pa-1jn (an-1)
i.e., we sample diffusion paths a A:: at the next state s', cal- culate the Q-values (s'(s,, a²9) and the accumulated penal- ties along the path, and perform a one-step TD backup. For the diffusion policy, its objective can be expressed as:
DRL n.a."" ",0 Pn-1jn 11
1n-1//
R.8 Here, the expectation is taken w.r.t. O:N ~ Po: p" is the behavior diffusion policy trained via Eq. (9) on the offline dataset D to approximate the data-collection policy, and
N
max no:N -Po
E
(²((aa)) -1 -1 e
(aa)
(14)

---

Behavior-Regularizet Diffusson Policy Optimization for Offline Reinforeement Learning
Q" (1 al-1)
(T
Tp-1
W (af))
V Va 11
V22 (af)
V
(at)
Figure 3. Semantic illustration of the TD backup for the Q-value function Q" (blue) and diffusion value function V (orange). The update of Q" (Eq. (13)) requires reward, penalties along the diffuusio trajectory, and the Q-values at the next state. The update of V, (Eq. (15)) involves the single-step penalty and the diffusion value at the next diffusion step n
At first glance, one might treat the pathwise KL as a whole and optimize the diffusion policy by back-propaggting the gradient throughout the sampled path a.,, similar to Diffusion-QL (Wang al., 2023). Nevertheless, this re- quires preserving the computation graph of all diffusion steps and therefore incurs considerable computational over- head. Moreover, the inherent stochasticity of the diffusion path introduces significant variance during optimization, which can hinder the overall performance.
Algorithm 1 Behavior Reguuaiied Diffuion Policy Opti- mization (BDPO)
Input: Offline dataset D, initialized diffusion policy p" and p". Q-value networks Q diffusion value networks Von.
1: // Pretrain p"
2: Pretrain the behavior diffusion policy p" via Eq. (9)
3: Initialize pd with p"
4: // Train px
5: for i
1.2. Nevaa steps do
Instead, we opt for bi-level TD learning framework. The upper level updates Q" according to Eq. (13) and operates in the environment MDP. In the lower level, we maintain diffusion value functions va="((a)) which are designed to estimate the values at intermediate diffusion steps. The TD target of V is specified as:
6: Sample minibatch B {(S dq r.)) from D
7: Update (Q jk= via Eq. (19) using B
8: Extend each (st, s/, r.) in B by sampling ~ U[1, N] and oZ ~ (n)o(-
Update (Vo, K via Eq. (19) using B
10: if actor update interval 0 then
11: Update p" by performing gradient ascend with the
To V (a²) = (x((,a T, V. (a") -1p
objective defined in Eq. (16) and B
11 (a")
E
12: end if
13:
end for
(15)
Intuitively, v(T**(a") receives the state s, intermediate gen- eration result a", and the diffusion step as input, and estimates the expected cumulative penalties of continuing the diffuusin path at a" and step n. It operates completely inside each environment timestep and performs TD updates across the diffusion steps. The semantic illustration of the value function backups is presented in Figure 3. The fol- lowing proposition demonstrates the convergence of policy evaluation.
Assumption 4.3. Let II be the set of admissible diffusion policies satisfying Vp= E 11, E
T(a0(s) () S PuiN alEA v(a0ls) S Po.N (a:N dda < c0,
(N Jda
sup
where p" is the behavior diffusion policy.
Proposition 4.4. (Soft Policy Improvement, proof in Ap- pendix C.4) Let p be the optimizer of the problem de- fined in Eq. (16). Under Assumption 4.3, V (a") 2
With the help of n to
we can update the policy according
a (ax) holds for all n I,. N} and (8,a)
S x
max Pn-1in
-me -
+ E Pn-ilm
[ (a"-1))
(16)
Combining Proposition 4.4 and similar convergence argu- ment in Soft Actor Critic (Haarnoja et al., 2018), we obtain that the policy is guaranteed improve w.r.t. the pathwise KL problem in Definition 4.1.
Note that the maximization is over every state 8, diffusion step n, and intermediate action a". In this way, we only require single-step reverse diffusion during policy improve- ment, providing an efficient solution to the problem defined in Eq. (16).
Proposition 4.5. (Policy Iteration, proof in Appendix C.5) Under Assumption 4.3, repeated application of soft pol- icy evaluation in Eq. (13) and Eq. (15) and soft policy improvement in Eq. (16) from any p" E II converges to a policy px such that V ((a) 2 V.z (a) for all p" e II. n E {0, 1.......] NN) and (8,0) E S A.
Through simple recursion, we can demonstrate that updating p" with Eq. (16) leads to the same optimal solution to the original policy objective Eq. (14).
5

---

Behavior-Regularie Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
ned0
p-30
n=20
n=10
actirr
0.8
0.4
0.0
nw10
0.8
0.4
9.0
hw
n=
samp
n-60
n=30
n-20
n-10
n-9
0.8
0.4
0.2
n=50 n=
samples
Figure Generation paths of BDPO on the 8gaussian (top), 2xpirals (middle), and moons (down) datasels. The regularization strength set 0.06, which is identical to Figure 5. The first five columns depict the diffusion generation process different time intervals, with green dots indicating the starting points these intervals, red dots indicating the ending points, and grey lines between representing intermediate samples. The background color depicts the output from the diffusion value functions V7 in the entire 2D space. The rightmost figures depict the final action samples. We use DDIM sampling (Song et al., 2020) for better illustration.
8gaussians (dataset) 2spirals (datasel)
moons (dataset)
value functions to account for both inter-timestep and inner- timestep policy evaluation, thereby offering low-variance policy gradients.
4.3. Practical Implementation
Calculation of KL Divergence. Since each single-step reverse transition Pn-1 n. an isotropic Gaussian distribution (Eq. (8)), the KI. diver- gence is analytically expressed
8gaussians in= 0.06) 2spirals (n= 0.06)
moons (n 0.06)
K.5.0n is approximately parameterized as
|| (a")
1 "a*((a"112 (17)
T.3.a" DkL. P 1P
20
That is, each penalty term is simply the discrepancy between the vectors of the reverse transitions, weighted by factor depending the noise schedule. The transition of the reverse process becomes Gaussian exactly in the continuous limit, in which case, the KI. divergence can be computed through Girsanov's theorem (Oksendal, 2013), and the sum of mean squared errors (MSEs) is replaced by an integral (Franzese al., 2024). Appendix D, we demonstrate that the pathwise KL is consistent with the result from Girsanov's theorem in the limit of infinitesimal di diffusion step size.
Figure 5. Illustration of the 8gaussians, 2spirals, and datasets. The top row depicts the original data distribution paan while the second row depicts the target distribution Parge at = 0.06 by re-sampling data points according to their energies.
Remark (Actor-critic across two time scales). Our treat- ment of the environment MDP and the (reverse) diffusion MDP resembles DPPO (Ren et al., 2024), which unfolds the diffusion steps into the environment MDP and employs the PPO (Schulman et al., 2017) algorithm to optimize the diffu- sion policy in the extended MDP. Compared to DPPO, our method adds penalty each diffusion step to implement behavior regularization. Besides, we maintain two types of
Selection of States, Diffusion Steps and Actions. The pol- icy improvement step requires training V and improving pd on every (s, a", n) triplet sampled from the on-policy
6

---

Behavior- Regularizgd Diffusion Policy Optimization for Offline Reinforcement Learning
Table Comparison of BDPO and various baseline methods locomotion-v2 and antmaze-v0 datasets from D4RI.. We 'm' as the abbreviation for medium. 'r' for replay, and "e" for expert. The performances of the baseline methods are taken from their original paper. BDPO, we report the average and standard deviation the performances across evaluation episodes (100 for antmaze datasets) and 5 seeds. We bold values that are within 95% of the top-performing method.
Dataset
CQL
IQL.
DD
SMBC
IDQL-A
QGPO
SRPO
DTQL Diffusion-QU
EDP
DAC
BDPO (Ours)
halfcheetahhmm
44.0
47.4
49.1
45.9
51.0
54.1
60.4
57.9
51.1
52.1
59.1
71.2:10.9
hopper-m
58.5
66.3
79.3
57.1
65.4
98.0
95.5
99.6
90.5
81.9
101.2
100.6++0.7
walker2d-m
72.5
78.3
82.5
77.9
82.5
86.0
84.4
89.4
87.0
86.9
96.8
03.4
halfehectah-m--r
45.5
44.2
39.3
37.1
45,9
47.6
51.4
50.9
47.8
49.4
55.0
58,9+0.9
hopper-m-r
95.0
94.7
100.0
86.2
92.1
96.9
101.2
100.0
101.3
101.0
103.1
101,4+0,5
walker2d-m-r
77.2
73.9
75.0
65.1
85.1
84.4
84.6
88.5
95.5
94,9
96.8
95
halfcheetah-m--e
91.6
86.7
90.6
92.6
95.9
93.5
92.2
96.8
95.5
99.1
108.7110.9
hopper-m-e
105.4
91.5
111.8
108.6
108.6
108.0
100.1
109.3
111.1
97.4
111.7
111.3
walker2d-me m
108.8
109.6
108.8
109.8
112.7
110.7
114.0
110.0
110.1
110.2
113.6
115.6+00,4
locomotion sum
698.5
749.7
736.2
680.3
739.2
779.2
783.8
798.3
791.2
769.3
836.4
856.7
antmaze-u
74.0
87.5
92.0
94.0
96.4
90.8
94.8
93.4
94.2
99.5
98.4+11.5
antmaze-u-div
84.0
62.2
85.3
80.2
74.4
59.0
78.8
66.2
79.0
85.0
13.0
antmaze-m-paa -lay
61.2
71.2
81.3
84.5
83.6
73.0
79.6
76,6
81.8
85,8
92.0+22.9
antmaze-m-div
53.7
70.0
82.0
84.8
83.8
65.2
82.2
78.6
82.3
84.0
93,2-4-2.8
antmaze-l-play
15.8
39.6
59.3
63.5
66.6
38.8
52.0
46.4
42.3
50.3
69.0++8.0
antmaze-l-div
14.9
47.5
45.5
67.9
64.8
33.8
54.0
56.6
60.6
55.3
84.0
antmaze
303.6
378.0
445.4
474.9
469.6
360.6
441.4
417.8
440.2
459.9
519.6
distribution of p² However, we employ an off-policy ap- proach, by sampling (s,a) a) from the dataset, n according to the Uniform distribution U[1, .V] and a" according to (njo--aa,, which we found works sufficiently well in our experiments. We note that on-policy sampling (s, a", n) may further benefit the performance at the cost of sam- pling and restoring these triplets on the fly, similar to what iDEM (Akhound-Sadep) et al., 2024) did in its experiments.
where n is sampled from U11, N] and a" from gn0o-laa)
The pseudo-code for BDPO is provided in Algorithm
5. Experiments
We evaluate BDPO with synthetic 2D tasks and also the D4RL benchmark (Fu et al., 2020). Due to the space limit, a detailed introduction about these tasks and the hyperparam- eter configurations is deferred to Section A and Section B. More experimental results are deferred to Appendix E.
Cower-Confidence Bound (LCB) Value Target. Follow- ing existing practices (Zhang al., 2024a; Fang et al., 2024), use an ensemble of K = 10 value networks (1yk, Φx K for both V and QT and calculate their target as the LCB of Eq. (13) and Eq. (15):
5.1. Results of Synthetic Datasets
We leverage synthetic 2D datasets used in Lu et al. (2023) to further our understanding of the mechanism of BDPO. Each dataset contains data points x, paired with specific energy values E(x,). By ra-sampling the data according to the Boltzmann distribution p(æ:) x exp(E(x;)/n) we can obtain the ground truth target distribution at a specific regularization strength (Haarnoja et al., 2018; Nair et al., 2020):
JQ = Avga 1 Q
PV Var [=QQ
(18)
yv Avga T V
PV Vark V ni
n.o
where de and wi are exponentially moving average versions of the value networks, and Avg, Var denote averages and variances respectively. The objectives for value networks are to minimize the mean-squared error between the prediction and the target:
Parget (I:) Pamaa (x;) exp(E(z))/n))
As an example, Figure 5 illustrates both the original dataset and the re-sampled dataset with r) 0.06 for the 8gaussians, 2spirals, and moons datasets.
(((va)K))) (s
lyQ - - as,
Our objective is to analyze the properties of the diffusson pol- icy and the diffusion value function trained with BDPO. The results are depicted in Figure (full results on all datasets (19) in Figure 13), where each row corresponds different
K
[((p))K)) = BBoaa))D.m.a
(uv - n,o. )²

---

Behavior- Regularized Diffusson Policy Optimization for Offline Reinforcement Learning
hallcheetah-meedam-22
halfcheetäh-mellluu-eeerrrrr
walkerr2d-medium-z2
hepper-meddum-experss
Normaized Score
100
100
100
80
50
50
40
0.5M
1.5M
2M
1M
0.5M Triniin Steps
1.5M
3M
Training Steps
Trairing Steps
Training Steps
n=0.00
n-0.00
0,0.3
g-0.5
1=1.0
Figure 6. Sensitivity analysis of the regularization strength 1). For each configuration, report the mean and the standard deviation of the performances aggregated from 5 independent seeds and 10 evaluation episodes for each seed.
hallcheetah-mediam-22
hallcheetaa--mddii---ppprrrrr
walker22--medium-22
ho happee--medium-expettt
70
100
75
Nomraized
80
50
50
50
60
40
0.5M
1.5M
2M
1M
0.5M
1.5M
3M Training Steps
Training Steps
Training Steps
Training Steps
p=0.0
p=0.5 p=1.
p=1.5
p=2.0
Figure 7. Sensitivity analysis of the lower confidence bound coefficient p. For each configuration, we report the and the standard deviation of the performances aggregated from 5 independent seeds and 10 evaluation episodes for cach seed.
dataset and visualizes the iterative sampling process. From left to right, the samples evolve from an initial noisy distribu- tion toward well-defined structure as the reverse diffusion progresses. In the initial steps (n = 50 to n = 40), the sam- ple movement is subtle, while the later steps (n 30 to = 0), the samples rapidly converge to the nearest modes of the data. The final column the right presents the ulti- mate samples generated by the policy, which closely match the ground-truth target distribution visualized in Figure 5.
representatives using straightforward one-step policies; 2) Decision Diffuser (DD) (Ajay et al., 2023), which uses diffu- sion models for planning and extracts actions from planned trajectories; 3) SiBC (Chen et al., 2023), IDQL (Hansen- Estruch al., 2023), and QGPO (Lu al., 2023), which use diffusion to model the behavior and further refine actions using the value functions; 4) SRPO (Chen et al., 2024a) and DTQL (Chen et al., 2024b), which use diffusion to provide behavior regularization for one-step actors; 5) Diffusion- QL (Wang et al., 2023), which propagates the gradient of Q-value functions over the whole generation process; and 6) EDP (Kang al., 2023) and DAC (Fang et al., 2024), which accelerate the training of diffusion policies by using action approximation using proxy objectives. The performance of each algorithm is measured by the normalized score on each task (see Appendix A for details).
Besides the generation result, the background color in Fig- ure 4 depicts the landscapes of the diffusion value functions V= at 50/40/33 10/0032220 /0/1/0. In earlier, noisier steps (e.g., n = 50), the output values across the space are simi- lar, resulting in smoother landscape and weaker guidance during sampling. As the noise level decreases in later steps (e.g., = 10). the value outputs vary more sharply, creating stronger guidance on the diffusion policy. Such progression enables the model to refine samples while exploring suffi- ciently across the space, finally yielding results that align with the target distribution.
The results are listed in Table 1. Our findings indicate that diffusion- -based methods, particularly those with diffusion- based actor and regularization (including BDPO, DAC, and Diffusion- QL), substantially outperform their non-diffusion counterparts, especially in locomotion tasks. Meanwhile, BDPO consistently achieves superior performance across nearly all datasets, underscoring the effectiveness of com- bining diffusion policies and the behavior-regularized RL framework. Besides the final performance, we plot the train- ing curves of RDPO in Section E.4. Overall, BDPO achieves fast and stable convergence except for some of the antmaze
5.2. Results on Continuous Control Tasks
To assess the performance of BDPO on complex continuous control problems, utilize datasets from D4RL our test bench and compare BDPO against wide spectrum of offline RL algorithms. Specifically, we include 1) CQL (Kumar et al., 2020b) and IQL (Kostrikov et al., 2022), which are
8

---

Behavior- Regularized Diffusion Policy Optimization for Offline Reinforeement Learning
wallker2d-medium-22
halfchhetah--edlumm-2
Table 2. Runtime comparison among HIPPO, DAC, and Diffusion- QL (denoted D-QL in the table), measured in minutes.
75
60
Normalized
50
Alg
Phase
N 5
N
20
N
50
policy
17
20
18
BDPO
value
63
135
285
Training
Training
total
80
155
303
halfchestah-mediumm mww.immmmlayy---a
halfehe etwh--edii-------eeeeeee expert-v2
660
policy
30
31
30
DAC
value
30
109
251
Nomaa
44
total
60
140
285
30
policy
57
132
308
1.5M
1.5M
D-QL
value
31
80
252
Training Steps
Training Steps
total
88
212
560
BDPO
Deterministic
Gaussian
Figure& 8. Comparison of different policy parameterizations, Re- sults are taken from 10 evaluation episodes and seeds.
timal value. The results are illustrated in Figure 8, with additional details regarding the implementations provided in Appendix B.3. While simpler architectures generally converge more quickly, their performance at convergence often lags behind that of BDPO, indicating limitation in their capacity for policy parameterization.
datasets, where variations may occur due to the sparse re- ward nature of these datasets.
Discussion on Runtime, Table 2 compares the runtime of BDPO, DAC, and Diffusion-QL. using the walker2d-mediume v2 dataset. Our method consists three key steps: pretrain- ing behavior diffusion, training value functions (Q" and Vr and optimizing the actor. The pertaining phase takes around 8 minutes and is therefore omitted in further analysis. The Q" training follows the same approach as Diffusion- QL and DAC, involving diffusion path sampling at the next state, while V introduces acceptable overhead since only requires single-step diffusion. For actor training, both DAC and BDPO use single-step diffusion, ensuring compu- tational costs remain constant regardless of diffusion steps. In contrast, Diffusion-QL QL needs to hack-propagat through the entire diffusion path, leading to significantly higher runtime, especially when the number of diffusion steps in- creases.
5.3. Analysis of BDPO
Regularization Strength 1/. The hyperparameter 1) controls the regularization strength: larger forces the diffusion pol- icy to stay closer to the behavior diffusion, while smaller r) places more emphasis on the guidance from value networks. Figure 6 demonstrates this effect: in general, smaller does lead to better performances. However, excessively small values can result in performance degradation, as observed in walker2ddmmdiuum--2 In contrast auto-tuning 7) via dual gradient descend (Fang al., 2024; Zhang al., 2024a), we find that adjustable ') causes fluctuations in penalty calcula- tion. Therefore, employ fixed 1) throughout training.
Lower Confidence Bound Coefficient The hyperparam- eter P determines the level of pessimism on OOD actions since the variances of Q-values on these actions are com- paratively higher than in-dataset actions. In Figure 7, we discover that there exists certain range of p where the lower confidence bound value target works best, while excessively larger smaller values lead to either underestimation or overestimation in value estimation.
6. Conclusions
Our core contribution to extend the behavior-regularried RL framework diffusion policies by implementing the regularization the accumulated discrepancies of each single-step transition of the reverse diffusion. Theoretically, we demonstrate the equivalence of the pathwise KL and the commonly adopted KI. constraint on action distributions. In practice, we instantiate the framework with an actor-critic style algorithm that leverages value functions on two time scales for efficient optimization. The proposed algorithm, BDPO, produces generation results that closely align with the target distribution, thereby enhancing its performance and applicability in complex continuous control problems.
Policy Parameterization. As discussed in Section 1, im- proper assumptions about the action distribution may lead to inaccurate penalty calculations and ultimately degrade the overall performance. To validate this claim, we conduct series of ablation studies that replace the actor parameteri- zation with Gaussian distributions and Dirac distributions, respectively, while keeping other experimental configura- tions unchanged. For fair comparison, we fine-tune the regularization strength by searching within a predefined range and report the performance corresponding to the op-
9

---

Schavior-Regulariin Diffusson Policy Optimization for Offline Reinforcement Learning
Acknowledgements
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems (NeurlPS), 2021.
This work is primarily supported by the National Science Foundation of China (62276126, 62250069). Chenjun Xiao gratefully acknowledges funding from the National Natural Science Foundation of China (62406271).
Chen, T., Wang, Z., and Zhou, M. Diffusion policies creat- ing trust region for offiine reinforcement learning. arXiv preprint arXiv:2405. 19690, 2024b.
Impact Statement
This paper presents work whose goal is to advance the field of machine learning- There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
Christodoulou, Soft actor-critic for discrete action settings. arXiv preprint arXiv: 191000000100000000 2019.
Fang, L., Liu, R., Zhang, J., Wang, W., and Jing. B.-Y. Diffu- sion actor-critic: Formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning. arXiv preprint arXiv:2405.20555 2024.
References
Ajay, A., Du, Y., Gupta, A., Tenenbaum, J. B., Jaakkola, T. S., and Agrawal, P. Is conditional generative model- ing all you need for decision making? In International Conference on Learning Representations (ICLR), 2023.
Franzese, G., BOUNOUA, M., and Michiardi, P. MINDE: Mutual information neural diffusion estimation. In Inter- national Conference on Learning Representations (ICLR), 2024.
Sadeeh, T., Rector-Brooks, J., Bose, A. J., Mittal, S., Lemos, P., Liu, C., Sendera, M., Ravanbakhsh, S., Gidel, G., Bengio, Y., Malkin, N., and Tong, A. Iterated denoising energy matching for sampling from boltzmann densities. In International Conference on Machine Learn- ing (ICML), 2024.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4RL: Datasets for deep data-driven reinforcement learning, arXiv preprint arXiv:2004.07219 2020.
Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning. In Advances in Neural Informa- tion Processing Systems (NeurIPS), 2021.
Alonso, E., Jelley, A., Micheli, V., Kanervisto, A., Storkey, A., Pearce, T., and Fleuret, F. Diffusion for world mod- cling: Visual details matter in atari. arXiv preprint arXiv:2405.123399 2024.
Fujimoto, S., Meger, D., and Precup. D. Off- Off-policy deep reinforcement learning without exploration. In Interna- tional Conference on Machine Learning (ICML), 2019.
An, G., Moon, S., Kim, J., and Song. H. O. Uncertainty- based offline reinforcement learning with diversified Q- ensemble. In Advances in Neural Informaiinn Processing Systems (NeurIPS), 2021.
Gao, C.-X., Wu, C., Cao, M., Kong, R., Zhang, Z., and Yu, Y. ACT: Empowering decision transformer with dynamic programming via advantage conditioning. In AAAI Conference on Artificial Intelligence (AAAI), 2024.
Bai, C., Wang, I.., Yang, 7.,, Deng, Z., Garg, A., Liu, P., and Wang, Z. Pessimistic bootstrapping for uncertainty- driven offline reinforcement learning. In International Conference on Learning Representations (ICLR), 2022.
Garg, D., Hejna, J., Geist, M., and Ermon, S. Extreme Q-learning Maxent RL without entropy. In International Conference on Learning Representations (ICLR), 2023.
Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Rein- forcement learning with deep energy-basee based policies. In International Conference on Machine Learning (ICML), 2017.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. OpenAl gym. arXiv preprint 16Xiv:1606.015500
2016.
Chen, H., Lu, C., Ying, C., Su, H., and Zhu, J. Offline reinforcement learning via high-fidelity generative behav- ior modeling. In International Conference on Learning Representations (ICLR), 2023.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with a stochastic actor. In International Conference on Machine Learning (ICML), 2018.
Chen, H., Lu, C., Wang, Z., Su, H., and Zhu, J. Score reg- ularized policy optimization through diffusion behavior. In International Conference on Learning Representations (ICLR), 2024a.
P., Kostrikov, I., Janner, M., Kuba, J. G., and Levine, S. Idql: Implicit Q-learning as an actor- critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
10

---

Behavior- Regularized Diffusson Policy Optimization for Offline Reinforcement Learning
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion prob- abilistiic models. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Luo, F, Xu, T., Cao, X., and Yu, Y. Reward-consistent dynamies models are strongly generalizable for offline reinforcement learning. In International Conference on Learning Representations (ICLR), 2024.
Janner, M., Du, Y., Tenenbaum, J. B., and Levine, S. Plan- ning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning (ICML), 2022.
Mao, L., Xu, H., Zhan, X., Zhang, W., and Zhang, A. Diffusion- In-sample diffusion guidance for offline reinforcement learning. In Advances in Neural Informa- tion Processing Systems (NeurIPS), 2024.
Jia, C., Gao, C., Yin, H., Zhang, F., Chen, X.-H., Xu, T., Yuan, L., Zhang, Z., Zhou, Z.-H., and Yu, Y. Policy rehearsing: Training generalizable policies for reinforce- ment learning. In International Conference on Learning Representations (JCLR), 2024.
Micheli, V., Alonso, E., and Fleuret, F. Transformers are sample-efficient world models. In International Confer- ence Learning Representations (ICLR), 2023.
2024.
Nair, A., Gupta, A., Dalal, M., and Levine, S. Awac: Accel- erating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006. 09359, 2020.
Kang, B., Ma, X., Du, C., Pang, T., and Yan, Effi- cient diffusion policies for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Oksendal, B. Stochastic differential equations: An intro- duction with applications. Springer Science & Business Media, 2013.
Kingma, D. P. and Ba, Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A.. Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Kostrikov, I., Nair, A., and Levine, S. Offline reinforce- ment learning with implicit Q-learning. In International Conference on Learning Representations (ICLR), 2022.
Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, S. Stabilizing off-policy Q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems (NearIPS), 2019.
Ran, Y., Li, Y., Zhang, F., Zhang, Z., and Yu, Y. Policy regularization with dataset constraint for offline reinforce- ment learning. In International Conference on Machine Learning (ICML), 2023.
Kumar, A., Gupta, A., and Levine, S. Discor: Corrective feedback in reinforcement learning via distribution cor- rection. In Advances in Neural Information Processing Systems (NeurIPS), 2020a.
Ren, A. Z., Lidard, J., Ankile, L. L., Simeonov, A., Agrawal, P., Majumdar, A., Burchfiel, B., Dai, H., and Simchowitz, M. Diffusson policy policy optimization. arXiv preprint arXhv:2409.0055 2024.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Con- servative Q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020b.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and
Klimov, O. Proximal policy optimization algorithms. 2017.
arXiv preprint
Shribak, D., Gao, C.-X., Li, Y., Xiao, C., and Dai, B. Dif- fusion spectral representation for reinforcement learning. arXiv preprint arXiv:2406. 16/21, 2024.
Kumar, A., Singh, A., Tian, S., Finn, C., and Levine, S. A workflow for offline model-free robotic reinforcement learning. In Conference on Robot Learning (CORL), 2021.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint arXiv:2010.025500 2020.
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint ar arXhv:2005.01643, 2020.
Song. Y., fohl-Dicksteen, J., Kingma, P., Kumar, A., Er- mon, S.,, and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021.
Lu, C., Chen, H., Chen, J., Su, H., Li, C., and Zhu, J. Contrastive energy prediction for exact energy -uidey diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning (ICML), 2023.
Sun, Y., Zhang, J., Jia, C., Lin, H., Ye, J., and Yu, Y. Model- bellman inconsistency for model-based offline reinforce- ment learning. In International Conference on Machine Learning (ICMI.), 2023.

---

Behavior- Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
Tarasov, D., Kurenkov, V., Nikulin, A., and Kolesnikov, S. Revisiting the minimalist approach to offline rein- forcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M.,
Dudzik, A., Chung. J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., Oh, J., Horgan, D., Kroiss, M., Danihelka, I., Huang, A., Sifre, L., Cai, T., Agapiou, J. P., Jaderberg, M., Vezhnevets, A. S., Leblond, R., Pohlen, T., Dalibard, V., Budden, D., Sulsky, Y., Molloy, J., Paine, T. L., Gülçehre, Ç., Wang, Z., Pfaff, T., Wu, Y., Ring, R., Yogatama, D., Wünsch, D., McKinney, K., Smith, O., Schaul, T., Lillicrap, T. P., Kavukcuoglu, K., Hassabis, D., Apps, C., and Silver, D. Grandmaster level in starcraft II using multi-agent reinforcement learning, Nature, 2019.
Wang, Y. Wang, L., Jiang, Y., Zou, W., Liu, T., Song, X., Wang, W., Xiao, L., Wu, Duan, J., al. Diffu- sion actor-critic with entropy regulator. arXiv preprint 1rXiv:2405.15177, 2024.
Wang, Z., Hunt, J., and Zhou, M. Diffusion policies as an expressive policy class for offline reinforcement learning- In International Conference Learning Representations (ICLR), 2023.
Wu, Y., Tucker, G., and Nachum, O. Behavior regu- larized offline reinforcement learning. arXiv preprint 1111 2019.
Xu, H., Jiang. L., Li, J., Yang, Z., Wang. Z., Chan, W. K. V., and Zhan, X. Offline RI. with no OOD actions: In-sample learning via implicit value regularization. In International Conference Learning Representations (ICLR), 2023.
Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J. Y., Levine, S., Finn, C., and Ma, MOPO: Model-based offline policy optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Zhang, R., Luo, Z., Sjölund, J., Schön, T. B., and Mattsson, P.
diffusion policy with genseeblees for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2024a.
Zhang, R., Luo, 7., Sjölund, J., Schön, T. B., and Matts- son, Entropy-regulart ensembles for offline reinforcement learning. arXiv preprint arXiv:2402.0400 2024b.
diffusion policy with Q-
12

---

Behavior-Regularizet Diffusion Policy Optimization for Offline Reinforcement Learning
A. Introduction about the Benchmarks
Synthetie 2D Datasets. We leverage the synthetic 2D datasets from Lu et al. (2023) as a sanity check and a convenient illustration of the mechanism of BDPO. This task set encompasses 6 different datasets. For each dataset, the data points 2; are associated with different energies (x;). By sampling from the Boltzmann distribution associated with the temperature y::
p(z; exp(E(x))/m), xy E D, we can obtain ground-truth sample results at a specific regularization strength y/ (see Figure 9).
swissroll
swissroll =
swissroll (g 0.06)
8gaussians
Bgaussi (p=
Bgaussians (g 0.06)
0.8
-2.5
0.0
moons
moons (q-0.25) 0.25)
moons (1p - 0.06)
rings
rings (g 0.25
rings (n 0.06)
0.4
0.0
25
255 25
2.5 0.0 255
255 0.0
25 0.0
2.5
2.5 0.0
2.5
checkerboard
chockerboard (g- 0.25)
chockerboard 0.06)
2spitals
2spirals (q- 0.25)
2spirals (n- 0. 0.06)
2.5
0.0
25
25
0.0
255
25
00
25
25 0.0
25
25 0.0
25
25 0.0
25
0.0
Figure Illustration of synthetic 2D datasets and sampling results with various temperature 7).
We also use Gym MuJoCo (Brockman al., 2016), which provides series of continuous control tasks to assess the performance of BDPO as well as the baseline methods.
Locomotion Tasks. We choose 3 tasks from Gym MuJoCo: 1) halfcheetah, which is a robot with 9 body segments and 8 joints (including claws) as shown in Figure 10a. The goal is to apply torque to 6 of the joints to make the robot move forward as quickly as possible. The reward is based on forward movement distance. 2) hopper, which is a simpler robot with torso, a thigh, shin, and a foot as shown Figure 10b. The goal use torque on joints make the robot hop forward. 3) walker2d, robot with torso and two legs, each consisting of a thigh, shin, and foot shown in Figure 10c. The goal is to apply torque to 6 joints to make the robot walk forward, rather than hop.
For the offline dataset, we choose the -v2 datasets with three levels of qualities provided by D4RL (Fu et al., 2020): 1) medium, which is collected by policy that achieves approximately 30% of expert-level performance; 2) medium-replay which includes all data from the replay buffer of the training process of the medium-level policy; and 3) medium- a mixture of medium-leved data and expert-level data in a 1:1 ratio.
Navigation Tasks. We use antmaze as the representative of navigation tasks. In this task set, the agent needs control an eight degree-of-frredom anf robot to walk through a maze, shown in Figure 10d. We three different maps: umaze, medium, and large; and employed three different types of datasets: 1) the ant reaches fixed target from fixed starting position; 2) In the diverse dataset, the ant needs to go from a random position to random target; 3) In the play dataset, the ant goes from manually selected positions to manually selected targets (not necessarily the same as during evaluation). We use the - versions of the datasets, also provided by D4RL (Fu et al., 2020).
Metries. We measure the performance using the normalized score, which can be calculated by
Score(T) Scorezandom
Normalized Score(T) Scortexpen Scorerandom
where Score(T) is the average cumulated rewards achieve by the policy π. Score and Score are reference scores for random policies and expert policies, provided by D4RL..
13

---

Behavior- Reguuarr- Diffusion Policy Optimization for Offline Reinforeement Learning
B. Hlyper-parameter Configurations
B.1. D4RL Datasets
We largely follow the configurations from DAC (Fang et al., 2024). Parameters that are common across all datasets are listed Table 3. The hyperparameters of value networks the table are the same for both the Q-value networks Qù and diffusion value networks V.-
(a) halfcheetah
(b) hopper
(c) walker2d
(d) antmaze
Figure 10. Illustration the Locomotion Navigation Tasks.
Table 3. Common hyperparameters across all datascts.
MLP (256, 256, 256, 256) for locomotion tasks
Diffusion Backbone
MLPP(512, 512, 512, 512) for antmaze tasks
Diffusion Noise Schedule
Variance Preserving (Song al., 2021) DDPM
Diffusion Solver
Diffusion Steps
5
Behavior Learning Rate
0.0003 for locomotion tasks and 0.0001 for antmaze tasks
Actor Learning Rate
1e-5
Actor Learning Rate Schedule
Cosine Annealing
Actor Gradient Clip
1.0
Actor EMA Rate
5e-3
For antmaze-umaaze-livves generate 1 sample and execute
Actor Sampling
For others, generate 10 samples and select the action with the highest
Actor Update Frequency
Value Learning Rate
0.0003
Value Ensemble
10 (20 for the hopper-medium--22 task exclusively)
Value Backbone
MLP (256, 256, 256)
Value EMA Rate
0.005
Discount
0.99 for locomotion tasks and 0.995 for antmaze tasks
Batch Size
256
Optimizer
ADAM (Kingma & Ba, 2015)
Pretrain Steps
2e6
Train Steps
2e6
Value Warmup Steps
5e4
Sample Clip Range
[-1, 1]
We list the hyper-parameter that vary across different datasets in Table 4. We specifically tune the regularization strength and the LCB coefficient for locomotion tasks, plus the value backup mode for antmaze tasks.
For evaluation, at each timestep St, we use the diffusion policy to diffuse = 10 actions, and select the action with the highest Q-value as the final action a, to execute in the environment. We report the score normalized by the performances of random policies and expert policies, provided by D4RL (Fu al., 2020).
14

---

Behavior- Bchavior-Regularizet Diffusion Policy Optimization for Offline Reinforcement Learning
Table 4. Hyper- Syper--arameters that vary in different tasks.
Dataset
p
Max-Q Backup (Wang al., 2023)
halfcheetah matbbhectth-mmdium-22
halfcheetah- halfcheetah-medhuu--rppee -edium-replay-v2
0.5
0.05
False
halfcheeall-meefin--eeeeezz2
hopper-medium--v2
hopper-medduu--eeplyyyy2
2.0
0.2
False
hopper-meedium-exxprr--22
walker2d-meddium-v2
walker22-mmediun--rplay---
1.0
0.15
False
walkerz2l-medduu--xxxert---
antmaze-umaazo-y
0.8
0.5
Truc
autmaaz-umaazz-diierss-000
antmazze-mcdiuum-paayy0000
0.8
0.2
True
uatmaze-mediuum-ddver-----
antmaze-large-plaa-v00 autmaree-lrrge-divers---
0.8
1.0
True
B.2. Synthetic 2D Datasets
In the experiments with Synthetic 2D Datasets, basically followed the parameters in D4RL in Table 3, except that is set to O, 17 is set to 0.06, batch size is set to 512, and the number of diffusion steps is set to 50. We also turn off sample clipping for BDPO since the actions in these tasks are within 4445,
B.3. Details about the Ablation on Policy Parameterization
In this section, provide more details about how we implement the variants that deterministic policies Gaussian policies. The only difference between these variants and BDPO lines in the actor parameterization, which means that we keep the implementation of the value networks (e.g., ensemble tricks and LCB value target) and the hyperparameters (e.g..
the LCB coefficient p) identical to BDPO. We inherit the general behavior-reguil regulaaiee RL framework defined in Eq. (1), where we update the soft Q-value functions using the target
B (= R(s, a) + (3 ()(
(20)
and improve the policy ı via
Ea (Q" a)] nD)
(21)
where denotes certain types of regularization specific to the actor parameterization.
For deterministic policies, we follow Tarasov al. (2023) and use the mean squared error between actions from the policy and the behavior policy:
D(T, v) = 2 [T())) =(s)]12
(22)
For Gaussian policies, we parameterize as Diagonal Gaussian distribution T--1s) = - and regulariza- tion implemented as the KI. divergence:
(s) (s +
D(T(s),u(s)) DKL [T{()][[(()) log o,
(23)
20
We find that using the analytical KL computation for Gaussian policies works better than using sample-based estimations.
To ensure fair comparisons, we search for the optimal for both deterministic policies and Gaussian policies within certain range. Specifically, we choose the v) from (0.1, 1, 0.25, 0.5, 0.75, 1, 3, 5, 10). We observed that a higher eta would lead to
15

---

Recaavor-Regularii Diffusion Policy Optimization for Offline Reinforeement Learning
decline in performance, but excessively small eta would cause the performance oscillate drastically or even collapse in some cases. Therefore, select values that ensure stable performance and deliver the highest level of performance as shown in Table 5.
Table 5. Values of used in the ablation study with the policy parameterization.
Dataset
Deterministic Policy
Gaussian Policy
halfcheetah tasks
0.25
0.75
hopper tasks
1.0
0.1
walker2d tasks
1.5
0.25
C. Proof for the Main Results
Theorem C.I. Suppose that the behavior policy is diffusion model that approximates the forward process g by minimizing the objective
Eeowv(s) (ampo(-1aa) [DKr. (In-1ju,n(-aa M1p (-1aa)))1aa)] (-1"")] Vn (1,2,...,NN)) E.
(24)
Then, for any given Q-value function (s8, a), the optimal solution p of Eq. (14) is the reverse process corresponding to s
the forward process Ca:N (a II On (a" la-"a")a">)) where To(als) v(als) exp(Q(x,
Proof. Let's first consider the behavior diffusion policy p", which minimizes the following problem for every and state s:
min
(a (Anjo(""((" (a" DKL. 11-110 (laa), ,,") "="""="""
n-1 (1am)]
Pa-1jo
f ph- "a"""a""""" =1
s.t.
The corresponding Lagrange function is T(pp-1pu: A) = (e((1a))
(1a")]] (njo(a"la")Die ( a² )lp (" + A(1 pa- (a-1)n(a"~1100 la"aaa
jdaa-1
(25)
where À is the multiplier. Setting aL 0pmin-i
0, we obtain
1/1 (a" n- (a--1la")
- Ea (ano(a"(a")nnnnoonlle aaa)
(26)
Since P2-1 (1nt111a11pppp11 (an-I|a" aa" aan=-aaa)daa-- -da= 1, it follows that
/ Eaone(le) [anjo(a"|a "a0="""a"""-000maaa---0"00""0001 ,a")] d-a
(27)
-aaaaerl [anu(a"(a")]
Substituting Eq. (27) into Eq. (26),
Ex (njo(a" (a)) ( 10.n a")]
Pn-1jn (a (a"=
E ( [nnjo [
Ex (a) aa 9n- 0 aa))
(a")
(28)
v(a ((a)))njo(a"10)) (nna)))n1100
da
(a))
-h "1 (aa=)la"),
16

---

To do this, we first recognize that the reverse (generative) process of the diffusion policy constitutes a Markov Decision Process (MDP), where the state at time step n is the tuple (s, a²) and the agent's policy follows a a=-1 ~ t 1j-1 (-la"). The transition in this MDP is implicit and deterministic: upon selecting the action a=-1 the state immediately transitions from (s, a²)) to (s, 1)= At the last time step 0, the agent will receive an ending reward of Q(s, a(s, In this sense, the RL problem with Gaussian parameterized policy Behavior- Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
That is, through optimizing Eq. (24), the derived behavior diffusion policy is consistent with 2z-1/n from the forward process.
Next, we prove that the optimizer of Eq. (14) also satisfies Pn-1|n
=q:3 In-1/x"
problem defined in Eq. (14) can be recognized as behavior-regularzze
n. P/- and behavior policy Pn-1|n" Following Wu et al. (2019), the optimal diffusion value functions satisfy the recursion:
(a²) = Q(s, (s)
(29)
(a") = log E an 1np-in [exp(V (a""\/=1a"""
and the optimal diffusion policy is given by:
Pn-1/n (a Ham) exp( (0-
(am
P n. (a="(a"a11aa))
(30)
E Ean- tap [exp(V n- (ap(V==(a**~)) (aa-1)/1)
Expanding the recursion of diffusion value functions, have
(**"("") = 17 log 1app- [exp(V n-1 (aa-1)/m)
= log Egy lmp an 2xp
(a-mm/mm
(31)
17 log Baii
[exp(Vo'"aa))/)
17 log Eg Eai
[exp(Q(s.a")/m))
Similarly, the optimal policy can be expressed as:
Pa-1In (a"-l|a") Ba0 [exp(Q(s,
vis
P (am-1la²)) a²))
(32)
Ey Mam-laphe 1app
BaCnpe p (exp(Q(s,a9)/n))
E
Let's consider the following expression:
Barnii (exp(Q(s,a')/n)
= /
sojn (a exx("")))(x((())))))))))
v.8
|a """ (())s
exp(Q(xs ()))))))))/ 2 (a
(33)
In (a")
d
Z(s)
Ja (a
1
Egontg 1nj (a
Z(s)g" (a*)
where Z(s) is the normalizing factor, and the second equation follows from 1/ -
Substituting
17

---

Behavior- Regularizgd Diffusion Policy Optimization for Offline Reinforeement Learning
Boonpgn (exp(Q(s,a"))/n) and Paampopa (exp(((ss with Eq. (33) into Eq. (32),
P Pn-1jn (a (aaaa1"a")a )9" (am) BaooTo 10-110 (a (0000(a"110 (a9)
n/6 Pn-1/n (a" (a")
On 11-1(a" (1a0 BB
(nja(a" (am)
9njo vi
On 1/1 (a" (a- |a") (a*) ( Bagwar
(a 10 100)
0 (an (aa-1))Ba BaONTO
9n (40(aaaaa))
In|n (a (a" n- 1 In- 11-110aa-11000
(34)
E Injo (900(a/aa) (ao)
79.5 Jnln (a" 1a" n- "0""|a"1))B"0mss In- (0-111(a"~11100
( Te
EnenaR
Injo (a (ao)
9n-1/n (n-1In(a"*00 (a"),
TQ.8
where the second line follows from Pn-1/n vi
vi
= 1/n and the fourth line follows from the fact that Ø/6
v.a =g
9i
for any
i >>> j.
In conclusion, we obtain Pn (a" la")= In-1/n
(a- (a-=1la") meaning that the reverse process pa** exactly corresponds
to the forward process starting from (als)) xx v(als)
Theorem C.2 (Theorem 4.2 in the main text). Let p" be the behavior diffusion process. The optimal diffusion policy p" of the pathwise KL-regularized RL problem in Eq. (12) is also the optimal policy nt" of the KL regularized objective in Eg. (1), in the sense that n (a|s) = Po: ( U:NN (N:NN6((e )ō(a dda va E S, where ō the Dirac delta function.
Proof. Denote the optimal Q-value function of the pathwise KL-regularized RL as Q", According to Theorem C.I, for any state s S, p is the reverse process of the forward diffusion process = e Since the pr is also the reverse process of q"t², we have
0.8 DEL PoiN [Po:]] DKL [ * 2.5 [
(35)
For and V0:N share the same transition kernel and differ only in the initial distribution. Hence, Notice that 1o:N
DRL
[PoN I
DKL [oo:N ||
DKL [90 To * 119
+ DKL (iNijollo::
(36)
DKL [90 mig "19 2.3]
DKL [Tq. (-1s)][((-ss)]
Therefore, the optimal Q-value function Q' (s, a) satisfies
N.N M QQ" - nKk [P ]]
Q" Q" a) = -R(s, a) + YE
R[ss a) + y.. Ea (Q" a') DDI.. (To- (-1s') (((())))
(37)
R(s, a) + 7m log Eg (a (exp((((s')) (s', a')/n)))
which is exactly the optimal Bellman iteration of the problem defined in Eq. (1). This means that Q* also the optimal --vuuue function of the problem defined in Eq. (1), and σQ- is correspondingly the optimal solution л². Using Theorem C.1, we know that =** and x¹ are equivalent:
(als)) = / (a aN Es
(38)
The proof is completed.
18

---

Behavior- Behavior-Regularize Diffusion Policy Optimization for Offline Reinforeement Learning
Lemma C.3 (Soft Policy Evaluation (adapted from Haarnoja et al. (2018))). Consider the soft Bellman buckup operator 8 in Eq. (13) and function Q" : A + R. Define Q = 8 Qh. Then the sequence Qh will converge to the soft
Q-value of vt as k under Assumption 4.3.
Proof. Define the KL-augmented reward
N
Rd (s, a) = R(s, a) m N..N PooN
DKL
Pn-1In 11pp
T.o
1/m
and rewrite the policy evaluation update rule as
+ (*+) t (s, (((..) + + JoiN PPok (0"(s'aa²))
When (0,1), 1), this update satisfies the --contraction property, which is demonstrated in Haarnoja et al. (2018) to converge to a unique solution. Since Q satisfies Q" = 8 Q*, we obtain that the sequence Q* converges to Q". Note that we employ bounded probability ratio to ensure the boundedness of the Q-value function rather than requiring the action space to be a finite set.
Proposition C.4 (Soft Policy Improvement). Let pr be the optimizer of the problem defined in Eq. (16). Under Assumption 4.3, Va (a") 2 Vifiasi (a") holds for all {0, 1.. and (s,a) E ESX. A.
Proof. Since pm the optimizer of the problem 16, it satisfies
T (a")
E
V n2 (a=""((0"11)
(a"
VI ViTon:((a") (a*) S TT (a)")
E
Voodi (a=-1) (a
T. P
P
For O, recursively applying the above inequality leads
V0 VoTus va=))
(a")
(TM
N
R(s,aa) + 7IE
Q Quo (s) - -7
Masio (a")
a Po.N
oattio"
i=1
N
=R(s,a") + TEE
Vilus (a
oo (a't)
a PPPN
==1 Tanid (a
R((saa) >1E aNN ap
Tauio
-Rss,a)) y a.N ppppnn
Vaii (1) ne Taa (d
Taaial
RR(s,a)) yi, a ~p
Vaannds N-1 (a'N-1)- Te (a/Ni)
-R(s,a0) + TTE ae n
V Tacio N-2 N2
nerunr (a.N-1) - m/ Trow (a
< R(x,a²) + 7IF
(a IN N2) N-2
@tannd (a'N-1) ey (alar)
aix aIN- wo
-
N
< R() + y a Po.N
Vitueee (a=0) - >
naue (a
1=
< V.TTaans (a²),
where the fifth line follows from the definition of the diffusion value functions, the seventh line follows from the above inequality, and the last line follows from recursively expanding the values of successor states, For n. E (1,2,...; N). the
19

---

Behavior- Behavior-Regularize Diffusion Policy Optimization for Offline Reinforcement Learning
proof is similar:
V Vadi (a"))
= -mpea (a") + am
Tuaie n-1
(a (a=-1))
Pn-1/n
S n pie 8 (a") + EE iaa
V (a" (a"-1))
nn-
N (a*) + ree
17 M (a (a"-1) +
V V ( (a" (an
P
Pn-2
Bravi (aari((a") + Vauaao(aa))
1=1
E
g (a') + (a))
(a))
Faee
V. E (a")
where the last inequality is obtained previously. This completes the proof. Note that the = 0 case already guarantees the policy improvement in the environment MDP.
Proposition C.5 (Soft Policy Iteration (adapted from Haarnoja al. (2018))). Under Assumption 4.3, repeated application of soft policy evaluation in Eq. (13) and Eq. (15) and soft policy improvement in Eq. (16) from any p" -1l converges to policy p² such that V . (a)) 2 V.T. (a) for all p E II, n {0.1....,.N and (s,a) E S x A.
Proof. Let nd be the diffusion policy at iteration By Proposition C.4, the sequence Vi is monotonically increasing. Given that Vir finite due to the boundedness of rewards and the pathwise KI., the sequence converges to some p" Since the policy improvement has converged, it must be V *(a) Z V (a) for any diffusion policy p", (s,a) E e x A and n {0, I.. This completes the proof.
D. Continuous-Tine Perspective of the Pathwise KL
Song et al. (2021) build the connection between diffusion models and reverse stochastic differential equations (SDEs) that gradually remove noise from data and transform prior noise distribution into the target distribution. Based on the SDE understanding of diffusion models, we aim investigate whether the introduced pathwise KL can be extended to continuous-time diffusion modeling.
We define the forward diffusion SDE, which perturbs data into prior distribution, as the following:
dr [xx, ()dt + g(()dw,
(39)
where w is the standard Wiener process, f is the drift coefficient, and g is the diffusion coefficient. The reverse process of this SDE is also diffusion SDE:
= ( lo [x)] di + g())dw
(40)
where v is the reverse-time standard Wiener process and dl is a infinitesimal negative time interval. In practice, we can use a parameterized neural network s (z, t) to estimate the score of the marginal distribution using score matching:
0" = arg min (o,, [A(t]] (z((),)) (1)) log qnj (2)((((()))))))))) (=(0)=(()))
(41)
where 4no is the distribution of æ(t) given the initial sample x0)). After the training is completed, we can substitute the score function in Eq. (40) and solve the reverse SDE to obtain samples that follow the target distribution.
We consider two forward SDEs with the same drift and diffusion coefficient, but with different initial distributions qō and lh Let the approximated score networks be x and s", and therefore we consider the following reverse SDEs:
20

---

Behavior-Regularize Diffusion Policy Optimization for Offline Reinforcement Learning
1. dx [f(x,1) -1) di g()dd under measure PT.
2. dr -(()))))g(t}$+'' dt (t) under measure IP"
Our goal is compute the KL divergence between the distributions of the two reverse SDEs. Using Girsanov's theorem (Ok- sendal, 2013), have
d
DKL
1P
[Pd]]
Ep
log
dPP
[If - ( (=t)) [f s0 (x, t)] dt
Ep
(Girsanov's theorem)
2g(1)²
(42)
g(0)" (2r - (xx,, di
Ep-
0
2g(1)²
T
g())²
Ep
(x (x, sx, (x,,)))
2
Instantiating the SDE with Variance Preserving noise schedule (Song et al., 2021):
f(x,t) 1 3(t)a,
(43)
g(t) = B(t))
we have
DKL [P|| [P]]= = Egt |-"
3(t) 1|s (xx t) (x,,t))11
(44)
Eq. (42) presents the analytical form of the KL divergence between two reverse diffusion processes. In the following content, we continue to demonstrate that the pathwise KL introduced in this paper corresponds to Eq. (42) in the limit of N co. Examining the pathwise KL objective defined in Eq. (11),
DKL [po. N llp
",2" 1lp"""" DRL
P
1x=" n) - 1tt (x [x)
-Eps
20 n1
(45)
N
82
-Ep
[,,)(x²) (1"))
20 (1 Ba)
N
3a 2en)
Eppi
[15((²)) - (x0))
2(1
an (1 Br)
where 8n is the score function at step n.
Define At n. by Song et al. (2021), we can show that in the limit of N - oo, we have 3, 3( At
)t
21

---

Behavior- Bchavior-Regularize Diffusion Policy Optimization for Offline Reinforeement Learning
Runtime Comparison by Algorithm and Diffusion Steps
BDPO Critic
560
DAC Critic
500
Diffusion-QL Critic
BDPO Actor
DAC Actor
Time (minutes)
Difrusion-QL Actor
300
282
242
200
100
T-5
T-20
Figure 11. Algorithm Runtime of BDPO, DAC and Diffusion-QL.
x" x n2 N
and sn sn(x") = s(x( a2 N
a7 )
Therefore,
N
Bn -ān)
lim 100
DKL 1o..llPu.
lim Epg.v N
(177(2)) (x")11
2(1 ān-1) 3m)
Bn
lim Epg,
1 (x(" - "1
N
==1
(46)
nT
nT
nJ
nf
lim
E
s [x
s
(I<(
N
2
N
N
N
N
1=1
B(t)
Ep7
1s (x(t), - 5 (x(()))
2
which exactly corresponds to the KI. divergence of SDE in Eq. (42).
E. Supplementary Experiment Results
E.1. Discussions about Training Time
systematic breakdown and comparison of the runtime between 3DPO, DAC, and Diffusion-QL. is illustrated in Figure 11. We evaluate BDPO, DAC, and Diffusion-QL. with workstations equipped with NVIDIA RTX 4090 cards and the walker2d- mediumn-rella--v2 dataset.
Our method consists of three distinct subroutines. The first one is pre-training the behavior diffusion with standard diffusion loss. Our observation is that this phase accounts for a minor fraction (about 8 minutes) of the overall runtime, so we ignore in the following discussion. The second one is training value functions, including QT and V The approach to training Q" identical to established methods such as Diffusion-QUI (Wang et al., 2023) and DAC (Fang et al., 2024), which essentially sampling diffusion paths the next state s' and calculating the temporal difference target. In addition Q", method additionally trains V= However, this supplementary computation is not resource-intensive, it only requires single-step diffusion to compute the training target V/a Therefore, the additional cost of training diffusion value functions V is constant that does not scale with diffusion steps. The third subroutine is training the actor. Akin to EDP (Kang et al., 2023), BDPO only requires a single-step diffusion rollout, and thus the cost of training the actor is also a constant that does not scale with diffusion steps. In contrast, Diffusion- needs to differentiate with respect to the whole diffusion path, which causes a drastic increase actor runtime.
22

---

Behavior- Behavior-Regularize Diffusion Policy Optimization for Offline Reinforcement Learning
Inference Time Comparison
QGPD[Tarch)
5.73 ms
Diffusion-QL (((orch)
1.22 ms
DTQL(Torch)
0.41 ms
BDPO]AX]
0.31 ms
DAC[JAX]
0.30 ms
Uiftusion-QUjA1
0.25
JAX
Porch
5
Inference Time (ms)
Figure 12. Inference Time Comparison.
E.2. Discussion about Inference Time
Apart from training time, we also evaluate inference time since the latency is critical in real-world applications like robotic control. To benchmark the inference time accurately, we sampled 10,000 states from the dataset, exccuted the diffusion policies with a batch size I, and calculated the average inference time across 10,000 trials. Figure compares the inference speed of BDPO and baseline algorithms. Since the specific choice of deep learning backend may have significant impact on computational latency, we explicitly indicate the backend used for each result.
Among diffusion-baate based methods, Diffusion-QL. generates actions by traversing the reverse diffusion process, and BDPO and DAC generate N = 10 actions in parallel, followed by Avalue-based sample selection. Given that GPU acceleration effectively amortizes the cost of parallel generation and the calculation of Q values only requires inference through relatively small network, BDPO and DAC are about 10% slower than JAX-based Diffusion QL. In the meantime, implementations based on PyTorch are much slower than JAX-based implementations due to the dynamic graph execution. The computational overhead is most pronounced in QGPO, which incurs substantial inference latency due its classifier-guie guided generation process.
E.3. Generation Path On Synthetic 2D Tasks
We plot the generation results of BDPO on all synthetic 2D datasets in Figure 13. Throughout these datasets, we witness a close resemblance between the final sample distribution and the ground truth target distribution as depicted in Figure 9, The sampling begins with gradual movements across the 2D plane and converges in later diffusion steps. This pattern is further demonstrated by the diffusion value functions, which offer weaker guidance during the initial steps but provide stronger guidance in the final steps. On the contrary, the generation results (Figure 14) of DAC, which leverages Va a²) rather than the gradient diffusion values Va Va. (a") to guide the generation, fails to capture the target distribution. This is consistent with the findings from QGPO (Lu et al., 2023), which demonstrates that the guidance in BDPO is exact energy guidance, while the guidance used by DAC not.
E.4. Training Curves of D4RL Datasets
The curves of evaluation scores on D4RL datasets are presented in Figure 15 and Figure 16.
E.5. Ablation on Policy Parameterization
Full results of different policy parameterizations are presented in Figure 17, from which we found that the diffusion policy achieves the best overall performance. Note that we kept the hyperparameter tuning effort the same across different methods (see Section B.3).
23

---

Behavior-Regulariee Regularized Diffusion Policy Optimization for Offline Reinforeement Learning
Behavior-
Bgaussians ni n=40
misdd n-
ni:30 2-
1o:20 10
ni30 D
Actor sample
2spirals
n-50 ninto
ni40 ni300
B=20
actbr sample
checkerboard
n=50
7o-20
moons
rings
n-10
samole
p-46d
p-30
64
02
0.0
swissroll n-50
1.0
m-it n=30
ACTOY sammile
n-40
0.6
final samples
Figure 13. Illustration of the diffuusion policy and the diffusion value function from EDPO on synthetic 2D datasets. The regularization strength set 17 0.06.
24

---

Behavior-Regularie Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
Behavior-
Bgaussians
1.3
ni500
nid0
ni
n-10
sctor sample
n-30
n=20
0.8
0.6
0.4
0.2
0.0
2spirals
n-50 nw460
ni40
30
n-
n=10
sctor sample
n=30
n=
n-12
0.8
chackerboard
sample
moons
p=440
0.8
0.6
0.4
0.2
0.0
rings
n=50
actor vample
1.0
p=40
02
0.5
0.8
0.4
0.2
swissrell
1.0
m-50
n-33 n-
n-20
1=10 0w0
4ctor satple
n-40
n-30
n-
0.6
n=50 n=40
n=40 n0
n=30 n=
n= - n=10
n=10 n=0
final samples
Figure 14. Illustration of the diffusion policy and the Q-value function from DAC on synthetic 2D datascts. The regularization strength is set 7 0.06.
25

---

Behavior- Bchavior-Regularizet Diffusion Policy Optimization for Offline Reinforeement Learning
halfchestah-mediumyy
hailcheetth-meddiunrreepaayy
halichettl-mediumeeeprrtttt
Normalized Score
70
100
50
50
20
40
0.5M
2M
1M
1.5M
0.5M Training Steps
1.5M
2M
Training Steps
Training Steps
hopper-medium .2
hopper
hopper-medium-eperrr expert-v2
100
100
100
50
50
0
0.5M
1.5M
2M
0.5M
1M
1.5M
0.5M
1.5M
2M
Training Steps
Training Steps
Training Steps
waller22--medium-rreaaay--
wwllkerz--medduumxxxeettzz
100
80
100
60
0
0.5M
1M
1.5M
2M
0.5M
1M
1.5M
2M
0
0.5M
1M
1.5M
2M
Training Steps
Training Steps
Training Steps
Figure 15. Evaluation of BDPO D4RL Locomotion datasets. Results aggregated using independent seeds and 10 evaluation episodes for each seed.
antmaze-umaze-vv
antmaaze-umaze-divrreew
antmmaze-medium-playye
Normalized Score
75
50
60
25
0
1.5M
2M
0.5M Training Steps
1.5M
2M
0.5M Training Steps
1.5M
2M
Training Steps
antraaze-medium-diivess-o-
antmaze-large-play..0
antmaze--arge-diivrse-o
75
50
50
25
0
0
1.5M
2M
1.5M
2M
1.5M
2M
Training Steps
Training Steps
Training Steps
Figure Evaluation scores BDPO on D4RL Locomotion datasets. Results are aggregated using independent seeds and 100 evaluation episodes for each seed.
26

---

Behavior- Bchavior-Regularized Diffusion Policy Optimization for Offline Reinforeement Learning
haifcheetah-meduum-2
hopper-medium-v2
walker22dmeeluum-2
Normalized Score
75
100
50
75
50
BDPO
25
Deterministic
50
Gaussian
0.5M
1M
1.5M
2M
0.5M
1M
1.5M
2M
0.5M
1M
1.5M
2M
Training Steps
Training Steps
Training Steps
aopper-madium-rpplayyy
walker22d medium-roplay--ee..
Normaiized Score
60
100
100
50
50
20
0.5M
1.5M
2M
0.5M
1M
1.5M
0.5M
1.5M
2M
Training Steps
Training Steps
Training Steps
halfcheetah
halfcheetah-medium medium expert.v2
hopper- meeere--edmmmmmmm expert-v2
walker2d medium expertiv2
Normalized Score
100
100
100
50
50
50
0
1.5M
2M
2M
Training Steps
Training Steps
Training Steps
Figure Ablation study with the policy parameterizations. Results are aggregated using 5 independent seeds and evaluation episodes for each seed.
hopper-medium--2
walker266-meiumm-z
Normalized Score
100
100
80
80
60
step=2
40
20
step=5
step=10
20
0
0.5M
1M
1.5M
2M
0.5M
1M
1.5M
2M
Training Steps
Training Steps
hopper-medium-replayy.2
hopper-medimm-expert/
125
Normalized Score
100
100
75
50
50
25
25
0
0.5M
1M
1.5M
2M
0
0.5M
1M
1.5M
2M
Training Steps
Training Steps
Figure Ablation study with the number of diffusion steps Results are aggregated using independent seeds and evaluation episodes for each seed.
27

---

Behavior- Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
E.6. Ablation on Diffusion Steps N
The number of diffusion steps N impacts both generation quality and divergence calculation. We empirically evaluate three configurations (N 2, 5, and 10) in Figure 18. Generally, the performance of BDPO degrades substantially when N is too small; however, the improvement diminishes when N exceeds certain threshold. Based on these observations, adopt N = as our default choice to trade off the efficiency and performance.
28


Reviews:
[{"review_id": "7mRsampWt5", "reviewer": "Reviewer_3ubG", "summary": "The paper introduces Behavior-Regularized Diffusion Policy Optimization (BDPO), a framework for offline RL that integrates diffusion-based policies with behavior regularization. The key innovation is formulating KL regularization along the whole diffusion steps instead of on the final result, enabling more effecient computation. The authors propose a two-time-scale actor-critic algorithm that optimizes value functions, which further improves computational efficiency and training stability. Theoretical results establish equivalence between the pathwise KL regularization and the standard KL-regularized RL objective. Experiments on synthetic 2D tasks and D4RL benchmarks demonstrate superior performance compared to baseline methods, particularly in locomotion and antmaze navigation tasks.\n\n## update after rebuttal\n\nThe rebuttal has addressed most of my concerns. Given that the paper is strong with relatively minor deficits, I have decided to maintain my original rating of weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for his/her constructive feedback. Below is further clarification for the reviewer's concern. \n\n**Q1: Using EDP for policy optimization?**\n\nOur policy improvement objective is to maximize the expected Q-values while also minimizing the KL divergence:\n$$\\max\\_{p^{s,\\pi}}\\ \\mathbb{E}\\_{a^0\\sim p^{s,\\pi}\\_0}[Q(s, a^0)]-\\eta \\mathrm{KL}\\left[p^{s,\\pi}\\_{0:N}\\|p^{s,\\nu}\\_{0:N}\\right]$$\nLet us ignore the second KL term for now, since EDP implements this constraint using diffusion loss. For the first term, in order to circumvent backpropagating the gradient of Q through the diffusion path, EDP introduces the following action approximation:\n$$\\hat{a} = \\frac{1}{\\sqrt{\\bar{\\alpha}\\_n}}a^n - \\frac{\\sqrt{1-\\bar{\\alpha}^n}}{\\sqrt{\\bar{\\alpha}^n}}\\epsilon^{\\pi,s}(a^n, n)$$\nwhere $\\epsilon^{\\pi,s}$ is the output of the score network. Afterwards, they use $\\hat{a}$ as the approximation for $\\mathbb{E}\\_{a^0\\sim p^{s,\\pi}\\_{0|n}}[Q(s, a^0)]$. However, we emphasize that this approximation is **inexact and biased**. The action approximation is essentially the mean of the posterior distribution:\n$$\\hat{a}=\\mathbb{E}\\_{a^0\\sim p^{\\pi,s}\\_{0|n}}\\left[a^0\\right]$$\nmeaning that they are using $Q(s, \\mathbb{E}\\_{a^0\\sim p^{\\pi,s}\\_{0|n}}\\left[a^0\\right])$ to approximate $\\mathbb{E}\\_{a^0\\sim p^{s,\\pi}\\_{0|n}}[Q(s, a^0)]$, which is biased and inconsistent with our theory. \n\n**Q2: How to select the action with the highest Q value? What about the inference time?**\n\nYes, we use the average of the ensemble Q networks to select the actions. During inference, BDPO first generates $N_a=10$ candidate actions **in parallel**, calculates their $Q$-values and selects the best one. The following table presents the inference latency per state, averaged over 100K trials:\n\n|Algorithm|Inference Time (ms)|\n|:-----:|:-----:|\n|BDPO (JAX)|0.310|\n|DAC (JAX)|0.298|\n|Diffusion-QL (JAX)|0.251|\n|Diffusion-QL (PyTorch)|1.22|\n|DTQL (PyTorch)|0.411|\n|QGPO (Pytorch)|5.73|\n\nWe found that the inference cost of BDPO is comparable to DAC, which also generates 10 actions and selects the best one with the highest Q-value. The JAX implementation of Diffusion-QL is faster than BDPO, since it only generates one action and does not query the Q-values. The PyTorch version of Diffusion-QL is much slower. For DTQL, since its policy network is simply a one-step policy, its inference time is comparable to BDPO. Finally, QGPO requires taking the gradient of the Q-value network to calculate the guidance, which results in heavy computation overhead during inference. \n\n**Q3: About hyperparameters**\n- **The range of parameter sweeping for eta and rho?**\n  \n  For locomotion tasks, we swept eta and rho among {0.5, 1.0, 1.5, 2.0}. For antmaze tasks, we swept rho among {0.5, 0.8, 1.0} and eta among {1.0, 5.0, 10.0}. \n- **The different trend w.r.t. the hyper-parameters in halfcheetah and walker2d tasks, and what about other datasets like medium-replay?**\n  \n  The sensitivity analysis of eta and rho for medium-replay tasks is provided in the following anonymous links: https://f005.backblazeb2.com/file/bdpo-review/rebuttal_abla_eta.pdf and https://f005.backblazeb2.com/file/bdpo-review/rebuttal_abla_rho.pdf . Overall we found that the trend is similar across most of the datasets, that is, excessively large or small $\\rho$ and $\\eta$ may result in fluctuation or degradation in the performance. The halfcheetah-medium-v2 dataset seems to be an exception in that it is more tolerant to extreme parameter values. \n - **Setting $\\eta=0$ or $\\rho=0$?**  \n  For results with $\\rho=0$, please refer to the above link to ablation w.r.t. rho. For results with $\\eta=0$, we additionally provide curves on medium-expert datasets, and the results are provided in the anonymous link: https://f005.backblazeb2.com/file/bdpo-review/rebuttal_eta0.pdf . The lower confidence bound technique is conceptually similar to the commonly adopted double Q-network trick that penalizes the Q-values of OOD actions, and therefore when there is no LCB penalty, the performance drops sharply due to severe over-estimation. Setting $\\eta=0$ also results in performance fluctuation and decrease for walker datasets due to insufficient constraint. However, in halfcheetah datasets, $\\eta=0$ improves performance, likely because LCB already penalizes out-of-distribution (OOD) actions effectively.\n - **About the sensitivity to rho and advice on choosing hyper-parameters?**  \n  After inspecting the training details, we observed that an excessively small $\\rho$ leads to severe over-estimation in Q-values, while an excessively large $\\rho$ causes severe underestimation instead, both of which result in performance degradation. We found that a $\\rho$ value yielding stable value estimations generally correlates with strong performance. Thus, we recommend to first adjust $\\rho$ until stable value estimations are achieved, and then gradually decrease $\\eta$ to strike a balance between robustness and performance.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes. I have checked all proofs for theoretical claims.", "experimental_designs_or_analyses": "Yes. I have checked all experimental designs and analyses.", "relation_to_broader_scientific_literature": "The work builds on two key strands: behavior regularization and diffusion policies.\n- Behavior regularization is commonly used in both online and offline RL to provide an expected objective to the policy. Specifically, previous offline RL works usually utilize the KL divergence between the learning policy and the behavior policy. This paper extends behavior regularization to intermidiate diffusion steps, while keeping high computational efficiency through the two-time-scale optimization scheme. This scheme resembles DPPO but incorporates penalty in each diffusion step.\n- The diffusion policies model RL policies by diffusion models, instead of traditional Gaussion policies, As diffusion models enjoy better capability on complicated multi-modal distribution, they have surpassed their counterparts using Gaussion policies. This paper proposes a novel regularization and training scheme to further improve the performance of diffusion policies, while keeping the training complexity relatively low.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths\n- The method is novel on incorporating behavior regularization on each diffusion step, while maintaining training efficiency by the two-time-scale scheme. \n- The theoretical proof is clear and well-written, which links pathwise KL to standard KL regularization and continuous-time SDEs.\n- The empirical results across diverse tasks are strong.\n\nWeaknesses: see questions.", "other_comments_or_suggestions": "Here are some possible typos:\n- Title of Figure 8: \"Illuration\" -> \"Illustration\".\n- Title of Figure 7(a): $\\beta$ -> $\\eta$.\n- Eq. (18), $V(s,a)$ -> $V(a^n)$.", "questions_for_authors": "- The bi-level TD-learning framework is proposed to avoid preserving the computation graph of all diffusion steps like Diffusion-QL. I wonder have you tried the optimization in EDP[1], which improves the efficiency of Diffusion-QL and also avoid perserving the whole computation graph. Is their method inconsistent with you behavior regularization objective?\n- As you use an ensemble of Q networks, how do you select the action in evaluation by the highest Q value? Do you use the mean of these Q values? Also, could you please report the inference time of BDPO, and compare it with your main baselines?\n- For two key hyperparameters $\\beta$ and $\\rho$\n  - In what range do you search $\\eta$ and $\\rho$?\n  - Figure 7(a) shows that the trend of BDPO is completely different w.r.t. $\\eta$ and $\\rho$ in Halfcheetah and Walker2d tasks, though they are both medium dataset. Does this stem from some properties of these two environment? Also, is this trend similar to those on datasets other than medium (e.g., medium-replay)?\n  - It seems that smaller $\\eta$ in your tested range leads to better performance in Halfcheetah-medium. Could you please test $\\eta=0$, i.e., no behavior regularization? The same for $\\rho=0$, i.e., no LCB penalty.\n  - It seems that BDPO is sensitive to $\\rho$, as shown in Figure 7(b). Adjusting $\\rho$ from $1.0$ to $0.75$ or $1.5$ will lead to significant drop in performance on Walker2d-medium. Could you please explain this phenomenon?\n  - Finally, as exaustively tuning hyperparameters are impractical in many real tasks, could you please provide some insight to how to choose hyperparameters, or at least shrink the search range, for unseen tasks by their task properties?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AKzDwyEavE", "reviewer": "Reviewer_iYRy", "summary": "This paper use diffusion policis for offline RL and the main idea is to take the rollout quality of the diffusion process as extra regularization. In other words, for behavior cloning, the paper proposes to measure the similarity between the demonstration action and the learnt action by comparing their distributions of trajectories generated by the revserse procedure of the diffusion process. They show that such difference can be accumulated as another scale of value function and is useful to behavior cloning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing valuable feedback. Below, we provide further clarification and results to address your concern and we hope these materials can enhance your evaluation of our paper. **Due to the space constraint, we will post our discussion of the connection between BDPO and broader literatures (Q6 and Q8) in follow-up responses.** You can also access the responses in the anonymous link: https://f005.backblazeb2.com/file/bdpo-review/Q6-andQ8.txt . \n\n**Q1: Can an optimal policy be delivered by optimizing eq.11?**\n\nWe would like to clarify that Eq. 1 is also a **well-established RL objective widely adopted in the literature**, and therefore, optimizing Eq. 11 delivers optimal policies with respect to Eq. 1. Please refer to our response to Q5 for details. \n\n**Q2:  The pathwise KL regularization is stronger, making the policy suffer from sub-optimal data?**\n\nYes, BDPO additionally regularizes intermediate actions along the diffusion path. However, we emphasize that this is **not a stronger regularization**, as we have established the equivalence between pathwise and action-wise constraint in Theorem 4.2. Given this guarantee, the pathwise constraint is in fact preferable as it enables a finer-grained control over the generation process. Furthermore, when the behavior policy is suboptimal, the regularization strength $\\eta$ can be decreased to allow greater exploitation from the value function.\n\n**Q3: Intermediate actions are not present in the dataset.**\n\nWhen training value functions $V^{\\pi, s}\\_n$ for $n>0$, we optimize them using actions $a^n$ generated by first sampling clean actions $a^0$ from the dataset and then perturbing them using the forward diffusion process $a^n\\sim q\\_{n|0}(a^0)$. Consequently, **the action support for $V^{\\pi,s}\\_{n}$ with $n > 0$ is infinite and spans the entire action space** due to the unbounded support of the Gaussian noise distribution. The only risk of OOD evaluation comes from querying $V^{\\pi,s}_0(a^0)$ for some $a^0$ generated by the actor. However, this challenge is inherent to all policy iteration methods and can be effectively mitigated by tuning the regularization strength. \n\n**Q4: The assumption about concentrability is stronger?**\n\nWe clarify that Assumption 4.3 is made only to ensure the boundedness of the KL divergence in our theory, and it is not the concentrability used in papers like the reviewer mentioned. Similar assumptions are also made in papers that incorporate behavior regularization -- for example, SAC assumes $|\\mathcal{A}| < \\infty$ to ensure the policy entropy is bounded.\n\nRegarding the concentrability, as discussed in Q3, the marginal distribution $p^{\\nu,s}_n$ at $n>0$ has infinite support over the action space due to Gaussian perturbations. Consequently, the concentrability requirement reduces to the base case of $n=0$, meaning BDPO does not impose stronger concentrability assumptions than other methods. Our primary contribution is providing a practical implementation of the behavior-regularization RL for diffusion-based policies, rather than introducing new theoretical bounds.\n\n**Q5: The optimal policy differs from other works.**\n\nThe optimal policy differs because we consider the **behavior-regularized RL framework** (Eq. 1), which augments the standard RL objective with KL divergence. This framework helps us to shape the policy and is widely adopted in RL. For example, when $\\nu$ is specified as the uniform distribution or the dataset collection policy, the framework becomes MaxEnt RL (used by SAC) or regularized RL (used by offline RL methods like ReBRAC and XQL), respectively. Other offline RL algorithms (e.g., TD3-BC, AWAC) can also be categorized into this framework if we omit the KL divergence term during policy evaluation. This framework is also adopted by applications like RLHF, where KL regularization towards the reference model prevents model collapse. Therefore, we believe it is important to study this framework.\n\n**Q6: Lack of theoretical comparison with other diffusion-based works.**\n\nSee follow-up responses or the link. \n\n**Q7: Experiment results on random and expert datasets**\n\nWe compare BDPO against several baselines, including a previous model-free SOTA, ReBRAC and the best performing diffusion-based method, DAC. All results are averaged over 4 seeds.\n\n|Dataset|IQL|CQL|ReBRAC|DAC|BDPO|\n|-|-|-|-|-|-|\n|hc-random|19.5|31.1|29.5|28.6|28.6±0.9|\n|hc-expert|95.5|97.3|105.9|103.4|105.5±2.4|\n|hop-random|10.1|5.3|8.1|8.4|15.0±9.7|\n|hop-expert|108.8|106.5|100.1|98.6|110.2±2.3|\n|walk-random|11.3|5.1|18.4|4.1|0.61±0.15|\n|walk-expert|96.9|109.3|112.3|113.5|110.4±0.1|\n|sum|342.1|354.6|374.3|356.6|374.4|\n\nOverall, BDPO matches SOTA performance (ReBRAC) and outperforms DAC, though the margin is narrow. Note that model-free offline RL methods tend to struggle on random datasets due to the inferior data quality. \n\n**Q8: Lack of references about imitation learning**\n  \nSee follow-up responses or the link.", "claims_and_evidence": "1. To justify the proposed method, the paper says that optimizing eq.11 is equivalent to eq.1, which means that the behavior of the proposed method would be like a behavior clone method which takes eq.1 as its objective. However,  what we really interested in is the optimality condition under the offline RL setting - in other words, can an optimal policy be delivered by optimizing eq.11?\n\n2. The proposed method provides a stronger regularization for the divergence between the new policy and the behavior policy with the pathwise constraint. This would make the policy more likely to be suffered from the bad behaviors generated by the sub-optimal behavior policy. Could the author discuss more about this point?\n\n3. the authors consider the value of the actions generated during the diffusion procedure, however, such actions are artifical and are never been actually taken in practice, so how can you ensure that they would be appeared in the offline dataset? (assumption 4.3) In this sense, lots of OOD action would be evaluated by the proposed method, how to ensure the algorithm's stablity?", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "1. Assumption 4.3. This assumption is like the well-known Concentrability assumption, which means that the dataset should cover the visitation states of any policies from the candidate set. When the set is large enough, the coverage of dataset would be very large, which is hard to realize in practice. As a result, such assumption has been aborted in recent works, where only the optimal coverage assumption should be given. However, in our opinion, the Assumption 4.3 is even more stronger than the Concentrability assumption, because it assumes the coverage at each diffusion step. This makes the theoretical results hard to generalize to real-world scenarios.\n\n2. The 'optimal policy' in Theorem 4.2 seems different from that in other works, such as [Bellman-consistent Pessimism for Offline Reinforcement Learning], where the optimal policy is the global optimal policy in the true MDP. Therefore, Theorem 4.2 only guarantees the equivalence of Eq.(1) and (11), in my opinion, which is not so interesting as the analysis of the optimality of the learnt policy.\n\n3. Lack of the theoretical comparison with other works, especially with those works about diffusion-based policy.", "experimental_designs_or_analyses": "In Table 1, there lack the results on 'random' and 'expert' benchmarks, which is important as well.", "relation_to_broader_scientific_literature": "see below", "essential_references_not_discussed": "1. In the literature of deep learning on teacher student network, there are several works about recording the learning trajectory of the tearcher network, and asking the student network to imitate that learning behavior. This paper obviously follows the same idea - the new policy should imitate the diffusion trajectory of the behavior policy. So I think the paper lacks the reference on such works.", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "n/a", "questions_for_authors": "see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9XlC8fVX0v", "reviewer": "Reviewer_jbHb", "summary": "This paper proposes a diffusion model optimization method based on multi-diffusion-step regularization, which is different from previous behavior-regularized policy methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful feedback, which has greatly improved our work. We hope the following clarifications can further address your concerns and enhance your evaluation of our paper. \n\n**Q1: About the proof in Theorem C.1**\nFirst, we would like to restate the question to address any misunderstandings. Let $Q(s, \\cdot)$ denote the ending reward in the diffusion MDP with a discount factor of 1. The reviewer considers the following relationships between the value functions:\n$$\\begin{aligned}\nV^{\\star, s}\\_0&=Q(s, a^0)\\\\\\\\\nQ^{\\star, s}\\_0&=Q(s, a^0)\\\\\\\\\nV^{\\star, s}\\_n(a^n) &= \\eta\\log\\mathbb{E}\\_{a^{n-1}}\\left[\\exp(Q^{\\star, s}\\_n(a^n)/\\eta)\\right]\\\\\\\\\nQ^{\\star, s}\\_n(a^n)&=0 + 1.0\\mathbb{E}\\_{a^{n-1}}\\left[V^{*, s}\\_{n-1}(a^{n-1})\\right]\n\\end{aligned}$$\nwhich seems to contradict our results in line 820. However, we believe the main confusion here is that in the diffusion MDP, the “state” is actually the tuple $(s, a^{n})$ and the state transition is implicit. The policy follows the Gaussian distribution $p^{\\pi,s}(a^{n-1}|a^n)$, and upon selecting action $a^{n-1}$, the state instantly transitions to $(s, a^{n-1})$ deterministically. Therefore, the $Q$ function of the diffusion MDP should be defined w.r.t the “state” $(s, a^n)$ and action $a^{n-1}$, leading to the corrected relationships:\n$$\\begin{aligned}\nV^{\\star, s}\\_0(a^0)&=Q(s, a^0)\\\\\\\\\nV^{\\star, s}\\_n(a^n) &= \\eta\\log\\mathbb{E}\\_{a^{n-1}}\\left[\\exp(Q^{\\star, s}\\_n(a^n, a^{n-1})/\\eta)\\right]\\\\\\\\\nQ^{\\star, s}\\_{n}(a^n, a^{n-1})&=0 + 1.0V^{\\star, s}\\_{n-1}(a^{n-1})\n\\end{aligned}$$\nSubstituting the last equation into the second leads to the formulation in line 820\n$$V^{\\star, s}\\_n(a^n) = \\eta\\log\\mathbb{E}\\_{a^{n-1}}\\left[\\exp(V^{\\star, s}\\_{n-1}(a^{n-1})/\\eta)\\right]。$$\nAs for question (2), expanding the recursion gives\n$$\\begin{aligned}\nV^{\\star, s}\\_n(a^n) &= \\eta\\log\\mathbb{E}\\_{a^{n-1}\\sim p^{\\nu,s}\\_{n-1|n}}\\left[\\exp(V^{\\star, s}_{n-1}(a^{n-1})/\\eta)\\right]\\\\\\\\\n&= \\eta\\log\\mathbb{E}\\_{a^{n-1}\\sim p^{\\nu,s}\\_{n-1|n},a^{n-2}\\sim p^{\\nu,s}\\_{n-2|n-1}}\\left[\\exp(V^{\\star, s}\\_{n-2}(a^{n-2})/\\eta)\\right]\\\\\\\\\n&=\\ldots\\\\\\\\\n&=\\eta\\log\\mathbb{E}\\_{a^{n-1}\\sim p^{\\nu,s}\\_{n-1|n}, \\ldots, a^0\\sim p^{\\nu,s}\\_{0|1}}\\left[\\exp(V^{\\star, s}\\_{0}(a^{0})/\\eta)\\right]\\\\\\\\\n&=\\eta\\log\\int p^{\\nu,s}\\_{0,1,\\ldots,n-1|n}(a^0, a^1, \\ldots, a^{n-1}|a^n)\\exp(V^{\\star, s}\\_0(a^0))\\mathrm{d} a^{n-1}a^{n-2}\\ldots a^{0}\\\\\\\\\n&=\\eta\\log\\int p^{\\nu,s}\\_{0|n}(a^0|a^n)\\exp(V^{\\star, s}\\_0(a^0))\\mathrm{d}a^{0}\\\\\\\\\n&=\\eta\\log\\mathbb{E}\\_{a^{0}\\sim p^{\\nu,s}\\_{0|n}}\\left[\\exp(V^{\\star, s}\\_{0}(a^0)/\\eta)\\right]\n\\end{aligned}$$\nwhere the last but one equation is due to the marginalization over intermediate actions $a^1, a^2, \\ldots, a^{n-1}$.  We acknowledge that the derivation in the appendix is somewhat vague and will revise it to explicitly incorporate a proof akin to MaxEnt RL theory for clarity.\n\n**Q2: About the update of value networks.**  \nThe detailed update procedure of the value networks consists of two steps:  \n1 ) The first step is updating $Q^{\\pi}$. To calculate the target, we use the actor diffusion policy to generate paths $a’^{0:N}$ at the next state $s’$, calculate the target value $Q(s’, a’)$ and accumulated penalties along the path $\\sum_{n=1}^N\\ell^{\\pi,s'}_{n}(a^n)$, and perform temporal difference update as per Eq. 12;   \n2 ) The second step is updating $V^{\\pi,s}$. This is achieved by sampling $n$ and $a^n$ using forward process, performing one-step diffusion to obtain $a^{n-1}$, and update according to Eq. 14. For $n=1$, we directly use $Q(s, a^0)$ as the target, rather than additionally regressing the output of $V^{s}_0(\\cdot)$ to $Q(s, \\cdot)$.\n\nCrucially, the update of $Q^{\\pi}$ does not depend on $V^{\\pi,s}_N$. The diagram of the update is presented in Figure 3. Compared to other methods, the additional cost w.r.t. value function training comes from the second step, which is a constant that does not scale with diffusion steps (see Figure 10). \n\n**Q3: Eq. 7 should be $q\\_{n-1|n}(x^{n-1}|x^n,x^0)$**  \nWe appreciate the suggestion and would like to note that the actual distribution we wish to approximate is the posterior without conditioning on $a^0$, as $a^0$ is unknown during generation. When $a^0$ is given, the posterior distribution $q\\_{n-1|n, 0}$ is tractable, with its analytical form being: \n$$q\\_{n|n-1, 0}(a^n|a^{n-1}, a^0)=\\mathcal{N}(a^{t-1}; \\frac{\\sqrt{\\bar{\\alpha}\\_{n-1}}\\beta_n}{1-\\bar{\\alpha}\\_n}a^0+\\frac{\\sqrt{\\alpha\\_n}(1-\\bar{\\alpha}\\_{n-1})}{(1-\\bar{\\alpha}\\_n)}a^n, \\sigma\\_n I).$$\nSince exact $q\\_{n-1|n}$ is intractable, we turn to a parameterized distribution $p^\\theta\\_{n-1|n}$ and optimize it towards $q\\_{n-1|n, 0}$ via in Eq. 10. As shown in DDPM, this training objective yields $p^\\theta\\_{n-1|n} \\approx q\\_{n|n-1}$. We recognize that the current presentation lacks clarity and will revise the text to explicitly articulate the connection between $q\\_{n-1|n, 0}$ and $q\\_{n-1|n, 0}$ in the next version.", "claims_and_evidence": "Most of the claims in this paper are well supported by theory and experiments, but some parts remain difficult to understand. Please refer to the questions and comments.", "methods_and_evaluation_criteria": "This paper compares several classic offline RL methods. The choice of experimental environments is appropriate.", "theoretical_claims": "I have read the theoretical parts related to the paper, including both the main body and the appendix. However, some questions still remain. I have summarized them in the questions and comments.", "experimental_designs_or_analyses": "Please refer to the questions and comments for experimental concerns.", "relation_to_broader_scientific_literature": "Previous studies on behavior-regularized policy optimization often apply end-to-end regularization directly on the policy, i.e., focusing on the output actions. In contrast, for expressive diffusion models, this paper considers applying regularization at each generation step of the diffusion model, achieving better-constrained policy optimization.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Please refer to the questions and comments for review concerns.", "other_comments_or_suggestions": "1. In line 817: Regarding the generation process as a decision sequence and the final step reward is Q(s, a^0), then the optimal value function V^{\\*,s}_0=Q(s,a^0), Q^{\\*,s}_0=Q(s,a^0)$, and $V^{\\*,s}_{n}(a^n)=\\eta*\\log \\mathbb{E}_{a^{n-1}}[exp(Q^{\\*,s}_n(a^n)/\\eta)], Q^{*,s}_n(a^n)=0+1.0*\\mathbb{E}_{a^{n-1}}[ V^{*,s}_{n-1} (a^{n-1})]. (1) The formula of V^{*,s}_n(a^n) is different from the formula of line 820. (2) From the recursion, there exists expectation of a^{n-1}, a^{n-2},…, but I can not find the expectation in Equation (26). I suggest the authors show details of the derivation here to help readers understand these important results.\n2. In Algorithm 1: If understanding right, when updating the related networks, the training process contains the following steps: 1) Calculate the value Q(s, a^0) and use the results as target to train value V(s, a^0); 2) Get perturbed data a^n by performing forward diffusion process; 3) Run diffusion once to obtain a^{n-1} and use a^n and a^{n-1} to update V function; 4) Calculate the Q target: R+\\gamma*V(s, a^n) and update the Q function parameters. If the above process is right, I suggest the author revise Algorithm 1 to make the training process much clearer.", "questions_for_authors": "1. To better understand the background of diffusion models and correspond to the contents mentioned in line 142 right column, I suggest to write Equation (7) as q_{n-1|n}(x^{n-1}|x^n,x^0) rather than q_{n-1|n}(x^{n-1}|x^n).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Summary:**  
This paper presents Behavior-Regularized Diffusion Policy Optimization (BDPO), a principled and efficient offline RL framework that integrates diffusion-based policy modeling with pathwise behavior regularization. By introducing KL regularization across intermediate diffusion steps and employing a two-time-scale actor-critic optimization, the method achieves strong theoretical grounding, efficient training, and superior empirical performance across standard benchmarks.

**Pros:**  
1. Proposes a novel and elegant formulation of diffusion policy regularization that improves stability and expressiveness.  
2. Achieves strong results across a range of D4RL benchmarks, supported by clear ablation studies and insightful analyses.  
3. Theoretical contributions, including the equivalence of pathwise KL regularization and standard objectives, are well-articulated and rigorous.

**Cons:**  
Some assumptions may be restrictive in practice, and additional implementation details and comparisons with related baselines would further strengthen the work.
Decision: Accept (poster)