Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning
Authors: Linjiajie Fang
Ruoxue Liu
Jing Zhang
Wenjia Wang
Bingyi Jing
Venue: ICLR 2025 Poster (2025)
Keywords: Offline Reinforcement Learning
Diffusion Models
Actor-critic Learning
Policy Iteration
Policy Regularization
Primary area: reinforcement learning
TL;DR: Representing target policies using diffusion models that addresses KL-constrained policy iteration problem and achieves stable convergence and strong performance.

Abstract: In offline reinforcement learning, it is necessary to manage out-of-distribution actions to prevent overestimation of value functions. One class of methods, the policy-regularized method, addresses this problem by constraining the target policy to stay close to the behavior policy. Although several approaches suggest representing the behavior policy as an expressive diffusion model to boost performance, it remains unclear how to regularize the target policy given a diffusion-modeled behavior sampler. In this paper, we propose Diffusion Actor-Critic (DAC) that formulates the Kullback-Leibler (KL) constraint policy iteration as a diffusion noise regression problem, enabling direct representation of target policies as diffusion models. Our approach follows the actor-critic learning paradigm in which we alternatively train a diffusion-modeled target policy and a critic network. The actor training loss includes a soft Q-guidance term from the Q-gradient. The soft Q-guidance is based on the theoretical solution of the KL constraint policy iteration, which prevents the learned policy from taking out-of-distribution actions. We demonstrate that such diffusion-based policy constraint, along with the coupling of the lower confidence bound of the Q-ensemble as value targets, not only preserves the multi-modality of target policies, but also contributes to stable convergence and strong performance in DAC. Our approach is evaluated on D4RL benchmarks and outperforms the state-of-the-art in nearly all environments.

Published as a conference paper at ICLR 2025
DIFFUSION ACTOR-CRITIC: FORMULATING CONSTRAINED POLICY ITERATION AS DIFFUSION NOISE REGRESSION FOR OFFLINE REINFORCEMENT LEARNING
Linjiajie Fang' Ruoxue Liu²
Jing Zhang'
Wenjia Wang
Bing- Yi Jing
Hong Kong University of Science and Technology
2 Hong Kong University of Science and Technology (Guangzhou)
3 Southern University of Science and Technology
ABSTRACT
In offline reinforcement learning. is necessary manage out-of-distributin ac- tions to prevent overestimation of value functions. One class of methods, the policy-regularized method, addresses this problem by constraining the target pol- icy to stay close to the behavior policy. Although several approaches suggest representing the behavior policy as an expressive diffusion model to boost perfor- mance, it remains unclear how to regularize the target policy given diffusion- modeled behavior sampler. In this paper, we propose Diffusion Actor-Critic (DAC) that formulates the Kullback-Leibler (KL) constraint policy iteration a diffusion noise regression problem, enabling direct representation of target poli- cies as diffusion models. Our approach follows the actor-critic learning paradigm in which we alternatively train a cnffiusionmmodlles target policy and a critic net- work. The actor training loss includes soft Q- O-guidance term from the Q-gradient, The soft Q-guidance is based on the theoretical solution of the KL constraint pol- icy iteration, which prevents the learned policy from taking out-of-distribution actions. We demonstrate that such diffusion- based policy constraint, along with the coupling of the lower confidence bound of the Q-ensemble as value targets, not only preserves the multi-modality of target policies, but also contributes to stable convergence and strong performance in DAC. Our approach is evaluated on D4RI. benchmarks and outperforms the state-of-the-au of-the-art in nearly all environments. [
INTRODUCTION
Offline reinforcement learning (RL) aims to learn effective policies from previously collected data, without the need for online interactions with the environment (Levine et al. 2020). holds promise risky, expensive,
to implement RL algorithm to real-worke applications, where online interactions or even impossible. However, learning entirely from offline data brings a new challenge. The prior data, such as human demonstration, samples compared to the entire statc-action space. Learning policies beyond the level of behavior policy demands querying the value function of actions which are often not observed in the dataset. Despite off- off-policy RI. algorithms could be directly applied to offline data, those out-of-dine titto---dstribution (OOD) actions exacerbate the bootstrapping error of value function estimation, typically causing overestimation of action-valle values and leading to poor performance (Kumar et al. 2019).
often sub-optimal and cover only small part of
To alleviate the problem of overestimation of OOD actions, prior research of rolicy-regularized al- gorithm suggests regularizing the learned policy by limiting its deviation from the behavior policy. These methods generally regularize the learned policy by adding a behavior cloning term to the loss function (Fujimoto & Gu 2021 Wu al. 2019 Wang et al. 2022 training behavior sampler
to assist in evaluating the Q-learning target (Kumar et al. 2019 Fujimoto et al.! 2019 Wang et al.
"Corresponding authors.
Code is available at http://s//gtthtbb.c////gggiiiiia

---

Published as a conference paper at ICLR 2025
2022 Chen et al. 2022 Hansen-Estree et al. 2023). However, due to the intricacy of the behavior distribution, these methods require sufficient representative capacity of models and appropriate reg- ularization schemes to prevent sampling OOD actions and achieve strong performance (Wang al.
2022).
With the emergence of diffusion models (Sohl-Dickstein al. 2013 Ho et al. 2020). recent advances on policy-regularized algorithms suggest modeling the behavior policies using high- expressive diffusion models (Janner et al.) 2022 Wang et al.) 2022 Chen al., 2022 Hansen- Estruch et al. 2023). However, there are several limitations with the current implementations of diffusion models in offline RL. Some methods use the diffusion model as behavior sampler for et al. (023). Those methods re- quire generating lots of action candidates to choose from, which hinders the real-world applications for the slow inference process. Diffusion Q-learning (Wang et al. 2022) trains a biased diffusion model to aid in the estimation of the Q-learning target. Nevertheless, the biased diffusion model no longer prevents from sampling OOD actions (Figure [ 2).
subsequent action generation (Chen et al.] [222] Hansen-
and the back-propagation of gradients through the denoising process makes the training process time-consuming. Additionally, another drawback of modeling behavior policies diffusion models is the inability of the diffusion models to explicitly estimate density values. Techniques that rely on access to density functions are not directly applicable given a diffusion-modeled behavior policy (Peng al., 2019 Nair al., 2020). Furthermore, it remains unclear how to regularize policies to stay close to a diffusion-moder dedd be- havior that is both theoretically sound and effective practical performance.
In this paper, we propose Diffusion Actor-Critic (DAC) to address the offline RL problem by training diffusion-modeled target policy. We focus on the optimization problem of constrained policy iteration (Schulman et al. 20155 Peng et al., 2019 Nair et al., 2020) Chen et al.) 2022), where the target policy is trained to maximize the estimated Q-function while fulfilling the KL constraint of the data distribution. We derive that the optimization problem can be formulated as a diffusion noise regression problem, eliminating the need for explicit density estimation of either the behavior policy or the target policy. The resulting noise prediction target involves a soft Q-guidance term that adjusts the Q-gradient guidance according to the noise scales, which distinguishes it from both the guided sampling with return prompts (Janner et al. 2022, Chen et al. (222)) and methods where the Q- gradient is applied to the denoised action samples Wang et al. 2022). DAC follows the actor-critic learning paradigm, where we alternatively train diffusion liffusion-modeled target policy and an action- value model. During the actor learning step, we train policy model by regressing on a target diffusion noise in a supervised manner. For the critic learning, we employ the lower confidence bound (LCB) of a Q-ensemble to stabilize the estimation of Q-gradients under function approxing approximation imation error. This approach prevents the detrimental over- vver-esssimistic bias of taking the ensemble minimum as used in the previous research (Fujimoto et al. 20188) Fujimoto& & Gu 2021 Wang et al.) (022)). Experiments demonstrate that the LCB target balances the overestimation and underestimation of the value target, leading to improved performance.
In conclusion, our main contributions are:
Introducing DAC, a new offline RL algorithm that directly generates the target policy us- ing diffusion models. The high-expressivenesss of diffusion models is able to capture not only the multi-modality of behavior polices, but also the complexity of target polices as well. Moreover, the training of DAC avoids the back-propagation of gradients through the denoising path, which significantly saves the training time for learning diffusion policies.
Proposing the soft Q-guidance that analytically solves the KL constraint policy iteration us- ing diffusion models, without the need for explicit density estimation of either the behavior policy or the target policy. The necessity for constraint satisfaction in learning diffusion- modeled target policies is not only crucial for theoretical comprehension but also guaran- tees that the generated policy refrains from taking OOD actions.
We demonstrate the effectiveness of DAC on the D4RL benchmarks and observe that it outperforms nearly all prior methods by a significant margin, thereby establishing a new ofate-of-the-an baseline. Additionally, DAC shows stable convergence and strong perfor- mance without the need for online model selection (Wang et al., 2022 Kang et al. 2024). making more practical for real-world applications.
2

---

Published as a conference paper at ICLR 2025
2 PRELIMINARIES
We consider the RL. problem formulated as an infinite horizon discounted Markov Decision Process
(MDP), which is defined tuple (S, A, T,, do, 7,7) (Sutton & Barto 1999) with state space S, action space A, transition probabilities (s'|s, a). initial state distribution So ~ do. reward function r(s. a). and discount factor y E (0. 1). The goal of RL to train a policy stals) A > Do [0, 1] that maximizes the expected return: J(T) = E., do 20 ( ax)]. We also define the discounted state visitation distribution d² (s) = (1 - Lt= o Pn = s). Then the RI.
objective J (T) has equivalent form as maximizing the expected per-state-action rewards: J(x)) Egnd" a-ln(-le) [r(s, a)] (Nachum & Dai 2020). offfine RL, the agent has only access to a static dataset D, which is collected by potentally unknown behavior policy Tæ, without the permission to fetch new data from the environment.
Constrained policy iteration. Let Q x R be the Q-function of the policy π, which defined by Q" (s. a) = TEx,TIEE ('r, a5) = s, ag = al. In a standard policy iteration paradigm at iteration k. the algorithm iterates between improving the policy Ty and estimating the Q-function nT* via Bellman backups (Sutton & Barto 1999). Estimating QTM in the offline set- ting may request OOD actions that are observed in the dataset, resulting an accumulation of bootstrapping errors. To address this issue, off-policy evaluation algorithms (Fujimoto et al. 2019:
Kumar et al. 2019; Wu al. 2019 Schulman al., 2015 Peng al., 2019; Nair et al.] 20200 pro- pose to explicitly regularize the policy improvement step. leading to the constrained optimization
problem:
TA+1 arg max R
E Q"))
s.t. D(T, TT) I tb.
Commonly used constraints for D are members from f-divergence family, such as KL-divergence, <>-divergence and vari-tari distance (Peng et al. 2019 Nair et al. 2020 Nachum al. 2019
Nachum & Dai 2020). In this paper we consider 1) being the expected state-wise (reverse) KL- divergence: 1) TB) R DKI.(Tt ()) [| However, the complicated dependency of d on Ty makes it difficult to directly solve the KL constraint optimization problem (1) in offline RL. A typical approach for addressing this issue involves substituting the on-policy distribution d with the off-policy dataset D (Peng et al. 2019 Nair et al. 2020), resulting in the surrogate objective:
Tk+1 = arg max Egw Ea Q as.
(2)
s.t. E. (T((s) < Eb.
where E is a pre-defined hyperparameter to control the strength of the constraint.
Diffusion models. Diffusion models (Sohl-Dickstein et al. 2015 Ho et al. 2020 Song et al. 2020)
are generative models that assumes latent varibles following a Markovian noising and denoising process. The forward noising process {xo.r) gradually adds Gaussian noise the data xn p( Xx) with a pre-defined noise schedule {:
T
q(xET|XX) Xo) II (xxx(xx-1). (xx == N / Bix
(3)
t=
The joint distribution in 33 yields an analytic form of the marginal distribution
q (xx)x)) = N(xx V ayxo, (1 I) for all E {I....,T).
(4)
using the notation o 1 - B, and ā. II a., Given Xo, the noisy sample x, can be easily
obtained through the re-parameterization trick:
x; V vI a NO,
(5)
DDPMs (Ho et al. 2020 use parameterized models (x(x (x1) = Nxxx11 (), Eg ())
to reverse the diffusion process: pe (XxxT) WxT:: 0, IT'--po(xx (x (x:) 1 The practical imple-
mentation involves directly predicting the Gaussian noise e in (5) using a neural network co co(x,, t)
to minimize the original evidence lower bound loss:
(()) Pxxc-pplx)).----ul1l1
-ooo-oio + v1 aze, 1212
(6)
3

---

Published as a conference paper at ICLR 2025
Rebooor uppeot
Fera oma.ors.ooo sportt
Rehoo aqpert
Oredd
Und
Orguldancs
O dance
Darsbing
Denciaiig
Denstil
(6) Denoised Q-guidance
(b)) Hard Q: gugidaace
(e) Soft Q-guidance
Figure 1: A visual explanation of generating actions from noisy prior TT using (a) denoised Q- guidance (b) hard Q-guidance and (c) soft Q-guidance. The soft Q-guidance reduces the intensity of Q-guidance during the denoising steps, generating high-reward actions within the behavior support without the need to backpropagate the gradient through the path.
natural approach to employing diffusion models in behavior cloning involves replacing the noise predictor with a state-conditional model (x(X,, that generates actions X A based state s.
Score-based models. The key idea of score-based generative models (Vincent 2011 Song & Hr- mon 2019 2020) is estimate the (Stein) score function, which is defined the gradient of the log- likelihood Vx log p(x). Like diffusion models, score-based models perturb the data with a sequence of Gaussian noise and train a deep neural network sp(X,,t) t) to estimate the score Vx, log p(xx) for noisy samples X₃ ~ -N(x; Xg. I on different noise levels 1,2, .... T. The objective of explicit score matching (Vincent 2011) is given by:
E "xpoppar-NNixxxxxx222..--. ((())/ logp(x) - se (x: f)]
where X(t) is positive weighting function. Once the estimated score functions have been trained, samples are generated using score-based sampling techniques, such as Langevin dynamics
Song & Ermon 2019) and stochastic differential equations (Song et al. 2020).
DIFFUSION OCTOR-CRITTE
In this section, introduce the Diffusion Actor-Critic (DAC) framework that models the target policy directly as diffusion model, eliminating the need for density estimation of either the behavior policy the target policy. Initially, we formulate the KL constraint policy optimization diffusion noise regression problem, which yields soft Q-guidance term for the noise prediction process that enables the learning of the target policy in supervised manner. Additionally, ensemble to stabilize the Q-gradient estimation, which utilizes LCB to mitigate the over-pessimistic estimation associated with laking the ensemble minimum prior research.
introduce Q-
3.1
LEARNING DIFFUSION POLICY THROUGH SOFT Q-GUIDANCE
The problem of behavior constraint policy iteration 2 has a closed form solution + (Peng al.
2019 Nair et al. 2020 Chen et al. 2022) by utilizing the Lagrangian multiplier:
((s) T exp - (s, a)),
+ (a|s)
where 77 > 0 is a Lagrangian multiplier and (s) is state-conditional partition function. Obtaining
the closed form solution of गं+ directly from (8) challenging as requires estimation of the
density function of the behavior policy ny and the partition function (()). Prior methods (Peng
et al. 22019 Nair et al. 2020 Chen et al. 2022) suggest addressing this issue by projecting π onto parameterized policy тe using KL divereeee:
arg min Eg [ (
(9)
resulting in the policy update algorithm:
0x+1 = arg max ((s.a) llog (als)) exp L
1
(s,
(10)
4

---

Published as a conference paper at ICLR 2025
-10
Behavior
Denalsed
Hand
0,00
Soft
-1
10
130
130
Figure 2: Comparison of generated policies on 2-dimensional bandit using different Q-gradient guidance. We compare soft Q- Q-guidance (magenta) against hard Q-guidance (blue) that eliminates the noise scaling factor and denoised Q-guidance (Wang et al.) 2022) (brown) on 2-D bandit examples. The dots are behavior policies, which are colored based on the reward value. The dashed level curves represent the estimated Q-value field. Soft Q-guidance are capable of generating high-reward actions while remaining within the behavior support. We also observe that soft Q Q-guidance captures the multi modality of target policies as shown in the second plot. Experimental details can be found in Appendix B.5
Although 10 eliminates the necessity for estimating the partition function Z(s) and the behavior policy Tg. needs explicit modeling of the density function of the target policy ma. Such require- ment for Ty makes it unfeasible to directly use diffusion generative models due to the unavailability of density function estimation. Prior methods employ Gaussian policies to estimate my (Peng al., 2019 Nair et al. 2020), which limits the expressiveness of the target policy. To address these issues, we rewrite (8) using score functions:
V log + (als) = Va log Ta(aas) + 1 V" as, a a
(11)
where the action space is usually compact set in Rd for d-dimensional actions. seems that the target score function V log T (als) can be trained by regression on the right-hand-side How- ever, since Tg is unknown, we do not have the explicit regression target Va log T T (a|s). Drawing inspiration from explicit score matching with finite samples (Vincent, 2011), we smoothly extend the policy functions and the value function defined in A to the extended action space IR., Then we consider [II of the optimal score functions to hold for noisy perturbations of the observation set:
Xx, log (x, s) Xx, log (s) + Vx, Q X:),
X2 R.d
(12)
where pf (x,(s)
(x) (als)da and p. (x.ls) ge (x (axk/a)mga ( dd are noise distribu- When the perturbation
tions. The noisy perturbation x (x; a) is defined in with X,-
small, i.e. q: (x) d(x; a). then pł (x: ni+1 (ais) and pr (x; A Ta(a)s), which recovers the relationship between score functions within the action space A as described in (II. Tackling the score function of noise distribution is favorable, since Vx, logri (x/(s) itself serves means of generating Th using diffusion models, without the need for basee-based sampling methods such as Langevin dynamics, as described in the following theorem.
Theorem 1. Let e" (X;,s,t) - -VI āi V x. log p (s). Then e sx),,stt) is a Gaussian noise predictor which defines a diffusioo model for generating Th+1
Although 6" (xt, determines the diffusion model that directly generates the target policy, the estimation the noisy score function of behavior policy Vx, log P: (xx by (12). which is typically not accessible. To tackle this problem, we investigate the learming objective when utilizing function approximators. Specifically, we project onto parameterized conditional noise model co(x,, via L -
form of the target noise e" e'xx,, S, () necessitates
the target noise E" (x,, following the standard training objective of diffusion models:
arg min Eg~D.avx xer-ge (Xe (a) all (xt, s, -6" e'(x,, 8, 1)12
(13)
To eliminate the need for sampling from the unknown target policy T we approximate the ex- pectation through a क by the behavior data D. resulting in the surrogate objective:
arg min ((s,a)- D.x P.x (x. ||co (xx, s,t) (x,,,, s)
(14)
5

---

Published as a conference paper at ICLR 2025
Algorithm Diffusion Actor-Critic Training
Require: offline dataset D. batch size learning rates ag. ag. ag and Ctemas behavior cloning threshold 6, pessimism factor initial Lagrangian multiplier Thmi ensemble size H Initialize: diffusion policy cø. target diffusion policy Eg Co. Q-networks Qar, target Q- networks 2 Qar (h 1...... H), Lagrangian multiplier 1j = 1/mi
2: while training not convergent do
3: Sample batch of B transitions {(s, a, s,), C
4: Sample = Xo through denoising process using noise predictor eg (x(x,,,5)
5: for h in 21,2. .... H. do
6: Update &² f- o² as Var (d") end for
b Critic learning
7:
8: Sample c <~ (0, I).t ~ Unif(0, T) and compute x, = Vā,a + I a,e
9:
Estimate Q-gradient V ( (t) using [8
10:
x+o 0 - 0g VoLa (0)
(66
D Actor learning
7 x 7 + a,,(lEg S,
c112
Dual gradient ascent (optional)
12:
+ (1 Gema) )0 + Oema
13:
Th + (1 Rema)o* + Gema@
> Update target networks using EMA
end while
Such learning objective has an equivalent form which is easy to optimize.
Theorem 2. Training parameters 0 according to (14) is equivalent to optimize the following objec-
tive:
ct L -VI a. Ux. (T,
arg min
(15)
where Vaja VI -aaa age.
The learning objective (5) defines noise regression problem that approximates the solution of the KL constraint policy iteration (2) within the diffusion model framework, without requiring the estimation of densities for either the behavior policy or the target policy. We refer the last term in the noise target as the soft Q diffission guidance or simply soft Q D-guidancce Within the soft Q-guidance, the Q-gradient is weighted by the noise scale vI Czg. In typical diffusion model, the noise scale -I- + 0 as / + during the denoising process. This suggests that soft Q-guidance encourages the exploration of high-reward regions in the initial steps of the denoising process, and then gradually fades the guidance strength as the denoising step approaches the final output. In comparison to the hard Q-guidance that eliminates the noise scaling factor or guidance on denoised actions (see Figure I for illustrations), soft Q-guidance produces high-fidelity actions that closely resemble the behavior (OOD) actions (Figure 2}.
policies, thereby preventing the sampling of
To connect the learning objective [5) with
methods, we rearrange the terms in 15 and incorporate constant(s) into 17. resulting in the following actor learning loss:
(0) = ((),)) (alssa))e....(llle((x( [xl. t) - =el2 + vI- āe Ex( s,t) Vx, Q**
(16)
where the dot (.) implies inner product. The Lagrangian multiplier 7) determines the trade-off be- tween the behavior cloning and the policy improvement. As 1) - oc, the noise prediction loss (16) reduces to behavior cloning using parameterized conditional diffusion model, as used in the re- cent research (Chen al. 2022 Hansen-Estruch et al. 2023 Wang al., 2022). The second term involves a inner product between the predicted noise and the Q-gradient, promoting the acquired denoising directions align with the estimated Q- O-gradient field. In practical implementation, DAC balances the behavior cloning and policy improvement by controlling 7/ to be either fixed or learnable through dual gradient ascent. v) is learnable, DAC trains 7 to ensure that [|eg(Xe: s, = < for given threshold value b > 0.
During critic learning, we approximate Q** with neural networks. enhance the stability of es- timating the Q-gradient used in (16), we follow the method of using pessimistic Q-ensembles (Wul ct al. 2019 Agarwal et al. 2020 Smit et al. 2021 Lee et al. 202T 2022 An et al. 202T). Specifi- cally, we train ensemble of H parameterized Q Q-networks Qai and target Q Q-networks Qat- along
with lower confidence bound (LCB) value targets (Ghasemipour et al. 2022), which leads the

---

Published as a conference paper at ICLR 2025
critic learning loss:
Lo(d") - -B [r+ Qu.cnsss, a') - Qaa (s, (s.
(17)
QLCB(S a') og (s', a')] - PV Varn (s', a(),
where p 2 is a hyperparameter that determines the level of pessimism, and R2 [=] and Vara are empirical mean and variance operators over the H ensembles. Once the Q-functions are trained, the Q-gradient in the soft Q-guidance can be estimated by the ensemble's average of target Q-networks:
H
1
Vx, (s,
(18)
HC
hm1
where C' = (s ||a (s, a)] is estimated scaling constant that eliminates the influence of varying Q-value scales in different environments. We summarize the full algorithm of DAC for offline RL in Algorithm
3.2 POLICY EXTRACTION
We denote To (a(s) as the trained diffusion policy through denoisng process using noise predictor sx( f). While ma(a))) is capable of generating the target policy, we aim to reduce the un- certainty of the denoising process during the evaluation phase. To achieve this, we sample a small batch of Na actions and select the action with the highest Q-ensemble mean value, resulting in better performance:
(s) arg max max aN.... mo (-1s) E, (Q5g as)
(19)
This approach is commonly employed in methods where a stochastic actor is trained for critic learning, and deterministic policy is implemented during evaluation (Brandfonbrener al.] 2021 Haarnoja et al. 2018). Since (a(a))) is trained as a target policy, the sampling number N, can be relatively small. Through our experiments, we find that DAC can achieve superior performance with
Nu = 10. In comparison, SfBC (Chen et al.] 2022), Diffusion Q-learning (Wang et al.) 2022) and
IDQL Hunsen-EEttruce al.
2023
USe No 32, N₂ 50 and N₂ 128, respectively.
RELATED WORK
Offiine RL. Recent research on offline RL often use value-based algorithms based on Q-learning or actor-critic learning (Sutton & Barto 1999). Policy-regularizatio methods typically train biased behavior sampler help estimate the maximum Q-values within the behavior support. Among these approaches, BCQ (Fujimoto et al. 2019) learns conditional-Va.
(Sohn et al. 2015) to aid in sam- pling Q-learning targets; BEAR (Kumar et al., 2019) employs maximum mean discrepancy (MMD) to regularize the learned policy. Moreover, BRAC (Wu et al., (2199) is based actor-critic learning framework and explores various regularization methods as value penalties; TD3+BC (Fujimoto &
Gu 2021) 20211 adds behavior cloning term regularize the learned policy in supervised manner. Ad- ditionally, some methods implicitly regularize the policy by training pessimistic value-functions or using in-sample estimation. CQL (Kumar al. 2020) learns conservative Q-values on OOD actions.
IQL (Kostrikov et al. [2021) and IDQL (Hansen-Estruch Estruch et al. 2023) use asymmetric loss functions
approximate the maximal Q-value target via in-sample data. IVR (Xu al. 2023) also employs in-sample learning while within the framework of behavior-regularized
MDP problem, resulting
in two implicit Q-learning objectives. Extreme Q-learning (Garg et al., 2023
estimates maximal
Q-values using Gumbel regression. In addition to regularizing target policies to align with behav- ior policies, recent studies also emphasize the importance of regulating the steps of policy updates within the trust region (Schulman et al. 2015 2017) to ensure policy improvements (Zhuang al.) 2023 Zhang & Tan 2024; Lei et al.] 2023). Our method conducts relicc-regularizatton by focus- ing on KL-regularized policy iteration (Schulman et al. 2015 Peng al. 2019. Nair et al. 2020 which regularizes the policy improvement step to fulfill the KL-divergence constraint, preventing the bootstrapping error of estimating Bellman targets.
Diffusion models for offfine RL. Recent studies that utilize diffusion models for offline RL can be broadly categorized into two types: those that model entire trajectories and those that generate

---

Published as a conference paper at ICLR 2025
Table 1: Average normalized scores of DAC compared other baselines. We use the following abbreviations: "m"" for "medium"; "r" for "replay"; "c" for "expert"; "u" for "umaze"; "div" for "diverse" and "I" for "large". For locomotion tasks, we use "v-2" version; for antmaze tasks, we use "y-0" version. We highlight in boldface the numbers within 5% of the maximal scores in each task.
Furthermore, we also underline the highest scores achieved by prior methods.
Dataset
Onestep-RL.
CQL
IQL
IVR
EQL
Diffuser
DTQL
AlignIQL
SIBC
DQL
IDQL-A
DAC (ours)
halfcheetah-m
48,4
44,0
47.4
48.3
48.3
44.2
$7.9
46.0
45.9
51.1
51.0
59.1 - 0.4
hopper-m
58.5
66.3
75.5
74.2
58.5
99.6
56.1
57.1
90.5
65.4
101.2
walker2d-m
78.3
84.2
84.2
89.4
78.5
77.9
82.5
halfcheetah-m-r
38.1
44.2
44.8
50.9
37.1
47.8
45.9
4:
hopper-m-r
97.5
95.0
94.7
99.7
100.7
96.8
100.0
74,8
86.2
101.3
92.1
103.1 0.3
walker2d-mmrr
49.5
71.2
73.9
81.2
82.2
61.2
88.5
76.5
65.1
95.5
85.1
96.8 1: 1.0
halfchoctah-m-e
93.4
91.6
86.7
94.0
94.2
79.8
92.7
89.1
92.6
96.8
95.9
99.1 + 0.9
hopper-m- m-e
103.3
105.4
91.5
111.8
111.2
107.2
109.3
107.1
108.6
111.1
108.6
111.7 1: 1.0
walker2ddmm-e
113.0
108.8
109.6
110.2
112.7
108.4
110.0
111.9
109.8
110.1
112.7
113.6 3.5
Incomotion total
684.6
698.5
749.7
749.7
752.9
678.0
798_3
681.1
680.3
791,2
739.2
836.4
antmaazo-u
64.3
74.0
87,5
93.2
93.8
94.8
94.8
92.0
93.4
94.0
99.5 + 0.9
antmazze u-div
60.7
84.0
62.2
74.0
82.0
78.8
82.4
85.3
66.2
80.2
85,0 1: 7,9
antmaze-
0.3
61.2
71.2
80.2
76.0
79.6
80.5
81,3
76.6
84.2
85.8 : 5,5
entmaze-m-div
70.0
79.1
73.6
85.5
6.2
antmaze-l-play
39.6
53.2
46.5
59.3
antmazo-l-div
0.0
14.9
47.5
52.3
49.0
66.4
54.0
45.5
67.9
55.3 10.3
antmaze total
125.3
303.6
378.0
432.0
420.9
441.4
474.8
445.4
417.8
474.6
459.9
behavior policies. Diffuser (Janner et al. 2022) trains diffusion model trajectory planner. The policies are generated through guided-sampling with return prompts, similar to methods that mod- eling trajectories using Transformer (Chen et al. 202T Janner et al. 202T). Diffusion Q-learning [Wang et al.] 2022) employs a diffusion model as a biased behavior sampler, incorporating an ad- ditional loss to promote the denoised actions to achieve maximal Q-values. IDQL (Hansen-Estruuhh et al. 2023) and SfBC (Chen et al.) (022) use diffusion models to generate behavior policies. The actions. Diffusion trusted Q- learning (Chen et al. 2024) also models behavior policies as diffusion models, while the diffusion- modeled behavior samplers are used to extract Gaussian policies that focus on high-reward modes. In comparison, DAC directly utilizes trained diffusion model as the target policy in the actor-critic paradigm, rather than using as behavior sampler to estimate Q-learning targets.
target policies are extracted by re-sampling from leffusiion-gneerate
Diffusion models for online RL. Recent research also investigates diffusion policies for online RL. QSM (Psenka et al. 2023) learns diffusion policies by directly aligning with the Boltzmann distribution of Q-functions. DIPO (Yang et al. 2023) updates diffusion policies by adjusting action samples in the replay buffer. To support online exploration, (Wang et al. 2024) employs Gaussian mixture models to estimate densities and entropy. Although these methods also learn diffusion target policy, they lack
mechanisms to tackle the unique challenges of offline RI..
5 EXPERIMENTS
In this section, we empirically demonstrate the effectiveness of our proposed method by comparing it with comprehensive of recent approaches D4RL benchmarks (Fu et al.) 2020). We also conduct ablations to show the efficacy of soft Q-guidance in achieving stable convergence and strong performance. Additionally, we include sensitivity analysis of the Q-ensemble size and action sample size to demonstrate its robustness. Due to space limitations, we place experimental details and ablation studies on LCB and other hyperparameters in Appendix and
5.1 COMPARISONS ON OFFLINE RL BENCHMARKS
We compare our approach against extensive collection of baselines that solve offline RL using various methods. For explicit policy- rolicy-regularizztion methods, we consider one-step step RL (Onestep-RL. Brandfonbrener et al. 2021) which conducts single step actor-critic learning. For calee-constraint methods, we compare against Conservative Q-learning (CQL) (Kumar et al. (020). For in-sample estimation of maximal value targets, include Implicit Q-leariing (IQL.) Kostrikov et al. 2021
8

---

Published as a conference paper at ICLR 2025
Wall 2e medium-2
walcer2 wwlder22smedammepppooo-
wea Ioe med expert12
Densaed
26.
Sent parst
Normalizzed score
360 ale to
125
210
1i hooper mediu expert
ropper medium v2 106
hopper medium reply
are-
10-
850
350
ta medium-2
1ea
hatichestan.noe www.:mmmmmmmmmmiiommim resbbyy-
140
haPoreetonn medium expertie
850
1:00 0.55 125 150
a2s 015
Training steps (M)
Training curves of DAC with different Q-gradient guidance. We compare soft Q-guidance (soft), hard Q-guidance (hard) and the denoised Q-guidance (denoised) on locomotion tasks. DAC with soft Q-guidance achieves stable convergence and strong performance across all the tasks.
Figure 3:
Implicit Value Regularization (IVR) (Xu et al. 2023) and Extreme Q-learning (EQL) (Garg et al,
2023 Additionally, compare our approach methods that also learn diffusion policies. In this
category, we compare against the most recent works including Diffuser (Janner et al. 2022), SfBC
(Chen et al. 2022), Diffusion Q-learning (DQL.) (Wang et al. 2022), Diffusion Trusted e-rniing
(DTQL) (Chn et al. 2024). AlignIQL (He et al. (024)) and IDQL MlaasennEEstrute et al. 2023
Specifically, we present the results of "IDQL-A" variant (annsen-Estruch et al. (2023) for IDQL,
which permits tuning of any amount of hyperparameters and exhibits strong performance. For the baselines, we report the best results from their own paper or tables the recent papers (Wang et al.]
2022 Chen et al. 2022, Plansen-Estruck et al. 2003).. The main results are shown in Table As for additional results on Adroit and Kitchen tasks, we refer readers to Appendix C.I
Drawing from the experimental results, DAC outperforms prior methods by significant margin across nearly all tasks. DAC significantly enhances the overall score on locomotion tasks, with an average increase of 5% compared to the best performance in prior studies. Notably, for "medium" tasks, where the dataset contains numerous sub-optimal trajectories, DAC consistently achieves im- provements of over 10%. The antmaze domain presents a greater challenge due to the sparsity of rewards and the prevalence of sub-optimal trajectories. Consequently, algorithms must possess strong capabilities in stitching together sub-optimal subsequences achieve high scores (Janner et al. 2022). It is evident that DAC outperforms or achieves competitive outcomes on antmaze tasks, with an almost perfect mean score (== 100) on the "antmaze -umzzzee task. In the case of the most demanding "large" tasks, DAC performs comparably to previous methods, with the exception of IDQL-A, which consistently showcases superior performance. One potential explanation for this difference could be that we do not tune the rewards by subtracting negative number, as is done in previous studies (Kumar et al., |020) Kostrikov et al.) (202T] Wang et al., [202 [Hansen-Estrus et al.) 2023). This setting exacerbates the impact of reward sparsity in more intricate "large" environments, leading to slower convergence.
It is worth mentioning that we report the performance after convergence, which imposes stronger requirement for evaluation, as it necessitates the model training to demonstrate the capability of con- vergence, rather than relying online model selection (Wang et al. 2022 Kang et al. (024). These requirements hold significant importance for ensuring robust deployment in real-world applications.
5.2 ABLATION STUDY ON Q-GRADIENT GUIDANCE
To demonstrate the effectiveness of soft Q-guidance, we compare DAC to two variants: one that utilizes hard Q-guidance by eliminating the noise scaling factor, and another that employs denoised Q-guidance by conducting guidance with denoised actions (see Appendix B for details). The training curves are presented in Figure 3 with the corresponding final normalized scores in Table 2 DAC with soft Q-guidance achieves the highest performance nearly all tasks. The hard Q-guidance

---

Published as a conference paper at ICLR 2025
Table Q-guidance ablation. We compare soft Q-guidance against hard Q-guidance and denoised Q-guidance while maintaining the remaining settings the same, with the average normalized scores surpassing prior methods (Table highlighted in boldface.
walker2d
hopper
halfcheetah
Q-Target
Denoised
2.2 2.2
95.4 4
105 + 1.0
49.5 1..5114
71.9
56.7 1
5.3
1,76
1.0
85.2 16.1
96.9
1.
103.1
110.2
(Ouns)
3.6
1136
0.3
111.7
also performs well when the behavior dataset comprises adequate number of optimal demonstra- tions. However, when confronted with tasks that involve numerous sub-optimal trajectories (such as "medium" datasets), the hard Q-guidance falls behind in comparison the soft Q-guidance. Fur- thermore, the denoised Q-guidance often struggles to generate in-distribution actions and frequently fails. Nevertheless, yields the highest score on "halfchectah-m" task, which could be attributed to the fact that such task is more tolerant to OOD actions.
5.3 SENSITIVITY ANALYSIS
Q-ensemble sizes. We compare the per- formance of different Q-ensemble sizes in Figure We find that ensemble sizes 2 5 generally yield similar results, while using number as few as two can reduce perfor-
medium replay
medium expert
120
120
Normalized score
1,00
100
mance on certain tasks. This highlights the importance of accurately estimating the Q- gradient field in DAC for effective learn- ing. Larger ensembles help smooth the NN-estimated Q-gradient fields, leading to more stable performance. We choose an ensemble size 10 for all tasks to balance performance and computational cost.
35
0.5
Training stops (M)
Figure 4: Experiments of using different Q-ensemble sizes (H) on hopper tasks.
Number of action samples. Compared with methods that diffusion models behavior samplers, DAC directly models the target policy as diffusion model, per- mitting fewer action samples to achieve good results. To demonstrate that DAC is
medium
medium-replay
medium-expeet
:20
Normal loed scores
110
Rh-1
robust to the choices of Na. we conduct
experiments with various N₂ on hopper tasks, as shown in Figure [ We observe that increasing N₂ > 10 gives similar re- sults, while small Na is detrimental the performance by the randomness de- noising process of diffusion models.
Traming steps (M)
Figure 5: Experiments of using different number ac- tion samples (N,) on hopper tasks.
CONCLUSION
In this paper, we propose the Diffusion Actor-Critic framework, which theoretically formulates the KI constraint policy iteration as diffusion noise regression problem. The resulting policy im- provement loss includes a soft Q-guidance term that adjusts the strength of Q-gradient guidance based noise scales. This approach encourages the generation of high-reward actions while maining within the behavior support. Furthermore, DAC avoids gradient propagation through the denoising path, significantly reducing training time for diffusion policies. Experiments demonstrate that our algorithm achieves stable convergence and superior performance on D4RL benchmarks, outperforming previous methods across nearly all tasks without the need for online model selection.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International conference on machine learning, pp. 104-114. PMLR, 2020.
Gaon An. Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. snvertainty-basee offline inforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436- 7447, 2021.
David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline without off- policy evaluation. Advances in neural information processing systems, 34:4933349444 2021.
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209. /4548, 2022.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 5084-150977 2021.
Tianyu Chen, Zhendong Wang. and Mingyuan Zhou. Diffusion policies creating trust region for offline reinforcement learning. arXiv preprini arXiv:2405. 19690, 2024.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprini arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. arXiv preprint arXiv:2301.02328 2023.
Kamyar Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. Advances in Neural Information Processing Systems, 35: 35:18267-18281, 2022.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint ar arXiv:1812.05905. 2018.
Philippe Hansen-Estruchh Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
Longxiang He, Li Shen, Junbo Tan, and Xueqian Wang. Aligniql: Policy alignment in implicit q-learming through constrained optimization. arXiv preprint arXiv:2405.18187 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-683 6851, 2020.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as big sequence
modeling problem. Advances in neural information processing systems, 34:1273-1286, 2021.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint (tti::200500999) 2022.
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.

---

Published as a conference paper at ICLR 2025
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Ilya Kostrikov. JAXRL: Implementations of Reinforcement Learning algorithms in JAX, 10 2021. URL
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:2110.06169 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179-11 119-1191, 2020.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified frame- work for ensemble learning in deep reinforcement learning. In International Conference Ma- chine Learning, pp. 6131-6144 6141. PMLR, 2021.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pp. 1702-1712. PMLR, 2022.
Kun Lei, Zhengmao He, Chenhao Lu, Kaizhe Hu, Yang Gao, and Huazhe Xu. Uni-o4: Unifying online and offline deep reinforcement learning with multi-step on-policy optimization. arXiv preprint 1rXiv:2331.03335 2023.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.038 039983, 2016.
Diganta Misra. Mish: A self regularized non-monotonic activation function. arXiv preprint
arXiv:1908.08658 08681, 2019.
Ofir Nachum and Bo Dai. Reinforcement learning via renchel- duality. arXiv preprint arXiv:2001.01866, 2020.
Ofir Nachum, Bo Dai, Ily Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02074, 2019.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint arXiv:2006.093359, 2020.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. advantage--eiggheee regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:19100.00117,, 2019. 2019.
Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning a diffusion model policy from rewards via q-score matching. arXiv preprint arXiv:2312. 1752, 2023.
John Schulman. Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning. pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.06347, 2017.
Jordi Smit, Canmanie Ponnambalam, Matthijs TJ Spaan, and Frans A Oliehock. Pebl: Pessimistic ensembles for offline deep reinforcement learning. Robust and Reliable Autonomy in the Wild Workshop the 30th International Joint Conference of Artificial Intelligence, 2021.
12

---

Published as a conference paper at ICLR 2025
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2265, PMLR, 2015.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438-1222 12448, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.133456. 2020.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. Robotica, 17(2): 229-235, 1999.
Pascal Vincent. connection between matching and denoising autoencoders. Neural compu- tation, 23(7):1661-1674 2011.
Yinuo Wang, Likun Wang, Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang, Liming Xiao, Jiang Wu, Jingliang Duan, al. Diffusion actor-critic with entropy regulator. arXiv preprint 4rXiv:2405.
2024.
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.0679) 06193, 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: /9/1 1361, 2019.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xi- anyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. arXiv preprint arXiv:2303.1588 2023.
Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for rein- forcement learning. arXiv preprint arXiv:2305. 122, 2023.
Zhe Zhang and Xiaoyang Tan. An implicit trust region approach to behavior regularized offline rein- forcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 16944-16952, 2024.
Zifeng Zhuang, Kun Lei, Jinxin Liu, Donglin Wang, and Yilang Guo. Behavior proximal policy optimization. arXiv preprint 1331, 2023.
13

---

Published as a conference paper at ICLR 2025
A PROOFS OF THEORETICAL RESULTS
Lemma 1. Let E ~ (0, be a standard Gaussian noise, and x, = Vā,a + vI - āje be the noise perturbation of the action defined in ( Then the denoising score function V log qe (xx|a) maintains the property:
Vx, log (xxla)))
(20)
VI-
Proof. Since (x(x((x)) Nx v Vaxx (1 ([--))]) at)I), the density of noise distribution has the closed form:
(xi vā
(xxxaa)) exp
(21)
2(1 (is)
Therefore, we have
Vāra
v1 - Ō+C
Vx. Xx log (x(la))
āt
1 āt
(22)
V1
A.1 PROOF OF THEOREM 1
Proof. Let ~ (0, be a standard Gaussian noise. Consider the diffusion process ge (x(xaa)) (a)
defined in using reparameterization trick. Then e* solves the noisy score matching objective:
arg min t (-(a)))) 11 vI āi Vx (x)
(23)
We can rewrite the objective to obtain:
e" = arg min E Ea xx [1((x,, 11 +(x,,s,t) v1 - a. Vx, (x Ct.
(24)
Consider the second term:
Ex Ex (118) [(xx,, s,t) VI 1 Vx (xgp(((xls))
VI āi
(((x))))(((,,,,)) s, Vx, (x,
X1
VT ai
((x,, (x:
Xi
8(x,, Vx, / (als))a
v1 σ.
((x,, (((xx(a)V (x (als)dadxx loga
āi
((xx,8,1) -I vx, lo (x) (als (als)
Ears ( (1) vI ae Vx, (x)a(x
Thus we obtain:
e" arg min E Eawa f ( (a) 2
|((x;, s, +)ll+ E((,, vI as Vx, log + Ci
arg min E B
(ē(x;, 8,t) + vI oi Vx, log (x) +
= arg min Ea +xxeqg (((x;,s,t) <1 + C1 - C2. (by Lemma
Here C and C2 constants independent of é, which completes the proof.
14

---

Published as a conference paper at ICLR 2025
A.2 PROOF OF THEOREM 2
Proof. Let the diffusion process (x, a) be defined in 44) using reparameterizatioo trick by sam- pling standard Gaussian noise (0, I). We rewrite the training objective:
arg min DD
(x), s,t) E" (x, s, t)
arg min (Da)) EE (X,, s,t) vI ai Vx, (x
arg min
Xx: 1, v1 de Vx. logpe(x;)s)
1 Vx. Q s][
7
1
= arg min Da
1 Xx, s, t)l? + co(xi, vI āi Vx, (x
+ + s,t) vx, Qa* s). + Ci
Similar to the proof of Theorem 1 we can rewrite the second term to obtain:
E Exx xxxippl(a (11s) (x,,
vI Vx. (x,
V1 0z II ag
(x Vx, (x,
s,(x),stt) (x.P((x(/sddx,
60(x).8tt) Vx, / (
VI āe
60(x),8,1) / (o
ae x:
04 / (x(xx,,,t) Vx, log (xgg)
(a v - āeV (x:/a)
(a -e] (by Lemma Db.
Therefore, have:
arg min (( D. x (x. (a) (o),, (x,,
0
= arg min E
(x), 1) ex(x,, vI ai Vx, (x,la)
0
cx,, 8, 1) 1 Vx, x+* (x) +
arg min D. (x), 12 s,)
1
t t)
Vx. Q** (x) +Cr
17
arg min miinE Di (x),s))) - 6 1 - Q=* (x,, s)1 + Ci C2
0
"
= arg min E a (o( c 1 Vx Q (x., s| 2 + C1 - Cz. n
#
where x, v + -I a is given by the reparameterrization trick. C and C2 constants independent of 6a, which completes the proof.
15

---

Published as a conference paper at ICLR 2025
B EXPERIMENTAL DETAILS
We train our algorithm for 2 million gradient steps in order to ensure model convergence. For each environment, carry out 8 independent training processes, with each process evaluating performance using 10 different seeds at intervals of 10,000 gradient steps. This leads to a total of 80 rollouts for each evaluation. We report the average score of evaluations in the last 50,000 gradient steps without any early-stopping selection, which fairly reflects the true performance after convergence. We perform experiments taking approximately 4 hours to complete, including both the training and evaluation processes. Our code implementation built upon the jaxrl (Kostrikov) 2021 code base.
GeForce RTX 4090 GPUs, with each experiment
B. NETWORK ARCHITECTURE
We employ simple 3 layer MLP with hidden dimension of 256 and Mish (Misra 2019) activation for both the actor and critic networks. To enhance training stability, we implement target networks for both actor and critic, which track the exponential moving average (EMA) of the training networks. Specifically, we initialize the target networks Eg and Q with the same seed as training networks Cg and Q respectively. We update the target actor network eg every gradient steps while update the target critic networks Qøy after each gradient step to further ensure training stability.
B.2 HYPERPARAMETERS
We maintain consistent hyperparameter settings for the diffusion models and networks across all tasks. The hyperparameter settings are as follows:
Table 3: Hyperparameters for all networks and tasks.
Hyperparameter
Value
T (Diffusion Steps)
5
3. (Noise Schedule)
Variance Preserving (ong et al. 2020
H (Ensemble Size)
10
(Batch Size)
256
Learning Rates (for all networks)
3e-4, 1e-3 (antmaze-
Learning Rate Decay
Cosine (Loshchilov & Hutter 2016
Optimizer
Adam (Kingma & Ba 2014
Muit (Initial Behavior Cloning Strength)
[0.1, 1]
an (for Dual Gradient Ascent)
0.001
Oeema (EMA Learning Rate)
5e-3
N. (Number of sampled actions for evaluation)
10
(Behavior Cloning Threshold)
[0.05, 1]
(Pessimistic factor)
[0, 2]
Regarding the pessimistic factor p. we empirically find that selecting the smallest possible value for p without causing divergence in Q-value estimation yields good outcomes, as the learned target policy already avoids sampling OOD actions. This makes the tuning of p to be relatively straightforward. In terms of rplicy-regularization, DAC controls the trade-off between behavior cloning and policy improvement using either constant 7) = Tinit or learnable 17 by setting b for dual gradient ascent (Algorithm D For locomotion tasks, we employ dual gradient ascent which dynamically adjust to fulfil the policy constraint. As for antmaze tasks, we choose constant v) = Vini during the training process. Moreover, since different tasks involve varying action dimensions, we choose different hyperparameters for each task. We consider values Tinis E [0.1, 1]. [0.05, 1] and p e [0, 2]. We summarize the hyperparameter settings for the reported results in Table
B.3 VALUE TARGET ESTIMATION
For the critic learning, we need to sample a' ~ .oo (-Is') to estimate the value target in (7). To enhance the training stability, we samples M 10 actions from ne, (-Is') through denoising pro-
16

---

the Published as a conference paper at ICLR 2025
Table 4: Hyperparameters settings for tasks.
Tasks
"
p
Regularization Type
hopper-medium-v?
1.5
Leamaale
hopper-medium-replaye
1.5
Learnable
hopper-medium-experty
0.05
1.5
Learnable
walker24-medium--2
I
Learnable
wallerr2--modiuu---cpaaa-2
1
Learnable
wallker22-mmccduu--epprrr---
1
Learnable
halfcheetal--mediun-22
0
Learnable
haffcheetal-mdduum--epaayay
0
Learnable
hallcheetal-mmcduu---ppprrt
0.1
0
Learnable
antmaze-
antmaze--umzze-vl umaze-v0
0.1
1
Constant
antmazz-umazz-divereeee
0.1
1
Constant
antmaaz--eeduum-laa--
0.1
Constant
antmazz--mddum-ddieeee--
0.1
1
Constant
antmaze-large-play--e
0.1
1.1
Constant
antmaze-late laanzze-lrrge-divers---v
0.1
1
Constant
As for locomotion tasks, calculate the average value 1/M > (Qz a()] the sampled actions to estimate the target Q-value. While for antmaze tasks, use the maximum maxa ay. [2ay (s', a() to address the problem of reward sparsity, which is consistent with previ- ous research (Wang et al. 2022).
B.4 Q-GRADIENT GUIDANCE
To fairly assess the performance of different Q-guidance, shown in the 2D-bandit example (Figure 2 and an ablation study in Section 5.2 we modify the actor learning loss of DAC while keeping all the remaining settings the same. In the case of soft Q-guidance, we use the original loss of actor learning for DAC (T6). For the hard Q-guidance, we modify (16) to remove the noise scale factor:
Lhand (0) ((s.aa)D,e.. www..e......lllleeeeMM. [x]leo(x,, s, e||² cx( V QT* (s, xx)
(25)
Regarding the denoised Q Q-guidance used in Diffusion Q Q-learning (Wang et al. 2022). following denoised Q-guidance loss:
Letenisse (0) = F(.sa)) (x,,,,, <e|2 + Exp 1Q** (s, xo)]]
(26)
where the denoised action X, is obtained through the denoising process used in DDPM, and the gradient Ō (f) (f)/00 will back-propagated through the denoising path. All the Q-functions are re-scaled by an estimated constant ((s.a-)("""(Eaai) remove the influence of different Q-value scales.
B.5 2-D BANDIT EXAMPLE
For the 2-D bandit example (Figure we generate 400 sub-optima behavior actions by drawing samples from patterns with Gaussian noises. The reward values for each action are determined by the distances between action points and (0.4, -0.4), i.e., reward -v(r 0.4)² (y + 0.4)² + (O, 0.5I). Therefore, the majority of actions are sub-optimal and the estimated gradient field will tend to promote the generation of out-of-distribution (OOD) actions. However, given that the true reward values associated with OOD actions are agnostic to the learner, well-performing policy learned through offline RL should not deviate significantly from the behavior support. This high- lights the superiority of DAC in this protocol. To reproduce the results presented in Figure 2 train 20,000 gradient steps with batch size of 128, a learning rate of 1e-3, a diffusion step T 50, and behavior cloning threshold b 1.3 for all the methods.
17

---

Published as a conference paper at ICLR 2025
B.6 REWARD TUNING
We adhere to the reward tuning conventions for locomotion tasks in the previous research Kostrikov al. 202T), which is defined as:
r
1000 X maximal trajectory return minimal trajectory return
(27)
As for antmaze task, faces the challenge of sparse rewards, with the agent receiving a reward of upon reaching the goal and otherwise. Previous methods typically subtracts a negative constants (such as -1) from the rewards tackle the issue of reward sparsity (Kostrikov et al. 2022). However, we empirically find that DAC performs well for most tasks without the need such reward tuning technique. In our experiments, we simply employ the same tuning method (2) as the one used for locomotion tasks, which in fact scales the rewards by 1,000 for antmaze environments. This tuning method does not effectively tackle the problem of sparse rewards, which could potentially result in the inferior performance of DAC on the "large" antmaze tasks.
al. 202T; Wang
C ADDITIONAL EXPERIMENTAL RESULTS
C.I RESULTS ON ADROIT AND KITCHEN TASKS
We provide additional results of DAC on Kitchen and Adroit tasks from D4RL dataset. Here also involve the reported results from Diffusion Q-learning (Diffusion QL) Wang et al. (2022) as comparison. The results are summarized in Table ( and the training curves are show Figure and and Figure II] Given that Diffusion Q-learning reports the scores with online model selection, might be unfair make direct comparison. Nevertheless, we do observe that DAC outperforms Diffusion Q-learning in Adroit domain.
Table 5: Additional experiments on Adroit and Kitchen tasks. Similar to locomotion and antmaze tasks, we report the convergent performance of DAC by averaging the scores in the last 50,000 gradient steps. Given that Diffusion Q-learning reports the scores with online model selection, it might be unfair to make direct comparison. However, still observe that DAC outperforms
Diffusion Q-learning in Adroit domain.
Tasks
CQL
IQL
Diffusion QL
DTQL
DAC
pen-human-vl
35.2
71.5
72.8
64.1
81.3 1 4.9
pen-cloned-yl
27.2
37.3
57.3
81.3
63.9 + 7.3
Adroit Average
31.2
54.4
65.1
72.7
72.6
kitchen-complete-W
43.8
62.5
80.8
84.0
77.4 + 4.7
kitchen-paree partial-O
49.8
46.3
60.5
74.4
50.0 + 5.7
kitchen-mixed-v0
51.0
51.0
62.6
60.5
60.2 + 7.3
Kitchen Average
48.2
53.3
69.0
73.0
62.6
C.2 PESSIMISTIC VALUE TARGETS.
To demonstrate the importance of the LCB target in balancing the overestimation and underestima- tion of value targets, we compare a variant of DAC on locomotion tasks, wherein the LCB target replaced by the ensemble minimum (min) (Fujimoto et al. 2018 Fujimoto & Gu 2021 Wang et al.) 2022). The results are shown Table 6 and Figure g is noteworthy that the variants DAC without the LCB target also achieve competitive performance compared to prior research in many tasks.
C.3 ADDITIONAL SENSITIVITY ANALYSIS OF HYPERPARAMETERS
Diffusion step T. We conduct experiments of DAC with different diffusion steps hopper envi- ronments, as shown in Table and Figure We find that increasing the diffusion steps gives similar performance. Furthermore, tradicts the claim in Diffusion Q-learning. In our experiments, we use = 5 for all the tasks, and despite the shorter diffusion path, it is sufficient to achieve strong results.
observe that larger T converges slower than smaller T, which
18

---

Published as a conference paper at ICLR 2025
Table Ensemble Q-target ablation. We compare LCB target against the minimum target (min). We also involve the best scores of the prior methods (SOTA) from Table | for comparison.
walker2d
hopper
halfcheetah
Q-Target
m-r
m-c
m-r
m-c
m
m-r
m.c
SOTA
87.0
95.5
113.0
90.5
101.3
111.2
51.1
47.8
96.8
Min
83.9 + 0.22
66.3 = 9.7
110.0 +
0.2
100.8 0.9
102.9 0.4
111.3 0.1
49.3 =
0.3
43.1 #.2
43.2 0.1
LCB (Ours)
96.8 1: 3.6
96.8 1.0
113.6
1 3.5
101.2 1 2.0
103.1 0.3
111.7
1 1.0
59.1
0.4
55.0 1 0.2
99.1 1 0.9
Table 7: Sensitivity analysis of different length of diffusion steps.
Diffusion step I
5 (ours)
10
20
hopper-medium-v2
101.2
102.0
99.7
hopper-meddimm-reply--2
103.1
102.3
102.2
hopper-medium-expere
111.7
110.2
110.1
Pessimistic factor p. We compare the performance of DAC with different pessimistic factors p. which controls the pessimism of the LCB of value targets in (I7). The results walker2d tasks are shown in Figure (right). A small p usually leads to overestimation of value targets and cause the Q-networks to diverge +00, resulting in failed learning (as with larger P may allow converge, underestimating value targets negatively impacts final performance. In our experiments, we empirically find that selecting the smallest p that prevents divergence of the Q-functions is sufficient for strong results, making tuning P straightforward in practice.
0.5). While
Behavior cloning threshold b. We also conduct sensitivity analysis of b which controls the strength of behavior cloning in walker2d environments, as shown in Figure ? (left). Given the constraint [x,, s, t) < b. the agent prone generate behavior data with a smaller b.
C.4 TRAINING TIME COMPARISON
DAC trains significantly faster than Diffusion Q-leaming without requiring gradient back- propagation through the denoising process. This time-saving advantage is more pronounced as the number of diffusion steps T increases. To verify this, conducted experiment assess the average training time ratio of performing a single-step batch update in DAC compared to Diffusion Q-learning. The results are shown in Table g The experiment shows that when quires less than half the time needed by Diffusion Q-learning to perform one-step updates. While for a diffusion step as large as 100, Diffusion Q-learning more than 18 times slower compared with DAC. Considering that Diffusion Q-learning is unlikely to match DAC's performance with just one-tenth even one-third of the training steps, suggests that DAC can be trained more quickly.
5, DAC re-
Table 8: Time ratio between soft Q-guidance and denoised Q-guidance for conducting one-step batch gradient descent with different lengths of diffusion steps.
Diffusion step T''
5
10
20
50
100
Time ratio
45.1%
31.4%
19.9%
9.7%
5.4%
C.5 ADDITIONAL TRAINING CURVES
We also involve additional training curves on D4RI. tasks in Figure
Figure 10 and Figure
19

---

Published as a conference paper at ICLR 2025
medium
medium-reppay
medium-expert
120
120
120
Normalized score
100
100
80
60
60
40
20
T=10
T-5 (Ours)
40
20
T-20
20
0-
1.0
1.5
2.0
0.0
0.5
1.5
2.0
0.5
1.0
1.5
Training steps (M)
Figure 6: Sensitivity analysis of diffusion steps T on hopper tasks.
mediumiv2
B-0
123
2-0
Rope (Qurl
303
100
43
0.50
0.5
3.50
2.0
Figure 7: Sensitivity analysis of behavior cloning threshold (left) and pessimistic factor p (right) on the walker2d-medium task.
1er2 medium-2
160
<5c 025 330
2000
nds 6.5 sis
160
1e shs sin zns
02e als o.s in 169 ths 3.75
rasparmedium--
hopper anreereeddum.eo eccert122
6.20
0.5
0.2 hafchesbhimediun oxportv2
halfhhecchh medium
holdhherah Moccherahmmmuun modium replyvv2
0.
16n 136
nds a/s 1.5 1/6 2
125
stag
3ags
Kapa
196
Figure& 8: Value target ablation. minimum on locomotion tasks.
We compare the LCB target against the target using ensemble
20

---

Published as a conference paper at ICLR 2025
antmaze-umaze-vo
antmaze-umaze-
antmaze-meddum-pllyyoo
100
100
80
80-
60
40
40-
20
20
0.0
0.5
1.0
1.5
2.0
0.0
0.5
3.0
2.0
2.0
le6
medium diverse
large: play.vo
large diverse
80
40
20
20
0
2.0
stops
steps
stops
Figure Training curves for antmaze tasks.
pon human v1
pch donediv1
122-
120
102
200
New Wa
6-
3.00 Stago
1.5
2.00
7e0
lnc
Figure 10: Training curves for Adroit tasks.
52
50
10
2.5
1.5 <re4
0.2
0.5
2.2
0.0
1.0 stros
1.5
2.9
Figure 11: Training curves for Kitchen tasks.
21


Reviews:
[{"review_id": "llKqtEUeHY", "reviewer": "Reviewer_GEf6", "summary": "This paper integrates the advantage-weighted regression and diffusion model for offline reinforcement learning. This paper introduces the Diffusion Actor-Critic (DAC) framework, which theoretically formulates KL-constrained policy iteration as a diffusion noise regression problem. The experimental results demonstrate the advantages of DAC in terms of performance and stability. However, several concerns are as follows.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Overall the paper is well-written, making readers quickly capture the core idea. There are several advantages of this paper:\n\n1. DAC directly models the target policy as a diffusion model, which is an innovation in the field of offline reinforcement learning.\n\n2. Soft Q guidance adjusts the intensity of Q gradients according to noise scale, encouraging the generation of high-return actions while staying within the behavior support range. This approach effectively prevents actions outside the sampled distribution, enhancing the stability and performance of the algorithm.", "weaknesses": "1. The notation of this paper is complicated. \n\n2. The translation between Equations 13 and 14 probably is not acceptable. How could you replace the target policy with the offline dataset? Do you mean behavior policy?\n\n3. The design of Q-ensemble makes the results unconvincing. The Q-ensemble methods can solve the offline RL without any extra regularization. Given the design of the Q-ensemble, it's ambiguous whether the improvement comes from the proposed diffusion policy-constrained iteration or the Q-ensemble. The ablation of Fig. 4 seems to approve this. The baseline methods, such as DQL, should add a Q-ensemble to compare what makes it fairer. \n\n4. Although the paper emphasizes the importance of the LCB objective in stabilizing Q-value estimation, it lacks a deeper analysis of the impact of different pessimism factors, ρ.\n\n5. Insufficient comparison of offline RL, including but not limited to:\n\n[1] RORL: Robust Offline Reinforcement Learning via Conservative Smoothing.\n\n[2] Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization.\n\nSeveral previous methods using constrained policy learning for offline RL should be discussed at least:\n\n[3] Behavior proximal policy optimization\n\n[4] An Implicit Trust Region Approach to Behavior Regularized Offline Reinforcement Learning\n\n[5] Uni-o4: Unifying online and offline deep reinforcement learning with multi-step on-policy optimization", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aADnxpDf3E", "reviewer": "Reviewer_Wqqq", "summary": "This paper presents the Diffusion Actor-Critic (DAC) framework, aimed at addressing key challenges in offline reinforcement learning, particularly in policy optimization and high-reward action generation. By innovatively transforming the KL-constrained policy iteration problem into a diffusion noise regression problem, DAC introduces a soft Q-guidance mechanism. Through comparative experiments, DAC demonstrates superior performance over existing methods on multiple D4RL benchmark tasks, especially when handling datasets containing a large number of suboptimal trajectories. The experimental results indicate that DAC not only exhibits stable convergence but also surpasses other methods in overall scores, showcasing its potential in the field of offline reinforcement learning.", "questions": "- Soft Q-guidance relies on the accurate estimation of Q-values. If Q-values are overly optimistic or pessimistic, how does this affect the generated final policy? Have you considered the propagation of such uncertainty in the learning process? If the weight adjustment for soft Q-guidance is improper, could it lead to an over-reliance on Q-values in the generated policy, potentially resulting in overfitting?\n\n- Soft Q-guidance performs well in handling tasks, but that hard Q-guidance significantly deteriorates in performance when facing a large number of sub-optimal trajectories. Have you considered combining different guidance strategies to enhance the robustness of the algorithm? Specifically, have you evaluated the introduction of additional regularization methods in specific tasks to prevent the policy from overly relying on sub-optimal trajectories, thereby enhancing the model's stability and generalization ability?\n\n- In the antmaze task, the author noted that sparse rewards have a certain impact on the algorithm's performance. In this regard, have you designed more effective reward mechanisms or exploration strategies to address the challenges posed by sparse rewards? In practical applications, if the algorithm encounters sparse rewards, what measures do you anticipate taking to adjust the DAC algorithm to ensure it can still effectively learn and optimize the policy?\n\n- The authors mentioned using a lower confidence bound to counteract overly pessimistic estimates. Could this lead to a systematic bias in the Q-values? How significant is this bias in different task settings?\n\n- How does the $\\eta$ parameter in the objective function influence the optimization process? In certain scenarios, the choice of $\\eta$ might lead to oscillations or divergence of the algorithm. Is there a need for a sensitivity analysis regarding this choice? If gradient vanishing issues arise, particularly when the update frequency of the target policy is high, how should this potential problem be addressed?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\n\nThe DAC framework which combines diffusion models with policy optimization in reinforcement learning is novel. By introducing soft Q-guidance, the paper effectively overcomes the problem of overestimating Q-values in offline learning, enhancing the stability of policy improvement. This approach offers a new direction for future research.\n\nSignificance:\n\nThe authors conduct extensive experiments on D4RL benchmarks, covering a variety of tasks with results that are clear and convincing. The experimental results is strong\n\nClarity:\n\nThe paper is clearly written with illustrative examples", "weaknesses": "The authors should consider evaluate DAC in more offline reinforcement learning tasks, particularly in real-world application areas such as robotic control and atari games. And It would be better to provide specific choice of hyperparameter configuration", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9ixsHrNDr3", "reviewer": "Reviewer_sKYu", "summary": "This paper presents Diffusion Actor-Critic (DAC), a novel offline reinforcement learning algorithm designed to directly generate target policies using diffusion models. DAC formalizes the policy iteration problem with Kullback-Leibler (KL) constraints as a diffusion noise regression problem, achieving policy constraints through soft Q guidance, thus avoiding the generation of actions outside the offline data distribution. The paper demonstrates the outstanding performance of DAC in D4RL benchmark tests, surpassing current state-of-the-art methods in nearly all environments, while showing stable convergence and robust performance.", "questions": "How does DAC's generalization performance hold up in larger-scale real-world applications, such as autonomous driving or industrial robotics? Is there a plan to further test the scalability of this method?\n\nDoes the trade-off in soft Q guidance choice show differences in performance across various environments? Can the specific effects of different weight parameters on model performance be further explored?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Innovativeness: Introduces a new algorithm DAC and proposes the soft Q guidance, addressing several challenges in offline reinforcement learning.\n\nExperimental Performance: Demonstrates outstanding performance in D4RL benchmark tests, especially surpassing current state-of-the-art algorithms in locomotion and antmaze tasks.\n\nTheoretical Support: Provides clear theoretical justification by rigorously transforming the KL constrained policy iteration problem into a diffusion noise regression problem.", "weaknesses": "Lack of presentation: structure of the article is chaotic, failing to highlight key points, which creates some difficulty in reading.\n\nScope of Application: Although DAC performs well on standard benchmarks, its generalization ability in large-scale or complex real-world tasks has not been fully validated.\n\nThe advantages of Soft Q-guidance are not sufficiently pronounced compared to other methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g0utGnOnpq", "reviewer": "Reviewer_rDBF", "summary": "The paper proposes DAC, an offline reinforcement learning algorithm that uses diffusion models to manage out-of-distribution actions and prevent the overestimation of value functions. DAC formulates the KL-constrained policy iteration as a diffusion noise regression problem, enabling the direct representation of target policies as diffusion models. The approach includes soft Q-guidance for actor training and a Q-ensemble for critic training, stabilizing Q-gradient estimation. DAC outperforms state-of-the-art methods on the D4RL benchmarks, demonstrating its effectiveness across various environments.", "questions": "- line244 or so, did you experiment with estimating entropy in the EDP?\n- Why haven't you done experiments with Adroit Tasks and Kitchen Tasks and compared them with the latest algorithms? (It's not enough in the appendix, it's perfectly fine to put it in the body of the text for comparison)\n- In Equation 15, the Q-function is well-trained only within the original data distribution. However, the distribution of $x_t$ tends to be more random and can significantly diverge from the dataset distribution. How can we ensure that the Q-function's gradients with respect to $x_t$ remain accurate and avoid substantial extrapolation errors?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper incorporates diffusion models into constrained policy iteration by using score functions, addressing the issue of the untraceable KL divergence in diffusion policies. \n\n- The paper also presents extensive experiments and thorough ablation studies, demonstrating the effectiveness and robustness of DAC.\n\n- DAC captures the multimodality of policies using diffusion model.", "weaknesses": "- There are some 2024 papers missing in the offline RL summaries (existing citations are 2023 at best), and accordingly I would like to see your results against them in the experimental section eg. AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimization (He et al.), Diffusion Policies creating a Trust Region for Offline Reinforcement Learning (Chen et al.). \n\n- Recently, there is also a lot of work combining online RL and diffusion modelling, and I think a brief summary is also very necessary. Let me name a few famous works, you can add a paragraph to summarise eg. Policy Representation via Diffusion Probability Model for Reinforcement Learning(Yang et al.), Learning a Diffusion Model Policy from Rewards via Q-Score Matching(Psenka et al.), Diffusion Actor-Critic with Entropy Regulator(Wang et al.).\n\n- I don't seem to have seen you perform ablation experiments for LCB, as well as ablation experiments for diffusion step size. Especially with the ablation experiments for LCB, I'd like to see the proportionality of the improvement in performance of LCB for your final performance compared to the improvement in performance of your proposed method.\n\n- Experiments including $\\rho$ were not analysed for sensitivity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)