Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learning
Authors: Jifeng Hu
Sili Huang
Zhejian Yang
Shengchao Hu
Li Shen
Hechang Chen
Lichao Sun
Yi Chang
Dacheng Tao
Venue: NeurIPS 2025 poster ()
Keywords: offline reinforcement learning
decision making
generative models
Primary area: reinforcement_learning

Abstract: Conditional decision generation with diffusion models has shown powerful competitiveness in reinforcement learning (RL). Recent studies reveal the relation between energy-function-guidance diffusion models and constrained RL problems. The main challenge lies in estimating the intermediate energy, which is intractable due to the log-expectation formulation during the generation process. To address this issue, we propose the Analytic Energy-guided Policy Optimization (AEPO). Specifically, we first provide a theoretical analysis and the closed-form solution of the intermediate guidance when the diffusion model obeys the conditional Gaussian transformation. Then, we analyze the posterior Gaussian distribution in the log-expectation formulation and obtain the target estimation of the log-expectation under mild assumptions. Finally, we train an intermediate energy neural network to approach the target estimation of log-expectation formulation. We apply our method in 30+ offline RL tasks to demonstrate the effectiveness of our method. Extensive experiments illustrate that our method surpasses numerous representative baselines in D4RL offline reinforcement learning benchmarks.

Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learning
Jifeng Hu'
Sili Huang"
Zhejian Yang'
Shengchao Hu 5
Li Shen
Hechang Chen
Lichao Sun
Yi Chang"
Dacheng Tao
Jilin University Minzu University of China
& Shanghai Jiao Tong University *Shenzhen Campus of Sun Yat-sen University Lehigh University Nanyang Technological University
(hujf21, zjyang22) 00ai2s.malll..mmmmm jiu. edu. en {chenhc, yichang)@jlu edu, en huangsili Gmuc. edu. en mathshenli@gmail com charles tharles-hussjee ju, edu on 1is2218lehigh edu dacheng tao@gmail. com
Abstract
Conditional decision generation with diffusion models has shown powerful compet- itiveness in reinforcement learning (RL). Recent studies reveal the relation between sereg--funntionguuddancc diffusion models and constrained RL problems. 'he main challenge lies in estimating due to the oegexpectation formulation during the generation process. To address this issue, we propose the Analytic Energy-guided Policy Optimization (AEPO). Specifically, first provide a thcoretical analysis and the closed-form solution of the intermediate guidance when the diffusion model obeys the conditional Gaus- sian transformation. Then, analyze the posterior Gaussian distribution in the log-expectation formulation and obtain the target estimation the log-expectation under mild assumptions. Finally, train intermediate energy neural network to approach the target estimation of log-expectation formulation. We apply our method in 30+ offline RL tasks demonstrate the effectiveness of method. Extensive experiments illustrate that our method surpasses numerous representative baselines in D4RL offline reinforcement learning benchmarks.
intermediate energy, which is intractable
Introduction
Controllable generation with diffusion models has shown remarkable success in text-to-image genera- tion [9], photorealistic image synthesization [87], high-resolution video creation (8), and robotics manipulation [10] A common strategy to realize controllable diffusion models is guided sampling, which can be further classified into two categories: bussifierrguidde guided 66 I diffusion-basse methods.
48 and classifier-fic
Usually. classifier guidance and classifier-free guidance need paired data, i.c. samples and the
corresponding conditioning variables, to train controllable diffusion model 85 24 74). However, is difficult describe the conditioning variables for each transition in RI. We can only evaluate the value for the transitions with scalar and continuous function, which is generally called the action value function (Q function) [64] 48). Recently, mergg-function--uidee diffusion models provide an effective way to realize elaborate manipulation in RL. Previous studies also reveal the relation of equivalence between guided sampling energy- suwery--onnniiouuueeeee diffusion models and constrained RL
"Corresponding authors: Hechang Chen, Sili Huang, and Yi Chang. code: https///ithub..om/// hww.llyiie-neggg-guideee--eeee
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

problems 66 11. Considering the following formula of energy-funetion feres-fuuntio--uuieed diffusion models
min E Ex- (x)),
(1)
s.t. D
where a E RR, 8(x) : Rd - is the energy function, g(æ) and p(x) denote the unguided and guided data distribution, and (xI(.)) represents the KI. divergence. The optimal solution to the above problem
p(z) x q(x)e
(2)
where inverse temperature 3 is the Lagrangian multiplier, which controls the energy strength. Equa- tion (2) shows that the guided data distribution lies in the intersection region of unguided distribution q(x)) and energy distribution e -B)((m), Obversely, the guidance comes from the energy function {x)). The constrained RL problem
max DI (4) x
s.t.
(= <,
has the similar formula with Equation 1 where æ and μ are the learned and behavior policies. Thus,
π (a]s) x u(als))
The optimal policy '" [a]s) under state can be regarded as the guided distribution p(z). and actions can be generated with guided sampling. See Section 6.2 for more discussion and Appendix for detailed derivation.
Although sampling from p(z) is intractable due to the normalization term, it can be bypassed in score-based diffusion models, where the score function V, log p(r) can be calculated with intermediate energy {xx,) ) (Refer to Equation (8) for details.) [6] 28 However, the intermediate
methods has proved to be inexact theoretically and practically [64] 99). Besides, for classifier-guicde guided methods, the intermediate guidance can be influenced by the fictitious state-action pairs generated during the generation process. For classifier-fres grree-gided methods, the intermediate guidance is manually predefined during guided sampling. Though recent studies (64) propose the exact expression of & (zx), they do not investigate the theoretical solution of the intermediate energy.
energy introduced in classifier-guided and glassifier-free-guiie
In this paper, we theoretically analyze the intermediate energy (x;) (intermediate guidance is V., E,,,, (x;)) with a log-expectation formulation and derive the solution of intermediate energy by addressing the posterior integral. Based the theoretical results, we propose a new diffusion-based method called Analytic Energy-guided Policy Optimization (AEPO). In contrast with previous studies, leverage the characteristic of Gaussian distribution and find solution for the log-expectation formulation. Specifically, we first convert the implicit dependence on the action in the exponential term explicit dependence by applying Taylor expansion. After that, we investigate the posterior distribution formulation of the expectation term and simplify the intractable log- oog-xpectaation formula- tion with the moment- genertting function of the Gaussian distribution. Then, we propose to train a general Q function and an intermediate energy function to approach the simplified log-ex -egeeeecttaiin formulation given batch of data. Finally, during the inference, we can use the intermediate guidance to generate guided information to high-return action distributions.
To verify the effectiveness of our method, apply our method in offline RL benchmarks D4RL 21 where we select different tasks with various difficulties. We compare our method with dozens of baselines, which contain many types methods, such as classifier-guided and classifier -rideg diffusion models, behavior cloning, and transformer-baut models. Through extensive experiments, we demonstrate that our method surpasses state-of-the-art algorithms most environments.
2 Related Work
2.1 Offline RL
Offline RI. aims to learn a policy entirely from previously collected datasets, thus avoiding expensive and risky interactions with the environment, such autonomous driving [56]
35
53
30
60 44 45).

---

However, in practice, we may face the distribution shift issue, which means that the learned policy and the behavior policy are different, and the overestimation of out-of-distribution (OOD) actions will lead to severe performance drop 55
46 In order to solve these issues, previous studies can be roughly classified into several research lines. Policy regularization (54] focus on applying constraints to the learned policy to prevent it from deviating far from the behavior policy. Critic penalty methods [55] [3 65
27
22
methods 7 66
88 22
68 43] propose to train conservative value function that assigns low expected return unseen samples and high return on dataset samples, resulting in efficient OOD actions overestimation. Uncertainty quantification methods 88 2 4 [83] introduce uncertainty estimation to identify whether an action is OOD, thus
enhancing the robustness on OOD actions. Recently, generative models [26] 57 8 39 37 38 [5], such diffusion models, are proposed to augment the offline datasets with synthetic experiences or train a world model for planning.
2.2 Generative Policy Optimization
Recent advancements 48 in diffusion RL methods have shown the diffusion models' powerful
expression in modeling multimodal policies [1] 82 34), learning heterogeneous behaviors 58),
and generating fine ine-grained control [40] 16 33,. Diffusion- based RL algorithms cursorily contain two
types of guided sampling to realize policy optimization [12]. Guidance-based policy optimization [48
64 [1] uses the value function to instruct the action generation process of diffusion models. During the intermediate diffusion steps, the generated action vectors will incline to the region with high
values according to the intermediate guidance. Selected- basedd policy optimization 11 25 67
first generates a batch of candidate actions from diffusion behavior policy. Then, it constructs a critic-weighted empirical action distribution and resamples the action from for evaluation.
3 Preliminary
3.1 Diffusion Probabilistic Models
Diffusion Probabilistic Models (DPMs) 29 17 74 are proposed to construct the transformation between complex data distribution To and easy sampling distribution øт (e.g.. Gaussian distribution). By defining the forward transformation from data distribution 2o to simple distribution 27 T time range according to the following stochastic differential equation dr f(z, t)dt + g())dw, where f(;t):: 2 Rd is the drift coefficient, 9(-) is the diffusion coefficient, and w is the standard Wiener process, we can obtain a reverse transformation from the simple distribution ØT to data distribution To. as shown in dr [f( t)- v log (=xr)] dt - g(f)dw, where w is the standard Wiener process, t E [0. T is the diffusion time, = 0 means data without perturbation, t = T means prior cample-friendee distribution, dt indicates infinitesimal negative timestep. qe (x(x) is the marginal distribution of zg. and V, log q (z) is the score function. Usually, we adopt the Gaussian transition
distribution 62 63 which means
@no(=e/z0) N (are; a, of
(5)
where 04 > O a decrease function, at > is an increase function, (r1((1((m)) R ((T: 0, I) for certain o. Obviously, (=) will be independent zo, i... (21o(=T(ar)) (mo) RM qr (xT). Based on the above results, once we obtain V. log 4(xx), we can perform genera- tion from simple distribution 5 86). By constructing the score function s(xx), previous stud- ies (78 (77) show that the following objective is equivalent: (y((s)) [[]s(z,) Va, log (xx)))) +
(g(aa))e (x)) [lls(x))) V,, log ( [[]] and the reverse transformation can be an alterna- tive formula, i.e., probability flow ordinary differential equation (ODE)
dr,
1 = f(t)= 2 (t) z, log (x())
(6)
dt
where fxx,t) f(t)2, f(t) d logar and g(t) d do 2d log of because Equation (6) holds the marginal distribution unaltered [32] Considering that x, = a, o,e where e ~ (0, I). there exist sxx2) Va, log pe(x.) ae So, we can introduce neural network 60(21, 1), and
Easf Exo [Ileo(z,, t) =l12]
3

---

is the diffusion loss, where U--) is uniform distribution. 20 ((r)) means sampling data from the offline datasets. After training. we can obtain the score function value through Eg, i.e. Vn, log pe ~ se, Then, we can use Equation (6) for the generation process 62
3.2 Guided Sampling
In RL. guidance plays an important role in generating plausible decisions because the dataset quality is usually mixed, and naive modeling of the conditional action distribution under states will lead to suboptimal performance. Classifier-guided methods [17
82 50 define binary random optimality
variables O where O = means optimal outputs, and O = 0 means suboptimal outputs. For each generation step p( O) x 1(4 12), the guidance serves as gradient on the mean value modification p (z. O) = (z p4 + Σg V log p(O)az), Σ,), where μ4 and Σ, the predicted mean and variances of q(x,-112z)) During inference. the fictitious intermediate outputs that do not exist in the training dataset will lead to suboptimal intermediate guidance. Different from classifier- guided methods, the classifier-fre--giinee distribution between the data and condition variables C in the training phase [
methods implicitly build the joint
61 12]. Therefore,
we can use the desired condition as guidance to perform guided sampling. The classifier- free- gree-gguded training loss is Ezo (gzo)) ( ((aBB()) [I]e b* C, f) =l where B is binomial distribution, is the parameter of B. During inference, the guidance is also incorporated in the mixed prediction of conditional and unconditional noise, i.e., é = c0 t. () + w<co (24,, t, C) co (Zz, t, 0)), controls the guidance strength, 0 means b = 0. Due to the condition variables being needed before generation, we need to manually assign the value before inference. The predefined C will also restrict the model's performance if insufficient prior information on environments arises.
3.3 Diffusion Offline RI
Typical RL [69| is formulated by the Markov Decision Process (MDP) that is defined as the tuple M (S, A. P, r, 7), where S and A denote the state and action space, respectively, P(s'|s, a) is the Markovian transition probability, r(s, a) is the reward function, and y E [0, 1) is the discount factor. The goal is to find a policy æ that can maximize the discounted return E, D **((sk ax)], where A represents the RL time step which is different from diffusion step & [76). In offline RL 55 53 only a static dataset D,, collected with behavior policy is available for training. Ex- tracting optimal policy from offline datasets is formulated as a constrained RL problem 7721 84] as shown in Equation
B which can be converted to find an optimal policy π²" that maximizes ((ss.a) Y (+()))))
Diffusion offline RL usually adopts diffusion models to imitate the behavior policy μ [. Naively modeling the action dis- tribution will only lead to suboptimal policy. Thus, the well-trained value functions will be used to extract optimal policy, such as action value gradient guidance [48] and empirical action distribution reconstruction (25).
n max,, Eg- (a
4 Method
In order to generate samples from the desired distribution p(x) with the reverse transformation (generation process) Equation (6). should know the score function Vglog p(x). If the score function of the desired distribution p(r) has relation with the score function of g(x)
V log p(x)= = V log q(z) + V BE(x).
(8)
for any data x, we can obtain the score function of p(x) by compounding the score function of q(x)) and the gradient of E(x). However, it only exists pa (xo) (0( for the samples in the dataset rather than the marginal distribution of pt (2x)) and ge (2t)). Previous guided sampling
methods usually adopt MSE or diffusion posterior sampling (DPS) as the objective of training the intermediate energy, which cannot satisfy the relation p. (zx) qe (x, E thus leading to an inexact intermediate guidance. We summarize the results in Theorem 4.1
Theorem 4.1 (Inexact and Exact Intermediate Energy). Suppose po(zo) and (o( has the re- lation of Equation ( (=))())) and (rjo(z1/20) are defined by Pejo (2))))) == (e)o(z1220) = NZ of for all t (0,T). According to the Law of Total Probability, the marginal

---

distribution Po(at) and qe (Z())) are given by Pt (2s) f PHo(zz) |oo (o((zo)dzo and qe (e(x))
(ejo(Zelrr) (o (ro. Previous studies 164 E (Ix) = (2oo) (E())) >> 0; & (xx) (( (20)), 0.
67 141 define the inexact intermediate energy as
(9)
The exact intermediate energy is defined by
(e(xo)) Ec(tra) - -lg (Pal (e-(oo))) ,12 0.
(10)
It can be proved (See Appendix for details.) that ps (3t) x qe(xe)e -E((re)
Dor
(((((e)) exists under Equation (IO) is inexact for previous
rather than Equation ( So, the intermediate guidance (Equation (8)
cesssffee-guidees and classffer-freee-guuee
methods.
Similarly, in RL, as shown in Equation (4). the desired distribution is (als)) The intermediate energy (s, ag) and the score function Va, log (e(ay)s) with intermediate guidance V E, 0,) in RL are defined as follows
Es, = (s)(s,
1= 0
log B [e
t t> 0
(11)
Va, log (a Va, log He(arls) E
(12)
where we slightly abuse the input of because the value action should depend on the corresponding state in RL. Refer to Appendix D for the detailed derivation.
4.1 Intermediate Energy
Diffusion loss (Equation provides the way obtain the score function of pa(a:ls).
Va, log pr(a; ss) (s, 0t,
(13)
at
Obviously, the most challenging issue that we need to address is log P [s(,00)]] because of the intractable log-expectation formulation. Following previous research, we use a Gaussian distribution as the estimate for the posterior distribution ( aq, s), where the mean and covariance are denoted as Poje = N (Noje Eoj)) We will introduce how to approximate öop and En below. We first focus on converting the implicit dependence the action in the exponential term to explicit dependence by applying Taylor expansion. Expand Q(s, an) at ag = a with Taylor expansion, where is constant vector. Then, we have
Q an) x Q(s, ao)lama oQ(s, ao) laoma * (ao - a).
(14)
dag
Replacing Q function in log B Bag a e) with Equation 14) we will derive the following
log Eag-piaa s) se (Q(e,0o) xe (Q - (Q'(s, Ta + log Eagr-ptaola QQ'(e an
(15)
More discussion about the approximation of intermediate energy be found in Appendix and Note that the above derivation makes the complex dependence between Q(s, an)
where Q' aQ Bo
and ag easier. In other words, we transfer ag from the implicit dependence on Q(s, ag) explicit dependence on Q'(s, a) ag. Refer to Appendix H for more details. As for the only unknown item
(s,a) ao), the exact result can be derivated from moment generating function, Pao~a(eolas... [e (e
Ja Ea By simplifying Equation (15). we have which indicates that Ex,N(ex)) *I ea Nut
log (Bao-picolas.)) (Q(aa))) (aoa)"a0] {0o} 3Q'(s,, ( Aol= +
1 1 3 a) T Q',
Finally, the intermediate energy Ex(8, (z) with log-expectation formulation can be approximated by R(( m (Q a) (Q'(ss ((oo) a) Q²Q 2o a). log Fagnutallo, (
(16)
From Equation [6), we can see that the intermediate energy can be approximated with a, Pojo and Eoje- In the next section, will introduce how to approximate the parameters of the posterior distribution ul Ja, s). The training algorithm is shown in Algorithm I of Appendix A
5

---

4.2 Posterior Approximation
In this paper, we provide several methods to approximate the distribution of yuay at (), We use "Posterior i" to differentiate different posterior approximation methods in the following contents.
Posterior 1. Inspired by previous studies [6], the mean vector Poje of the distribution Mojo (aol at, 8) = (as; Ao Eoje). where the mean vector
the trained diffusion model eg to obtain
Poje
(a. 0rea(s, ot.
(17)
Ott
according to Equation
As for covariance matrix Eopz following the definition of covariance
1
pooo [(oe- axaa))aa ar,00) o7
(aj(
E -
EDE
(18)
ai
of
where in the second equation we use Ep llan az)[(a0 - + ar) * 2 Ea] =
(alla.. a) (aola.)) [oo] [|oo
- az)
o² Steg (Non De az))) Ce ot egegg , to eØ(s, ae !), and Maye = E Paje (aal lo). [co]. Besides,
ax we also notice that pa(a)) ~ "Naa;; aro0, I) thus we have
Ore
Buoj (R [[a+ - aga aqos) ! = I.
(19)
To simplify the problem of calculating the exact covariance matrix, we adopt two strategies. 1) The isotropic Gaussian assumption: Computing an exact covariance matrix is expensive (if the matrix size N, it requires N2 computation times.). Under the isotropic Gaussian assumption, we only need N computation times, which significantly reduces the computational cost. 2) The marginalization over ag: Since a, is sampled from a distribution, data points far from the mean can lead to instability marginalize over a, to use the average effect to replace the specific effect of each individual a, on ag. Based on the above two strategics, we have
during training. Therefore,
o [ En,(a x) (EEE I].
E : (p, (s) (at)
(20)
at
For simplicity, we usually consider isotropic Gaussian distribution, where the covariance Eoje satisfies Eoj = obe I, the covariance can be simplified as
a. - o.22 In the experiments, we adopt this posterior as the default setting to conduct all experiments. Refer to Appendix [. for the detailed derivation.
[ [lle (a,, 0)
1
(21)
at
As Equation (7 shows, we still adopt the same mean of the distribution Hoje(ao (at, s). We assume that the Goje az) obey the formulation Moje (agla, s) (N Molz= Egee) Again considering Equation (18. we reformulate (79) it as
Posterior 2.
(o)r(oe) = Poo (aa (a) un) 7 (Aoje - un) - -o)
(22)
where Up is any constant vector, which has the same dimension with Äop- When we consider isotropic Gaussian distribution and remove the inference of a,, the Eoje is simplified as Σnjt = 0 I and
off Var(as) - " [1) moll))
(23)
where = Eao [4o] and we can use the mean value of the whole action vectors of the dataset as an unbiased estimation. Var(ag) can be approximated from a batch of data or from the entire dataset. We can sample batch of data o₁ with different & calculate approximation solution of
the second term (o.. [ll/o vol mo In Appendix 1.2 we provide the detailed derivation. From the above theory, According to Equation [6), we have simplified intermediate energy:
solve the Gaussian distribution (aolaa, s)'s parameters (Mojes Σoje)
1 8 * I|Q'(ss, 2)l2.
E1,,,,) R 3Q(s, a) + 3Q' (s, a)T [Toj - a) + 2
(24)
and the intermediate energy training loss Li is
Lig E [| a, 1) - E, 2
(25)
where θ is the parameter, the mean and covariance are given by Equation
21 and 23

---

Table Offline RL algorithms comparison on D4RL Gym-MuJoCo tasks, where we use red color and blue color to show diffusion- based and non-diffusion-based baselines. Our method is shown with yellow color.
Dotaset
Med-Expest Hopper
Medium
Med-Replay Hopper
meeas score
total scare
HallChhesahh
Walker3d
HallChectah
Hogper
Walker2d
HallChoetsh
Walker2d
AWAC
42.8
55.8
74.5
43.5
$7.0
72.4
40.5
37.2
27.0
50.1
450.7
55.2
52.5
466.7
MOPO
67.5
379.3
MEOP
105.9
55.1
70.2
44.5
48.8
41.0
42.3
12.4
9.7
47.8
430.0
MORa.
53.1
108.7
95.6
42.1
95.4
77.8
40.2
93.6
49.8
72.9
636.5
TAP
91.8
105.5
107.4
45.0
63.4
64.9
4038
87.3
66.8
74.8
672.9
26.0
BCQ
57.5
33.1
457.7
COL
62.4
98.7
111.0
44.4
58.0
79.2
46.2
48.6
26.7
63.9
575.2
TIN+
90.7
98.
110.1
48.3
99.3
83.7
44.6
60.9
81.8
75.3
677.4
JQL
86.7
91.5
109.6
47.4
66.3
78.3
44.3
91.77
73.9
77.0
692.6
90.7
98.0
110.1
42.6
67.6
74.0
36.6
82.7
65.6
74.3
668.9
TT
95.0
110.0
101.9
46.9
61.1
79.0
41.9
91.5
82.6
78.9
709.9
BooT
94.0
102.5
110.
50.6
70.2
82.9
46.5
92.9
87.6
$1.9
737.4
86.2
D-QUa#
69007
Diffusee
38.9
103.3
106.9
42.8
74.3
79.6
37.7
93.6
70.6
77.5
697.7
TD
9016
111.8
108
49.1
70.3
82.5
30.3
100.0
75.0
81.8
736.4
IDQL
95.9
108.6
112.7
51.0
65.4
82.5
45.9
92.1
85.1
82.1
739..2
HUMI
76.4
99.6
80.7
143.0
AdaptDiftuser
749.8
87.8
99.1
95.9
759.2
HD-DA
92.5
115.3
107.1
46.7
99.3
84.0
33.1
94.7
84.1
84.6
761.8
Consistncy--e -CC
84.3
100.4
110.4
69.1
90.7
83.1
58.7
99.7
79.5
85.1
765.9
TCD
92.7
112.6
111.3
47.2
99.4
82.1
4016
97.2
88.0
85.7
771.0
777.1
QGPO
93.5
179.2
AEPO
94,4108
111.5411
109.3
49.6
100.2:05
862:11
43.7813
101.0165
90.8-15
87.4
786.7
4.3
Q Function Training
Due to the fact that (s, a²) rely on Q(s, a), we first incorporate an action value function (s a) with parameters di to approximate the action values [36]. As for the training of the (p a), leverage the expectile regression loss to train the Q function and V function:
Lv D),,a) [S(Vo(s) Qs(s,a))),
LQ E E(s,,,''>>, (l)r + Vo(s') Qu(s,
(26)
L5(w) = T - 1yy 0)1
where is the parameters of value function V, is the parameters of target Q controls the weights of different y. There are also some other methods suitable for learning the Q function from offline datasets, such as In-support Q-learning [64] and conservative Q-learning (55). However, they use either fake actions that are generated from generative models or over-underestimat values for out-of-dataset actions, which influence the learned Q values and the calculation of the intermediate energy. We defer more discussion of the training of the Q function on offline datasets in Appendix []]
4.4 Guidance Rescaling
As shown in Equation (2). the magnitude and direction of Va, log (e(aa)s ss) will be easily affected by the gradient of the intermediate energy. Previous studies usually need extra hyperparameter w to adjust the guidance degree to find better performance, i.e., V.. log ग. (a,|s) w W.. Es ).). However, it poses issues to the performance stability of different w during inference. Inspired by the experimental phenomenon that when the guidance scale is zero, the inference performance is more stable than that when the guidance scale is non-zero. We propose neroomaleize Va, log (a(a)))) the magnitude of Va, log Te(a:ls) by Va. log (arls) ||V., log (:(2:1))] V/ log [r(ar[s)]]-
Va. log 2e (r
Experiments
In the following sections, we report the details of environmental settings, evaluation metries, and comparison results.
7

---

Table 2: D4RL Pointmaze (maze2d) and Locomotion (antmaze) comparison. We select 12 subtasks for evaluation, including different difficulties and reward settings. We use red color blue color and yellow color
to show diffusion-ben baased baselines, non-diffusion--asee baselines, and our method.
Environment
maze2d-umaze
maze2d-medium
maze2d-large
mean sparse score
mean dense score
Envirumment lype
sparse
sparse
dense
sparse
dense
DT
31.0
8.2
2.3
13.8
BCQ
49.1
17.1
30.8
32.3
33.9
IQL
42.1
34,9
61.7
46.2
COMBO
76.4
68.5
14.1
53.0
TD3+BC
14,8
62.1
88.6
55.2
BEAR
65.7
25.0
81.0
57.2
88.9
14.6
38.3
1.5
17.1
42.9
16.0
CQL
29.6
62.0
TT
68.7
46.6
34,9
52.7
27.6
56.6
43.7
52.0
73.8
74.0
SynthER
66.4
143.3
102.9
Ditfuser
113.9
121.5
123.0
119.5
HDMI
120.1
121.8
128.6
123.5
HD-DA
72.8
45.5
42.1
54.7
80.7
45.7
63.2
48.6
TCD
128.1
29.8
132.9
41.4
146.4
75.5
135.8
48.9
116.2
83.2
122.3
78.2
125.9
23.0
61.5
AEPO
136.0
107.2
128.4
109.9
132.4
165.5
132.3
127.5
Environment
antmazc-umaze
antmazc-medium
antmazo-large
mean score
total score
Envirumment lype
diverse
play
diverse
play
diverse
AWAC
56.7
49.3
0.0
0.7
0.0
1.0
18.0
107.7
DT
59.2
53.0
0.0
0.0
0.0
0.0
18.7
112.2
BC
65.0
55.0
0.0
0.0
0.0
0.0
20.0
120.0
BEAR
73.0
61.0
0.0
8.0
0.0
0.0
23.7
142.0
BCQ
78.9
55.0
0.0
0.0
6.7
23.8
142.8
TD3+BC
3.0
0.2
27.3
163.8
CQL
74.0
84.0
61.2
14.9
50.6
303.6
IQI.
87.5
62.2
71.2
70.0
39.6
47.5
63.0
378.0
QDQ
98.6
67.8
81.5
85.4
35.6
31.2
66.7
466.8
DD
73.1
49.2
0.0
24.6
0.0
7.5
25.7
154.4
D-QI.
93.4
66.2
76.6
78.6
46.4
56.6
69.6
417.8
IDQL
93.8
62.0
86.6
83.5
$7.0
56.4
73.2
439.3
SIBC
92.0
85.3
81.3
82.0
59.3
45.5
74.2
445.4
QGPO
96.4
74.4
83.6
83.8
66.6
64.8
78.3
469.6
AEPO
100.0
100.0
76.7
83.3
56.7
66.7
80.6
483.4
5.1
Environments
We select D4RL tasks 21 as the test bed, which contains four types of benchmarks, Gym-MuloCo, Pointmaze, Locomotion, and Adroit, with different dataset qualities. Gym-MuJoCo tasks are com- posed of HalfCheetah, Hopper, and Walker2d with different difficulty settings datasets (e.g., medium, medium-replay, and medium-expert ture level of behavior policies and 'medium' represents uni-level behavior policy, In Pointmaze, we select Maze2D with three difficulty settings (umaze, medium, and large) and two reward settings (sparse and dense) for evaluation. Locomotion contains ant robot control task with different maze sizes, which we distinguish using umaze, medium, and large. These datasets include an abundant fraction of near-optimal episodes, making training challenging. Adroit contains several sparse-rewane high-dimensional hand manipulation tasks where the datasets are collected under three types of situations (human, expert, and cloned).
where 'medium-replay' and 'medium-expert' denote the mix-
5.2 Metrics
Considering the various reward structures of different environments, select the normalized score as the comparison metric. As an example, the normalized score Enorm is calculated by E- G-Erandam-
Enorm Eexpert -Erandom * 100, where Erandom and Exxpert are the performance of random and expert policies, and E is the evaluation performance. In the ablation study, we select the radar chart

---

to compare the holistic capacity of all methods, where the normalized performance is calculated by E, Enorm max (E, where E, is the evaluation performance of method i, indicates the index of comparison methods, and N is the total number of methods.
5.3 Baselines
For baselines, we compare our method with diffusion-basede methods (DiffuserLite 19. HD-
DA [TO] IDQL (25 AdaptDiffuser [59]. Consistency-AC [18], HDMI [57. TCD [1]
QGPO 664, SÍBC MI. DD [1. Diffuser [48]. D-QL [82]. and D-QL@1 1, etc.) and non-
diffusion-based methods, including traditional RL methods (AWAC [7] and BC), model-based
methods (TAP [4)], MOReL (51], MOPO [89. and MBOP B constraint-based methods
(CQL [55], BCQ [23]. and BEAR [54], etc.), uncertainty- methods (PBRL 4. TD3+BC [22]. and IQL [53], etc.). and transformer- methods (BooT [80], TT 47). and DT [3]
etc.). In summary, compared more than 30 competitive methods across over 30 subtasks.
5.4 Results
In [ we report the comparison results with dozens of diffusion- based and non-diffusion- based methods the D4RL Gym-MuJoCo tasks. Among all the algorithms on 9 tasks, the best performance of our method (AEPO) illustrates the effectiveness scenarios. Especially, in guided sampling, diffusion model, our method not only provides an approximate solution to guided sampling in theory but also surpasses most recent diffusion- based methods, such as QGPO, DiffuserLiit Lite, and D-QL, in abundant experiments. In order validate the effectiveness in sparse and dense reward settings, as well as hard tasks with the most sub-optimal trajectories, we conduct experiments on maze2d and antmaze tasks. The results are reported in Table where we can see that our method approaches or surpasses the SOTA algorithms, Limited by the space, we postpone more experiments and discussion of D4RL Adroit in TableBoof Appendix
ARMO
AEPO-QQL
AEPO-IOQU
bullchretal-medllaae-
hopper experte----t-tt
expert
decision-making
lepper-acediue walker2d medium
butlchretab- mi-mnniun---pla
7: hopper -edium--eepee
8: wolber26-undieee replay
Figure I: Q function training and posterior approx- imation ablation of AEPO on Gym-MuJoCo tasks.
Apart from Posterior 1 introduced in Section we also investigate the performance of Posterior 2 on the D4RL Gym-MuloCo tasks. To show the performance differences obviously, we adopt the holistic performance and show them on the polygon shown Figure [ where each vertex represents sub-task. The fuller the polygon, the better the overall performance of the model. From the figure, it can be observed that AEPO outperforms AEPO-Postentee attributed to the variance calculation. As shown in Section 4.2, Posterior is influenced by two factors one being the intrinsic variance of the dataset and the other being the diffusion model. In contrast, Posterior is influenced solely by the diffusion model. For the different Q function training strategies, we also conduct experiments on the Gym-MuJoCo tasks. AEPO-CQL uses the contrastive g-learning to train the Q function, and AEPO-ISQL adopts in-support softmax q-learning to train the Q function. The results in Figure [ illustrate that implicit Q-learning is an effective method to learn better Q function in offline training.
eco-rovale
This may be
e-43
4.4
Figure 2: Guidance rescale ablation of AEPO on Gym-MuJoCo seiterrddmmdiuum-eee task. The x-axis denotes training steps.
As shown in Figure 2 guided sampling, such as w = 0.1, is important to reach higher returns compared with non-guided sampling (w = 0). With the increase of w, the performance of 'non- rescale' decreases quickly, while 'rescale" can still hold the performance. Obviously, guidance rescaling makes it robust find better performance in wider range of w.

---

Discussion
6.1 Differences Discussion
Previous methods, such as QGPO, cannot solve the intractable intermediate energy with the log- expectation form, which leads to sub-optimal performance. Facing this issue, we resolve the in- tractable log-expectation of intermediate energy with Taylor expansion and the moment generating function, achieving better performance in 33 tasks with various domains. Below we discuss the differences between our method and QGPO in detail.
From the perspective of theory, our method provides a theoretical solution for the log-expectation of the intermediate energy, which takes further step compared with QGPO. QGPO only derives the formula of intermediate energy rather than the further results the log-expectation.
From the perspective of algorithmic design, in Theorem 3.2 of QGPO, where the authors contrastive learning loss to the intermediate energy, the authors mention that, under infinite data and model capacity, the contrastive loss (Equation (12) in QGPO) can perfectly approximate the intermediate energy. However, in practice, we cannot achieve infinite model capacity, and offline samples certainly limited. Our method does not require assumptions about the sample size model capacity.
From the perspective of the dataset, QGPO requires the of diffusion model to generate fake action vectors for CEP, and these fake action vectors may introduce the influence of out-of- distribution actions. However, our method does not require the generation of fake actions.
Compared with posterior sampling methods, such as DPS, the key difference between DPS and AEPO is that the intermediate energy formula used in AEPO exact, while that in DPS is inexact. The derivation of the exact and inexact intermediate energy can be found in Table 1 and Appendix E of QGPO. AEPO first bases on the exact intermediate energy, then solves the log-expectation that QGPO does not resolve.
6.2 Intuition Behind Using the Q-function as the Energy Function
In reinforcement learning, the Q-value Q(s, a) represents the expected return of taking action in state Higher Q-values indicate more desirable actions. In energy-based guided sampling, the energy function Exx) serves as a scoring function, where lower energy corresponds to higher sampling probability on By identifying the energy function as E(x) (Q(s, a). we make high-Q actions more likely to be sampled through the exponential weighting (Q(s, a). This aligns with the optimization objective in constrained RL: maximizing the expected Q-value under a KI. divergence constraint from a prior behavior policy.
Conclusion
In this paper, provide theoretical analysis of the intermediate energy that matters in conditional decision generation with diffusion models. We investigate the closed-form solution of intermediate guidance that has intractable log-expectation formulation and provide effective approximation method under the most widely used Gaussian-basee diffusion models. Finally, we conduct sufficient experiments in 4 types, 30+ tasks by comparing with 30+ baselines validate the effectiveness. Limitations and potential improvements can be found in Appendix B.8
Acknowledgement
We would like to thank Lijun Bian for her contributions the figures and tables of this manuscript.
We thank Siyuan Guo for his contributions to the writing suggestions of this manuscript. This work is supported part by the National Key R&D Program of China (No. 2023YFF0905400.
No. 20212D011 2500)) the National Natural Science Foundation of China (No. 624761 10, No. U2341229); the National Key R&D Program of China (No. 2023YFF0905400, No. 2021 12500): the Key R&D Project of Jilin Province (No. 20240304200SF); NSFC Grant (No. 62576364).
10

---

References
[1] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657. 2022.
[2] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
[3] Arthur Argenson and Gabriel Dulac-Amold. Model-based offline planning. arXiv preprint arXiv:2008.05556 2020.
[4] Chenjia Bai, Lingxiao Wang, Zhuoran Yang. Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-die drrivnn offline reinforcement learning. arXiv preprint arXiv:2202. 11566 2022.
[5] Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang. Estimating the optimal covari- ance with imperfect mean in diffusion probabilistic models. arXiv preprint arXiv:2206.07555 07309 2022.
[6] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. arXiv preprint arXiv:2201.06503, 2022.
[7] Alex Beeson and Giovanni Montana. Balancing policy constraint and ensemble size in uncertainty- offline reinforcement learning. Machine Learning, (33(1):443- 488, 2024.
[8] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL UtL/htps::///// httpss///peeaaat com/research/
conneration-oode
-s--srswwwo simulators
[9] Pu Cao, Feng Zhou, Qing Song, and Lu Yang. Controllable generation with text-to-image diffusion models: A survey. arXiv preprint arXiv:2403.04279, 2024.
[10] Chang Chen, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchi- cal planning with diffusion. arXiv preprint arXiv:2401 1.64444444, 2024.
[11] Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXi::2209.14::: 14548, 2022.
[12] Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimization through diffusion behavior. arXiv preprint arXiv:2310.07297, 2023.
[13] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement lcarning via sequence modeling, Advances in neural information processing systems, 14:15044-15097, 2021.
[14] Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Marc L Klasky, and Jong Chul Ye. Diffu- sion posterior sampling for general noisy inverse problems. arXiv preprint 2mXiv:2209.1462 2022.
[15] Yang Dai, Oubo Ma, Longfei Zhang, Xingxing Liang, Shengchao Hu, Mengzhu Wang, Shouling Ji, Jincai Huang, and Li Shen. Is mamba compatible with trajectory optimization in offline reinforcement learning? Advances in Neural Information Processing Systems, 37:51474-5150. 2024.
[16] Sudeep Dasari, Oier Mees, Sebastian Zhao, Mohan Kumar Srirama, and Sergey Levine. The ingredients for robotic diffusion transformers. arXiv preprint urXiv:2410.10050 2024.
[17] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.

---

[18] Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning. arXiv preprint arXiv:2309.16984 2023.
[19] Zibin Dong, Jianye Hao, Yifu Yuan, Fei Ni, Yitian Wang, Pengyi Li, and Yan Zheng. Diffuserlite: Towards real-time diffusion planning. arXiv preprint arXiv:2401. 15443, 2024.
[20] Clive Dym. Principles of mathematical modeling. Elsevier, 2004.
[21] Justin Fu, Aviral Kumar, Ofrr Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004. 07219, 2020.
[22] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
[23] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference machine learning, pages 2052-2062. PMLR, 2019.
[24] Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as alag-andpplay priors. Advances in Neural Information Processing Systems, 35:14715-14728,
2022.
[25] Philippe Hansen-Estrucee Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
[26] Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xuelong Li. Diffusion model is effective planner and data synthesizer for multi-task reinforcement learning, Advances in neural information processing systems, 36:64896-64917, 2023.
[27] Longxiang He, Deheng Ye, Junbo Tan, Xueqian Wang. and Li Shen. Robust policy expansion for offline-to-online rl under diverse data corruption. arXiv preprint arXiv:2509.24744 2025.
[28] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. / 2299, 2022.
[29] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[30] Jifeng Hu, Yanchao Sun, Hechang Chen, Sili Huang, Yi Chang. Lichao Sun, et al. Distributional reward estimation for effective multi-agent deep reinforcement learning. Advances in Neural Information Processing Systems, 35: 35:12619-12632, 2022.
[31] Jifeng Hu, Yanchao Sun, Sili Huang, Yiau Guo, Hechang Chen, Li Shen, Lichao Sun, Yi Chang, and Dacheng Tao. Instructed diffuser with temporal condition guidance for offline reinforcement learning. arXiv preprint arXiv:2306.04875. 2023.
[32] Jifeng Hu, Sili Huang, Li Shen, Zhejian Yang, Shengchao Hu, Shisong Tang, Hechang Chen, Yi Chang, Dacheng Tao, and Lichao Sun. Solving continual offline through selective weights activation on aligned spaces. arXiv preprint arXiv:2410. 00.4400555600 15698, 2024.
[33] Jifeng Hu, Sili Huang, Siyuan Guo, Zhaogeng Liu, Li Shen, Lichao Sun, Hechang Chen, Yi Chang, and Dacheng Tao. Decision flow policy optimization. arXiv preprint arXiv:2505.20350. 2025.
[34] Jifeng Hu, Li Shen, Sili Huang, Zhejian Yang, Hechang Chen, Lichao Sun. Yi Chang, and Dacheng Tao. Continual diffuser (cod): Mastering continual offline rl with experience rehearsal. IEEE Transactions on Neural Networks and Learning Systems, 2025.
[35] Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning. In European Conference Computer Vision, pages 533-549. Springer, 2022.
12

---

[36] Shengchao Hu, Ziqing Fan, Chaoqin Huang. Li Shen, Ya Zhang, Yanfeng Wang, and Dacheng Tao. Q-value regularized transformer for offline reinforcement learning. International Conference Machine Learning, pages 19165-19181. PMIR, 2024.
[37] Shengchao Hu, Li Shen, Ya Zhang, Yixin Chen, and Dacheng Tao. On transforming reinforee- ment learning with transformers: The development trajectory. IEEE Transactions on Puttern Analysis and Machine Intelligence, 46(1 (6611):8580-8555) 2024.
[38] Shengchao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Learning multi-agent communication from graph modeling perspective. In The Twelfth International Conference on Learning Representations, 2024.
[39] Shengchao Hu, Shen, Ya Zhang, and Dacheng Tao. Graph decision transformer for offline reinforcement learning. SCIENCE CHINA-INFORMATION SCIENCES, 68(6), 2025.
[40] Shengchao Hu, Wanru Zhao, Weixiong Lin, Li Shen, Ya Zhang, and Dacheng Tao. Prompt tuning with diffusion for few-shot pre-trained policy generalization. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, pages 2556-2558 2025.
[41] Sili Huang, Bo Yang, Hechang Chen, Haiyin Piao, Zhixiao Sun, and Yi Chang. Ma-trex: Mutli- agent trajectory-ranked reward extrapolation via inverse reinforcement learning. In International Conference on Knowledge Science, Engineering and Management, pages 3-14. Springer, 2020.
[42] Sili Huang. Yanchao Sun, Jifeng Hu, Siyuan Guo, Hechang Chen, Yi Chang, Lichao Sun, and Bo Yang. Learning generalizable agents via saliency-guided features decorrelation. Advances in Neural Information Processing Systems, 36:39363-39381, 2023.
[43] Sili Huang, Hechang Chen, Haiyin Piao, Zhixiao Sun, Yi Chang, Lichao Sun, and Bo Yang. Boosting weak-to-strong agents in multiagent reinforcement learning via balanced ppo. IEEE Trunsactions on Neural Networks and Learning Systems, 2024.
[44] Sili Huang, Jifeng Hu, Hechang Chen, Lichao Sun, and Bo Yang. In-context decision trans- former: Reinforcement learning via hierarchical chain-of-thought In International Conference on Machine Learning, pages 19871 19871-19885. PMLR, 2024.
[45] Sili Huang, Jifeng Hu, Zhejian Yang, Liwei Yang. Tao Luo, Hechang Chen, Lichao Sun, and Bo Yang. Decision mamba: Reinforcement learning via hybrid selective sequence modeling. Advances in Neurul Information Processing Systems, 37:72688-72709 2024.
[46] Sili Huang, Jifeng Hu, Hechang Chen, Peng Cui, Haiyin Piao, Lichao Sun, and Bo Yang. Gener- alizable causal reinforcement learning for out-of-distribution environments. IEEE Transactions on Industrial Informatics, 2025.
[47] Michael Janner, Qiyang Li, and Sergey Levine Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273- 1286, 2021.
[48] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.098: 09991, 2022.
[49] Zhengyao Jiang, Tianjun Zhang, Michael Janner, Yueying Li, Tim Rocktäschel, Edward Grefen- stette, and Yuandong Tian. Efficient planning in a compact latent action space. arXiv preprint ar08 10291, 2022.
[50] Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[51] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 10211800-2122 2020.
13

---

[52] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances neural information processing systems, 34:21696-21777, 2021.
[53] Nya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning, arXiv preprint TrXiv:2110.06160 2021.
[54] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping entor reduction. Advances in neural information processing systems, 32, 2019.
[55] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1179-1191, 2020.
[56] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[57] Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. In International Conference on Machine Learning, pages 20035-20064.
PMLR, 2023.
[58] Zechu Li, Rickmer Krohn, Tao Chen, Anurag Ajay, Pulkit Agrawal, and Georgia Chalvatzaki. Learning multimodal behaviors from scratch with diffusion policy gradient. arXiv preprint arXiv:2406.0068/ 00681, 2024.
[59] Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdif- fuser: Diffusion models adaptive self-evolving planners. arXiv preprint arXiv:2302.01877. 2023.
[60] Jonathan Light, Yuanzhe Liu, and Ziniu Hu. Dataset distillation for offline reinforcement learning. arXiv preprint arXiv:2407.20299 2024.
[61] Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 289-299, 2023.
[62] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
[63] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095. 2022.
[64] Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning, pages 22825-22855 PMLR, 2023.
[65] Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforeement learning. Advances in Neural Information Processing Systems, 35:1711-1724, 2022.
[66] Chengzhong Ma, Deyu Yang, Tianyu Wu, Zeyang Liu, Houxue Yang, Xingyu Chen, Xuguang Lan, and Nanning Zheng. Improving offline reinforcement learning with in-sample advantage regularization for robot manipulation. IEEE Transactions on Neural Networks and Learning Systems, 2024.
[67] Liyuan Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang, and Amy Zhang. Diffusion-dice: In- sample diffusion guidance for offline reinforcement learning, arXiv preprint arXiv:2407.20109 2024.
[68] Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported value regularization for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
14

---

[69] Volodymyr Mnih. Playing atari with deep reinforcement learning. arXiv:1312.5602, 2013.
arXiv preprint
[70] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprini arXiv:2006.09339, 2020.
[71] Runliang Niu, Jinglong Ji, Yi Chang, and Qi Wang. Screenexplorer: Training a vision-language model for diverse exploration in open gui world. arXiv preprint arXiv:2505. 2025.
[72] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage regressionn Simple and scalable aff-policy reinforcement learning. arXiv preprint arxiv:19100001177, 2019.
[73] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 1eXXi::7709 10087, 2017.
[74] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[75] Litu Rout, Yujia Chen, Abhishek Kumar, Constantine Caramanis, Sanjay Shakkottai, and Wen- Sheng Chu. Beyond first-order tweedie: Solving inverse problems using latent diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9472-9481, 2024.
[76] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06337, 2017.
[77] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances neural information processing systems, 32, 2019.
[78] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 133466, 2020.
[79] Jianlin Su. Diffusion model learning-optimen covariance estimation. Aug 2022. URL https 7/spaces ac. wwwpccc//accciiiaeeeeeeeeeeeeeee 9246
[80] Kerong Wang, Hanye Zhao, Xufang Luo, Kan Ren, Weinan Zhang, and Dongsheng Li. Boot- strapped transformer for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:34748-3470 2022.
[81] Renhao Wang, Kevin Frans, Pieter Abbeel, Sergey Levine, and Alexei A Efros. Prioritized generative replay. arXiv preprint arXiv:2410. /8082, 2024.
[82] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies an expressive policy class for offline reinforcement learning. arXiv preprint (rXXi:2208.06199 2022.
[83] Xiaoyu Wen, Xudong Yu, Rui Yang. Haoyuan Chen, Chenjia Bai, and Zhen Wang. Towards robust offline-to-onling reinforcement learning via uncertainty and smoothness. Journal of Artificial Intelligence Research, 81:481-509, 2024.
[84] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1ttiv::/111.113333 2019.
[85] Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge. Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8941-8951, 2024.
[86] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023.
15

---

[87] Ling Yang, Jingwei Liu, Shenda Hong, Zhilong Zhang, Zhilin Huang, Zheming Cai, Wentao Zhang, and Bin Cui. Improving ciftusion-baseed image synthesis with context prediction. Advances in Neural Information Processing Systems, 36, 2024.
[88] Shentao Yang, Zhendong Wang, Huangjie Zheng, Yihao Feng, and Mingyuan Zhou. A behavior regularized implicit policy for offline reinforcement learning. arXiv preprint 3rXiv:2202.09673, 2022.
[89] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 18:14122-1412, 2020.
[90] Hui Yuan, Kaixuan Huang, Chengzhuo Ni, Minshuo Chen, and Mengdi Wang. Reward-directed conditional diffusion: Provable distribution estimation and reward improvement. Advances in Neural Information Processing Systems, 36, 2024.
[91] Jinouwen Zhang, Rongkun Xue, Yazhe Niu, Yun Chen, Jing Yang, Hongsheng Li, and Yu Liu. Revisiting generative policies: A simpler reinforcement learning algorithmic perspective. arXiv preprint arX1v:2412.01225, 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Appendix B.8
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification:
Please refer to Section 4 Appendix
C
D
E
and
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Section 5.4 and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee 7/nips.com / ssiniioiscyp for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Please refer to Section 5.1 Section 5.2 Appendix [B.5
Section
5.3
Appendix
B.4 and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Appendix B.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://aaarag...caaaaaaa//aacaaa
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
23

---

A Pseudocode of AEPO
Algorithm 1 Analytic Energy-guided Policy Optimization (AEPO).
1: Input: Dataset D,. Max iterations M of training, environmental time limit T, generation steps T, Noise prediction model eg. intermediate energy model &e, value function Va. action-value function Qw. target action-value function Q
2: Output: Well-trained co. Ee, Vg. Q
3: // Training Process
4: for 1 to . do
5: // /'Trann Eg so that we can obtian Va, log ue (arls) [ - ae. t)/o,.
6: Train noise prediction model 6g according to Equation 7
7: // TTain the Q function so that we can obtain Q for any input (s, an) and Q' with autograd.
8:
Train V& and Q with loss Lv and fo (Equation (26
9:
// For each sampled data from the dataset, we calculate the approximate intermediate energy
value with the posterior distribution.
10:
Calculate the target intermediate energy shown in Equation
11:
I/ Use finite data point to fit intermediate energy function (, Ge, t) with neural network.
12: Update the parameter θ according to Equation 25
13: end for
14: // Evaluation Process
15: for i= to T do
16: Receive state from the environment
17: Sample ar from (V,
18: for to 0 do
19:
Calculate Va, log (r((ass) with Equation 13
20:
Obtain V. Es, ar, 1) by performing gradient on dz
21:
V o, log Te (s) Va, log Pe(ay]s) + Va. Ee og t)
Obtain score function Va, log n. (a ()) according to Equation (22
23:
// The implementation follows DPM-solver to realize lower generation steps and reduce time consumption.
24:
Perform action denoising with Equation
6
end for
26: Interact with the environment with generated action do
27: end for
The training and evaluation of AEPO are shown in Algorithm In lines 3-13, we sample data from the dataset and train the noise prediction model eo that is used to obtain V., log M4 (crls) = -co(s, az t) the intermediate energy ( a,, t) that can be used to approximate the intermediate guidance Ez at), and the Q function Q& that is used to calculate (s(s, Qz, t). During the evaluation (lines 14-27), for each state received from the environment, we perform generation with Equation (6. where we use DPM-solver [62] as implementation. After several generation steps, we obtain the generated action ag to interact with the environment.
B Additional Experiments
B.1 Additional Experiments on D4RL Adroit
D4RL Adroit is a hand-like robotic manipulation benchmark, which contains several sparse rewards and nigh-dimensional robotic manipulation tasks where the datasets are collected under three types of situations (-human, -expert, and -cloned) [73]. For example, the Pen is scenario where the agent needs to get rewards by twirling pen. The Relocate scenario requires the agent to pick up a ball and move it to specific location. The experiments of D4RL Adroit are shown in Table 3 where we compare our method with non-diffusion-basee methods highlighted with blue color and diffusion-based methods highlighted with red color. Our method is highlighted in yellow color. The results show that our method surpasses or matches the best performance in of the total 12 subtasks, which illustrates strong competitiveness of our method.
24

---

Table 3: Offline RL algorithms comparison Adroit. We select 4 tasks for evaluation, where each task contains types of difficulty settings. We use red color blue color and yellow color to show diffusion--aan based baselines, non-diffusion-een based baselines, and our method.
Task
peb
hammeer
doer
relocatic
seare
score
Dataset
human
cloned
human
cloned
human
expert
cloned
human
expert
cloned
BC
7.5
69.7
6.6
0.1
57.1
0.1
BEAR
-1.0
26.5
IQL
71.3
37.3
1.4
2.1
1.6
TT
36.4
72.0
11.4
0.3
15.5
0.5
0.1
94.1
-0.1
0.0
10.3
-0.1
20.1
240.9
CQL
37.5
107.0
39.2
86.7
2.1
101.5
0.4
95.0
40.3
483.8
UWAC
119.8
45.
128.8
1.2
10.7
105.4
1.2
108.
49.6
594.7
TAP
76.5
127.4
57.4
L44
127.6
1.2
8.8
101.8
11.7
0.2
105.8
-0.2
$1.9
622.6
59.6
HD-DA
-2.6
107.9
-2.7
0.0
-0.1
-0.2
DiffaserLite
33.2
20.7
2.1
0.1
HDMI
66.2
109.5
48.3
1.2
111.8
1.0
85.9
9.3
0.1
91.3
-0.1
44.3
531.6
D-QL
66.0
112.6
49.3
1.3
114.8
1.1
8.0
93.7
10.6
0.2
95.2
-0.2
46.
$52.6
QGPO
79.0
131.2
60.7
4.6
132.5
4.2
111.9
109.5
$4.6
655.5
AFPO
76.7
147.0
10.3
129.7
6.4
106.5
3.9
107.0
0.5
55.6
667.5
Table The best performance comparison of guidance rescale and non-guidance rescale with grid search.
Type
guidance rescale
non non-guidance rescale
walker2d-me
109.3
109.44.0.
90.8 walker2d-mr
92.8
halfcheetah-mr
43.7
43.3
120
rescale
120
120
mon-rescale
90
90
60
60
60
30
30
30
0
0-0.0
0
0-0.1
120
120
120
90
90
90
60
60
60
30
30
0
00..5
(0
0.8
10..
Figure 3: Guidance rescale ablation of AEPO D4RL. Gym-MuloCo halfchetta-mmediu--eppee task. The y-axis and x-axis denote the normalized score and training steps, respectively.
B.2 Additional Rescaling Ablation
We want to emphasize that the purpose of using guidance rescale is to make our method less sensitive to w. The main contributions of our method are our theoretical results of intermediate energy and systematic comparison results with dozens of baselines on tasks. The results Table show that the best performance achieved via grid search under both guidance rescale and non-guidance rescale settings comparable, indicating that guidance rescale not the primary factor contributing to significant performance gains.
25

---

120
rescale
120
120
non-rescale
90
90
60
30
0.0
0000.0
00
120
120
120
0=0.3
0=5.5
0=0.8
Figure 4: Guidance rescalc ablation of AEPO on D4RL Gym-MuJoCo hopper- sewpeer-mddiu--xppeeee task. The y-axis and x-axis denote the normalized score and training steps, respectively.
120
rescale
120
120
non-rescale
90
90
90
60
60
60
30
30
30
0
0--0.0
0
0-1
0=0.2
120
120
120
90
90
90
60
60
60
30
30
30
093.3
69 095
e 0.8
Figure 5: Guidance rescale ablation AEPO on D4RL Gym-MuJoCo walkeer2d-mediine task. The y-axis and x-axis denote the normalized score and training steps, respectively.
Apart from the guidance rescaling ablation on walker2d--medim-exxee task, also conduct the experiments on wtwicchhetacc-mmiiii----eeee
hopper-medium-exper and walker2d-medium tasks, where the results are shown in Figure 3 Figure E and Figure respectively. From the results, we can see that guidance is important to reach a better performance by comparing w = 0.1 and w = 0, where ( means no intermediate guidance. 'rescale' means apply the guidance rescaling strategy, and 'non-rescale' means we do not use guidance rescaling. With the increase of w, the performance of 'non-rescale" decreases quickly, while 'rescale" still hold the performance, which illustrates that guidance rescaling makes the model robust to the guidance degree w, thus leading to better performance in wider range of w.
B.3 Parameter Sensitivity
investigate influence of hyperparameters 3 and 7 that are used loss LIF (Equation 25 and Lo (Equation (26). From the results shown in Figure 6 we can see that our method shows slight sensitivity value. For the hyperparameter т, we find certain value is friendly for achieving 0.7 (Figure (g)).
good performance, such
26

---

3: hultihress-mmeen
4: khapprreellinn
6p-3
walherle mediun
71happer roplay
Figure 6: Parameter sensitivity D4RL Gym-MuJoCo tasks.
Table The hyperparameters of AEPO.
Hyperparameter
Value
network backbone
MLP
action value function +)
hidden layer
state value function (Va) +b hidden layer neuron
action value function Q
256
hidden layer
state value function (V₂) hidden layer neuron
256
intermediate energy function (Eg) hidden layer
intermediate energy function (Ee) hidden layer neuron
256
noise prediction function (eg) hidden block
noise prediction function (cg) hidden layer neuron
256/512/1024
inverse temperature 3 expectile weight r
0.5
guidance degree
0.1
0.001
a
Refer to Appendix K.2
B.4
Computation
We conduct the experiments on NVIDIA GeForce RTX 3090 GPUs and NVIDIA A10 GPUs, and the CPU type is Intel(R) Xeon(R) Gold 6230 CPU @ 20GGHZ Each of the experiments spanned about 48-72 hours, depending on different tasks.
B.5
Hyperparameters
The hyperparameters used in our method are shown in Table
B.6
Computational Efficiency
We select hopper-m as the test environment, set the batch size to 32, and control the same hardware conditions. The GPU is NVIDIA A10, and the CPU is Intel(R) Xcon(R) Gold 6230 CPU @ 2.10GHz. The results are shown in Table 6 where we report the results of runtime GPU memory usage, inference time consumption of every decision, and training time consumption of every neural network update. The results indicate that our method incurs lower GPU memory overhead compared to QGPO, primarily because does not require loading additional 'fake_ aactonaction' data. terms of training time, our method introduces higher overhead than QGPO, mainly due to the extra computation involved in calculating the intermediate energy of Equation (16). However, in inference time, the cost of AEPO and QGPO is comparable, both adopt the same number of generation steps.
27

---

Table 6: The computational efficiency comparison between AEPO and the SOTA siffusion-based model QGPO. We set the batch size to 32 and conduct experiments of hopper-m under the same hardware conditions. The GPU used NVIDIA A10, and the CPU Intel(R) Xeon(R) Gold 6230
CPU @ 10G
Method
AEPO
QGPO
training time of every updating on hopper-m (s)
0.018
0.015
GPU memory usage with batch size 32 on hopper-m (M)
1135
1161
inference time of every generation on hopper-m (s)
0.068
0.070
Table 7: The influence different training methods on the Q function. The experiments are conducted on AEPO, but with different training methods of Q functions.
Q function training type
IQL-like Q-function training
CQL-like Q-function training
walker2d-me halfcheetah-me
109 200
109.8
94. 2009
93.8
hopper-me
1 111.
111.2 111.2=24
B.7 The Effects of Q-function Training Method
Regardless of which Q-function training method is used, as long as the Q-function learning is suitable for offline RL, the performance obtained using method is similar. For example, with IQL-like Q function training methods and CQL-like Q function training methods, when obtaining accurately estimated Q functions, the performance of AEPO is similar. As shown by the experimental results in Table 7, this experiment used two methods to train the Q function, one being an IQL-like training method and the other using a CQL-like training method. We find that the experimental performance was similar, indicating that we just need to adopt the Q-function training method that is suitable and widely used in offline RI.
B.8 Limitation Discussion
For the limitation of this paper, we summarize the potential areas for improvement as follows.
The generative model requires multiple iterative steps for each decision, which may impact its deployment in environments with strict real-time requirements.
The posterior estimation method proposed in this paper is sampling-based We could also consider higher-order posterior estimation methods in the future, such as the second-order Tweedie method [75] However, this would require a trade-off between more accurate posterior estimation and increased computational cost.
The current method is primarily designed for continuous action spaces, where the use of the first-order Taylor expansion of the Q-function with respect to actions is mathematically justified due to the differentiability of Q(s, a) with respect to the continuous action variable.
C
The Constrained RL Problem
The constrained RL problem is
min BAND ((s,aa s.t. (xx(((((s)l(((1s))
(27)
which can be converted
R ((ss))))((ss.)) (
max E De
(28)
It reveals the optimal solution
<' ((aas)e (((sa))
28

---

where π (a]s) is the optimal policy that can produce optimal decisions by learning from the dataset, ((aas) is the behavior policy that is used to sample the data the dataset, i... the data distribution of the dataset induced from s((a(s), (s,, a) is the action value function, and we usually adopt neural network to learn the action value function.
Proof. The Equation (28) is actually the lagrangian function of the contained RL problem by selecting the lagrangian multiplier 3 and adding the implicit constraint f [Taas)de =1
min En-DuL
min EN [o -3" =Dx((((())ll(((b) + n / m(als)da-1) 19]
1
The functional derivative of L regarding क is
ac
L(T + δT) - ((T)
C(T + δπ))
= (T(als) + δπ (als)) a)da
/ (T(aas) + (r(aas)) (((()))))((a(s)))LLLL))))))))) log x(als) + (T((as)
da (((1s)
(f(n(alo) + b (a(aa)))d -1
+
Notice that the Taylor expansion of log
u(ar)
[(() at u(x) p(x) is
log (xx) u(x)=p(xx) = log p(z)
2xx))
g(x)
log u(x)
g(x)) lu(x)=p(x) p(x:)
Thus,
log ((x) g(x) m((r) (m(a))p(a) p(n) + log u(xx) ) In(s) (u)))) p(x))
u(x)
log u(x)
q(x))
Let u(x) = <(lss) + (als), p(x) = x(als). ((z) = u(als), we have
T(ass) (a(()s))
log <als)) + <<als))
log
dz (a(s)..
p(-(s)
p(aas)
Then, L(T + 8τ) can be simplified
L(T + 8T) =
+ + ( a)da
3" * /
log (als)
m(aas)) (aass) )<(((o) da
+ * (f + +oa(al))dda
1 *
xaass) da + 17 * (ʃ (f((as)da-11
= ((als) a)da
<(cls) log
3
u(aas)
+ / / (Q(s,a)=
1
log T(als)
+1 + r)
B (Tals) (oT(a
3
p(als)
~ [(T)+ + (T((s))
[(ss
1
log p(aas)
(alss)
1
+
3
29

---

Finally, we obtain the functional derivative is
aL
L + 8τ) L(x))
=
ÔT
1
[aass) p(aass)
((,
log
+ 1 + r
3
Let
8C 0%
= 0. we have
x² x'(als) = p(als) * (((()a))+))-
{ n- (als))d ulaass) (00(()))))))1
I u(als) = ( da = p
<'(ss) = p(als) e J [(aas)) (BaQ(saa))a
<a|ss) x p(als) @xQ(=,a)
D Analysis of Exact and Inexact Intermediate Guidance
Previous studies propose that through an ingenious definition (For clarity, we rewrite it below), guarantee the relation (x), qo (xx))e (x,) of p(x)) and q(x;) at any time where Ex(r)) is (=()))--E(()))
general formula of intermediate energy.
Theorem D.1 (Intermediate Energy Guidance). Suppose po (zo) and go(zo) has the relation of Equation 2 For E (O, let (zjo(2e and (0,0(2 be defined by
Pqo(aee (mm) == (00(2 = Naa ou30, r)
So the marginal distribution pe (r()) and ge (2x)) at time are pe(xt) (a (o)))po(3 dan and (x()))) (ooo(a/loo)g (oo))o(z)))aaa) (zo)dzo. Define a general representation of intermediate energy as
|8 (xu),
t= 0 (BB--xo))]
E((12)
log (x0lx( -E(zo)), t> 0
(29)
Then (2x) and pe( satisfy
(x())) & gc(zz)e--(ee
(30)
for any diffusion step and the score functions of (e(xe) and pr(st) satisfy Va, log)((x:)
Va, log (x1) Ee
The derivation can be found Appendix
While for inexact intermediate energy
SMSE (ara) E B (wol ( (E())) 0, &
Ep (xr) = & E (x) [zo])), 0,
that adopted in classifier classifier-guided and classifier-free-guine methods, it can be derivated that (x,) 2 EMSE (xx) when B = and >
Proof. By applying Jensen's Inequality, is easy to have
log Egg i (moo (8(mm)) > goo [lg (log e -00(2x)) (eol))))))) (8(o)))
When 3 = 1, we have Ex(x)) > EMEE((r))
30

---

Also, we can prove that these three intermediate guidance values are not equal to each other: (x,4(()) * Ve E (z,) * V., (x,)))
Proof. For t > 0,
V (x,4)(04) V., log P (E(
Ego (t V #1 / 20)c(2ol BE(x0) draa
1
= ex /
9011(20/12) Vx, log gaje me)e (E( dxo:
--ooo](z01er) [cll(e)) Vz, log go (au [mme]
where the chain rule of derivation and gradient trick to obtain the result. when / 1, we have E((e)) -B (mallre) (e(x(1))-E(xx) V., log 9ote (z)] Similarly, we can obtain the gradient
of E (.MEB(r)) and EPPS (3re::
V & (MSE(()))
= Va, E (ool)))))) (E(xo))
Vn, / (100
Zooo (x)l(()) V log Yo ( (
(Folua)) [E())) Va, log 2o (x))
V cDPS 2: c
(xt) = Vz, (V) Coopseet
VnE(H) Vx, LE yon (zol=.) [zo],
VyE(v) (ao Iro Va, log go (2ol
(xooe(2alle)) (Vy((y) V... log 20f (x)] From the results we can see that Ee,E(( + V E.,
EMSE (rr) + V EPPS (z+).
E Guidance of Intermediate Diffusion Steps
In order to guide the diffusion model during the intermediate diffusion steps, we first consider the following problem
min (x
(31)
s.t. (Po(Po(zO) ((o((o))
and the relation function
Po(zu) x qu(xu)e - (8(mo)
(32)
Define
p = [ xo) N (z (aria oi
Pa (xx)) Pa P=jo(xs(=o)po(a))) drro,
q: (x.)) = / (10 (zo)
(x.) (E(x)))
= 0
BE(2ol] log (xor (e
t> 0
Then, will obtain the following relation between function of (xx) and the score function of of (a()))
V log (xx)) V (xx)) E,
(33)
for each intermediate diffusion step.
31

---

Proof. Known that
(( - ()jo(1i/20)) = (2s aanto, r
(e(xz)) = / 4: (20 ddoo
pe(at)= / poo(
po(zo) ( -B()a)) (8(w))
By integral
7 = / vo( -E(=0) dro Byo((o)
-E((z))
and
(00) (0(xx))= (BE()))
know
po(xo) = (0(00)e--00aa)
Z
Thus,
pe(a,) /
/ Prjo(ar(10))
20 (2 -2((a))
dro
Z
/ (up((1)/20)
[0(xx)
)E(wa)
dor
2
-E(0))
90 (0) =x((z))
dro
(xe -005(x0)
Z
pe(as) x (r((teee- -E((o))
where we replace 4s o (2; (20) with ge (x) by using Bayes Law the fourth equation. Now, if want generate data from p(ro), we just need train diffusion model ((x))) and guidance Eg, then use
(x())) x (xx))e (ap(xr)e-diie E((xe)
(34)
V., log pr( Vz, log (x(x)) - Ex
to obtain the score function value of Va, log (xre). Then, we can use the reverse transformation (Equation (6)) to generate samples.
In RL, should Q(s, a) to denote Exro), because Equation ① and Equation 3 indicate that maximizing Qs,, a) is same with minimizing Eo (x0). Besides, we notice that approximating & (x,) needs calculate Exro). And actually E(xo) is the Q a). In other words, the intermediate energy guidance in RL is defined by
(B()))) - (B((s
t=0
(x1)
(35)
log E (1ro (xnola:)) (8E + log E (woloe)) (((e,)) 1>
32

---

Proof.
Te Te(arls)=
Tejo Tajo(araap s)to(ao|s)dar
=
/ (no 8) Mo s) (e (Q(0,0))
dao
Z
/ Teolaalo, 8)
Lo(ao|s s)e
(s(ra))
dao
Z
- -mo(aels) / Holaulas, 8)
dao
Z
me(ar]s) (panaolaaa ((
Z
pe(arl))eErta))
Z *<(aels) pa(aels)e
Here, we slightly abuse the notion of because, RL, the input the energy function is the concatenation of state and action.
F
Discussion of Previous Intermediate Energy Estimation Methods
Here discuss the differences between method (AEPO) and QGPO 64 and Diffusion- DICE [67] from several perspectives.
From a theoretical perspective: Our method provides theoretical solution for the log expectation of the intermediate energy, whereas QGPO and Iiffusion-DICE do not discuss further results for the log expectation term. In Lemma reformulates the estimation of the log expectation term of the intermediate energy convex optimization problem, which is different from our approach: we aim derive further analytical results for the log expectation term.
the Diffusion-Diie -DCCE paper,
From the action generation perspective: When generating actions, Diffusion-DICE. requires generating multiple candidate actions (Algorithm 1, line 10 Diffusion-DICE. whereas our method, like QGPO, does not require generating multiple candidates.
For additional assumptions: In Theorem 3.2 of QGPO where the authors use contrastive learning loss to fit the intermediate energy, the authors mention that, under infinite data and model capacity, the contrastive loss (Equation (12) in QGPO) can perfectly approximate the intermediate energy. However, in practice, we cannot achieve infinite model capacity, and offline samples certainly limited. Our method does not require assumptions about the sample size or model capacity.
For the dataset: fake action vectors for CEP, and these fake action vectors may introduce the influence of out-of-distribution actions. However, our method does not require the generation of fake actions.
Furthermore, QGPO requires the use of a diffusion model to generate
For training stability: Diffusion-DICE formulates an optimization problem as min ([f(z)e + y). where f(x) represents the Q function in RL, is usually positive. This results in adversarial training dynamic: the term min encourages larger values of y. while min pushes the output y lower. However, when > O, the gradient of e dominates that y. which can drive the model continuously increase its output prediction, eventually leading to unstable training. This concern is also raised by readers in the GitHub issues of Diffusion-DICE. Fortunately, this issue does not arise in our method (AEPO) and QGPO. Our method removes the log and exp operations through simplification and approximation, and QGPO stabilizes training by normalizing weights during each CEP update.
33

---

Table 8: The intermediate energies comparison between QGPO and AEPO. We select 'hopper- medium-v2' as the evaluation task, where the reference values are calculated with the analytic equation results (Equation (24). The estimated intermediate values of AEPO and QGPO are calculated with the learned neural network &g and the Sa trained with CEP, respectively.
method
AEPO
QGPO
mean error
48.8
53.2
performance
87.4
86.6
Experiments of the estimated intermediate energies: The quality of the intermediate energy estimation is directly related to the model's performance. During the action generation process, if the estimation of the intermediate energy is biased, the generated actions are unlikely to be good, leading inferior evaluation performance. We report the mean error between the estimated intermediate energies from AEPO and QGPO and the calculated value from Equation (16). The results shown in Table 8
G Discussion of the Q-function Smoothness and Locality
In Equation 14
we use the Taylor expansion convert the implicit dependence on the action in the exponential term to explicit dependence. Below we provide the necessary conditions analysis of Q-function smoothness and the locality when applying the Taylor expansion on Q-function.
For smoothness, we often use neural networks to learn the Q function in practical implementation, and by using specific activation functions, such SiLU (the activation function used in our source code), we can ensure that the Q function is smooth with respect to actions and at least twice differen- tiable. This is because: If the activation function <(-) is UU, then <6-)) is infinitely differentiable with respect to the input. Assuming the neural network is an L-layer feedforward network, i.e., then 1) affine transformations
Q(a) W (Wro@(WW-....... tion functions are infinitely differentiable, so the composite function ((a) with respect to is also infinitely differentiable, and at least twice differentiable. Even in cases where activation functions like ReLU are used, the Q-function remains piecewise linear and locally approximable by linear functions in regions away from differentiablle points.
infinitely differentiable, 2) activa-
For local linearity, since the Hessian of Q(a) with respect to a exists and is continuous, the closed ball B, (ā) with center and radius p is compact set, and by the Dollano-WWeiesiie theorem 200, the Hessian norm must have a maximum value on this compact set, denoted as L Therefore, the linearization error in the Taylor expansion of (a) with respect is bounded quadratically, satisfying the conditions for local linearity.
H Detailed Derivation of Intermediate Guidance
Implicit dependence of (ss, oo) on action hinds the exact calculation of log (poos(aolas,s) (( In order to approximate the intermediate energy Eg a;) shown in Equation [II. we first expand ((s, ag) at ag ā with Taylor expansion. i.e..
aQ(s, ao) la0 * (an - -),
Q(8, ao) R Q(s, uo)la
Dag
where is constant vector. Replacing it to log Eno~p( s)..)) s) (0 (((no))) we transfer the implicit dependence of Q(s, ao) on ag to explicit dependence on Q' (s( ag. Accordingly,
34

---

log s)) e is given by
log aa (Q(,0))
s log Eaomp(aolan. (((sav))ao=aa
AQ(eaa) 800
ag= -000
= log Ma
log Euomp(aalan.s) (.Q((s,)) e. (=
=log { ( SaQ + log bao (e
=8 (( 3 Q a) + log
Bao (s) je je (' (s,0)aao)
Furthermore, by applying the moment generating function of Gaussian distribution, have
log (a :Q(x.a))
=f+ B Q'(s, a) =a+ log le (aa))aa
= = - 3 ('s a) + 3 * Q'(s,ã) a) * Poje +
2 * 8 o0 11Q' (s,
= Q + B (s(saa))))))))) a)_ (Poje - a) + 2 * 32 T 11 (s, a)l1
I Detailed Derivation of Posterior Distribution
L.1 Posterior
If we want get the exact mean of (oje(ao (t,, s), we actually want to optimize the following objective
(ap la,, s) min E (apolaa [lloope [ol])]]
(36)
0
where Poje, (ao la, 8) can be further reparameterized by
1 (a: orea(s,,,,
Dojt.0
(37)
because this reparameterization follows the same loss function (as shown in Equation 7 as the diffusion model, and do need introduce additional parameters approximate the mean of (ajo(al s). For the covariance matrix Σojz, following the definition of covariance
Eoje (az) (as)
16 pa s) [(ao go <) ;oo+e
(at - o
E (ogolo))
(av
(a: (aco))(ao
a
(ao ai)(a0 _a))
1
o
(38)
(aolaa
? cgcp
a₁
1 E (aglo [(a: - aa )nau)
of
coc
of
af
where in the second equation replace Paje with Equation
37
In the third equation, we use
E (poo
0+)
col
04
a
1
= (Epore(aola..s)
[0o]
at
EB
ai
at
σt
= =Mojt
aa) #
= of
cocD)
T
aj
35

---

and Poy: = E paje(aal or. leol. Besides, we also notice that (lae) ~ (a; ayao, ofI), thus we have
Bue(arlst Hor
[(a a4ao) (,00) ']
Es(ools) B (aa (a: acao)(at - o240)
-pllols)) pappolaclo [[a+ creaa)(ar - (raa))
Es((ools) Ezjo
=
where in the first equation μs va (arls)) independent from the data dz, we can apply the expectation over Eoje (a)):
= p(aoos))rr((a au, s). To make the variance
Euje (arls))) (a))((a))
ot -
Cecllslllo ()e 1) 1)-
of
For simplicity, we usually consider isotropic Gaussian distribution, ie.. Do Ti which indicates that
o = oº [1 at pollllllll
1
1.2 Posterior
For the mean value of the distribution (oje(an at, 8), we still adopt the same reparameterization trick Pojr, 0, (a= osco(s, az, t)). But for the covariance, we can reformulate [79 it as
Eap
Eyoo (ao (aolae.))
[(ao (ojf)) (au [
((ao uo) - (No un)) ((ao un) - (Doje un)) "]
E (aolaaa
(39)
E (aopaaa)
[aa uo) (ao uo) (Po wo) uo)
where add constant vector up that has the same dimension with Äoje in the second equation. We use Poje = (aola to derivate the third equation. Perform expectation on Eaj((a)) and let
24 = Ear~z((n) [oo], we have
oe = Emilar (s) [a)e(
Epelas (s) B Mine (apla s3
[100 ua) un) (arle)))) (Moo uo) m40) || [aao un) uo) T Epstar (Poje - uo ( - uo) (40)
Eptagls) 1s) (osp(arloo:s)
Bu(ools) [1la0 - uol - Bm [ll uo Holl2].
Var(ag) ( vol121,
where Var(an) can be approximated from a batch of data or from the entire dataset and Ep.la,bb)lllop uol [ can be estimated by sampling a batch of data ag and a, with different
1.3 Another Method to Approximate Posterior
Majs
Consider the covariance estimation under Mojo * E (ag (aglas (), where Maje has been trained well using the diffusion loss function (shown in Equation (7), we can adopt the following negative log-likelihood optimization problem [5]
min E, (aaeeii(aaaa (al log P(ao: Po Eoj(
36

---

perform derivation w.r.t Eoje, we have
log P(ao; Pojes Eoje),
0
1
(00 -oo)) A Egyl (ao -Mole)
log
2n [
0
d
2x + log Eojel 2 (00 iaope) Enj mole)
1
1
log
2
2
5--1 0/t
5 - pa) - mojee) T Stole
So the objective reaches maximization when Soje = Eernuelar (ojooaaaa)))) [[ao - moje)(an - Aoooe]] Usually, we consider the isotropic Gaussian distribution, i.e. Σojz = 001 and
00 d
( [Ilag öo 112).
(41)
Furthermore, we also have the following result, which is the same as Equation (41
oif
Eag Bog-u((alss) (a.. (1la0 Poje 1l)).
1
|l00 - (a. o,ea(s, a, at
(42)
o 11w [||e ea ar [2] 07d
Inspired by the previous studies that Poje po (agla..e) [no] is suitable for training in practice, so we only focus on the performance of Posterior and Posterior this paper. We summarize the posterior mean and covariance in
Pope=
(a: - oteo(8, at, or, ()),
of 1
2 (Bm(aal [[eep(a, e)]
Off at
(43)
60 Var(ao) d (Bul(a))ll uol u0l121.
82 of afd
Foo. [[|c-1ol co(s, az, t)
The Training of Q Function
J.11 Q-function Training with Expectile Regression
Expectile regression loss
Lv E(s,a)D,, [5(Va(s) - Q=(s,)))
La = B(s,a,D~~,, [lr(ss,) +VV) - (s.
(y(v) 7 1(y < 0)ly2
provides a method to avoid out-of-sample actions entirely and simultaneously perform maximization over the actions implicitly. The target value of Q is induced from parameterized state value function, which trained by expectile regression on in-sample actions. Besides, effectiveness has been verified by many recent studies [53] 25). Thus, we adopt this method to train the Q function.
37

---

J.2 Q Function Training with In-support Softmax Q-Learning
Due to the natural property of data augmentation in generative models, such diffusion models, we can use diffusion models to generate fake samples 64] and augment the training of the Q function:
LisQr )) [lQQ -TQu,,)))
T*Qu(s, = r(s, + Ex ( (
En @
where ä' is the fake actions. This method needs additional samples train the Q function, where can not make sure all the fake actions are reasonable for certain states. To certain extent, these fake actions OOD actions for the value functions. Besides, additional actions will also cause large memory consumption.
J.3 Q Function Training with Contrastive Q-Learning
Contrastive Q-learning [55] is another notable method for training the function from offline datasets.
It underestimates the action values for all actions that do not exist in the dataset to reduce the influence of OOD actions. The training loss is
Loor = Es [IQu(s, (r x(dd
+ (E 1q a)] Bls.a)
The learned function tends to predict similar values for actions that do not exist in datasets, which will cause ineffective intermediate guidance and further affect the performance.
K
Implementation Details
K.1
Guidance Rescale Strategy
The guidance scale affects the amplitude and direction of the generation. The direction of generation must be changed due to the gradient guidance of the critic. So we can We find that when the guidance scale is zero, the inference performance is more stable than the guidance scale is non-zero, which inspires us to modify the amplitude the score of the optimal data distribution Va, log n. (a,a)s)) Mathematically, we re-normalize the amplitude of Va, log (a(aa))) by
Va, log ne (acls)
Va, log (a((ss))
|| log pe(a, |-]|
(44)
Vo, log (a(
K.2 The Choice and un
Although choose any vector, considering the error bound of Taylor expansion and the stability of the training process, we use the following method to obtain ā
(8. 11 (s. ao)il'
(45)
where v = 0.001. Considering the loss term of second-order approximation
R2(a) v ((s
where e is the point between [ā, a). Considering the definition of Equation 45
IVv ||e all
a)ll
(s, ao) 11 (s, ao)l
Vv.(( a)ll
=V.
38

---

So the error satisfies
(a = - (V.
a)}² 2 ((V.Q(s,)) (a a))² - v
as long as à close to a, i.e., the derivative between á and a is approximately constant. In practice, use Q((s,a) a) as the value of (', a) for two reasons: (1) Small v value leads approximately same derivative value at and a. (2) Directly use Q'(s, a) will reduce one calculation of gradient for Q'(s@ā).
Pyga.>lllep(ee [Ilep(a, e)] by sampling batch of data and the calculation process is (1) Sample batch of data ag from the dataset and sample from Gaussian distribution (0,, I). (2) Sample time step index t. (3) Obtain O₂ = agdo + dgC. (4) Get the noise prediction norm with diffusion model cn t, s). (5) Average on the batch of data.
Equation (43), we approximate Egja.)
In Equation (43).
No serves as a constant baseline to calculate oz. where Extar) [loooj uo | can be
approximated from the data. (1) We can sample batch size of data from the dataset and sample
from Gaussian distribution N (0, I). (2) Sample time step index t. (3) Obtain G = 0,00 + drE. (4)
Use the diffusion model to predict Moje (a, (a,, s)) (5) Calculate Ep(aa)) and average on the batch data.
39


Reviews:
[{"review_id": "iNdJ982x88", "reviewer": "Reviewer_kfG8", "summary": "In this paper, the authors study the problem of offline RL with diffusion-model-based algorithms. It falls in the category of learning a diffusion model for the *behaviour policy*, using an energy function based on the Q-function to guide the generation. The paper identifies that the theoretically optimal guidance signal for this task, termed the *intermediate energy*, is defined by an intractable log-expectation formulation. Consequently, prior methods have relied on theoretically inexact heuristics or potentially unstable training procedures to approximate this guidance, creating a gap between theory and practice.\n\nThe proposed algorithm, Analytic Energy-guided Policy Optimization (AEPO), is novel with tractable analytic approximation for this intermediate energy. The authors achieve this by linearizing the Q-function with a first-order Taylor expansion and then leveraging the closed-form moment-generating function of a Gaussian distribution to resolve the expectation. Experiments have been conducted to validate the idea.", "questions": "1. How is the locality and smoothness of the Q-function (Equation 14) validated, both theoretically and experimentally?\n2. Can the proposed AEPO be naturally extended to a discrete action space?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "### Strengths\n\n* This paper studies a highly significant and fundamental problem in diffusion-based reinforcement learning: the intractability of the theoretically optimal guidance signal.\n* They propose a principled and effective solution. The core contribution—deriving an analytic approximation for the intermediate energy—is highly original and technically sound.\n* The use of a Taylor expansion combined with the moment-generating function of a Gaussian is a clever and novel approach that elegantly transforms an intractable problem into a tractable one. It is a distinct and well-reasoned improvement over previous heuristics (like MSE/DPS) and more complex training schemes (like contrastive learning), and its derivation is clear and well-supported.\n*  The paper is very clearly written and well-organized.\n\n### Weaknesses\n\nThe paper presents several areas for improvement.\n\nFirst, the first-order Taylor expansion of the Q-function in Equation (14) assumes local linearity. The authors should clarify whether the necessary conditions of Q-function smoothness and the locality of $\\bar{a}$ around $a_0$ are met.\n\nSecond, the method's reliance on Gaussian mechanics restricts its applicability to continuous action spaces. This significant limitation is not adequately discussed in the main paper. While it may perform well in continuous spaces, its effectiveness in discrete spaces remains unclear.\n\nThird, regarding experiments, while the results are generally strong, further analysis and experiments on computational efficiency are needed. Given the potentially long training times, more profiling would help identify areas for improvement.\n\nFinally, the evaluation lacks diversity, focusing on a specific experimental setting rather than including real-world robotic tasks or domains with different physical properties. This narrow scope limits the assessment of the method's broader generalization and robustness, thereby reducing the stated impact of the results.", "was_revised": true, "final_justification": "All concerns have been addressed during discussion and no change on reviews.", "rebuttal": "Dear Reviewer kfG8,\n\nWe appreciate your suggestions and respond to your concerns below.\n\n### [I]. Explanation of Weaknesses\n\n> **[1/4] W1:** First, the first-order Taylor expansion of the Q-function in Equation (14) assumes local linearity. The authors should clarify whether the necessary conditions of Q-function smoothness and the locality of $\\bar{a}$ around $a_0$ are met.\n\nThanks for the rigorous comments.\n**For smoothness:** We want to clarify that we often use neural networks to learn the Q function in practical implementation, and by using specific activation functions, such as SiLU (the activation function used in our source code), we can ensure that the Q function is smooth with respect to actions and at least twice differentiable. This is because: If the activation function $\\phi(\\cdot)$ is SiLU, then $\\phi(\\cdot)$ is infinitely differentiable with respect to the input. Assuming the neural network is an L-layer feedforward network, i.e., $Q(a)=W_L\\phi(W_{L-1}\\phi(…..))$, then 1) affine transformations are infinitely differentiable, 2) activation functions are infinitely differentiable, so the composite function $Q(a)$ with respect to $a$ is also infinitely differentiable, and at least twice differentiable. Even in cases where activation functions like ReLU are used, the Q-function remains piecewise linear and locally approximable by linear functions in regions away from non-differentiable points.\n\n**For local linearity:** Since the Hessian of Q(a) with respect to a exists and is continuous, the closed ball $B_\\rho(\\bar{a})$ with center $\\bar{a}$ and radius $\\rho$ is a compact set, and by the Bolzano–Weierstrass theorem [1], the Hessian norm must have a maximum value on this compact set, denoted as $L_H$. Therefore, the linearization error in the Taylor expansion of $Q(a)$ with respect to $a$ is bounded quadratically, satisfying the conditions for local linearity.\n\nWe will add the above discussion in the revised version.\n\n\n> **[2/4] W2:** Second, the method's reliance on Gaussian mechanics restricts its applicability to continuous action spaces. This significant limitation is not adequately discussed in the main paper. While it may perform well in continuous spaces, its effectiveness in discrete spaces remains unclear.\n\nThanks for your insightful comments. We agree that the current method is primarily designed for continuous action spaces, where the use of the first-order Taylor expansion of the Q-function with respect to actions is mathematically justified due to the differentiability of Q(s, a) with respect to the continuous action variable.\n\nIn discrete action spaces, however, there are several key challenges:\n- The use of diffusion models in discrete domains remains an open and actively studied research area. Modeling and sampling in discrete diffusion processes (e.g., categorical diffusion) are non-trivial and require additional mechanisms;\n- More fundamentally, the Q-function is not differentiable with respect to discrete actions, making the Taylor expansion mathematically difficult to interpret.\n\nWe believe that it is an interesting and important future direction to extend our approach to discrete domains. We will explicitly acknowledge and discuss this limitation in the revised version and clarify that our current method is intended for environments with continuous action spaces.\n\n> **[3/4] W3:** Third, regarding experiments, while the results are generally strong, further analysis and experiments on computational efficiency are needed. Given the potentially long training times, more profiling would help identify areas for improvement.\n\nWe select hopper-m as the test environment, set the batch size to 32, and control the same hardware conditions. The GPU is NVIDIA A10, and the CPU is Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz. \nThe results are shown in Table 1, where we report the results of runtime GPU memory usage, inference time consumption of every decision, and training time consumption of every neural network update. The results indicate that our method incurs lower GPU memory overhead compared to QGPO, primarily because it does not require loading additional 'fake_actions' data.\nIn terms of training time, our method introduces higher overhead than QGPO, mainly due to the extra computation involved in calculating the intermediate energy of Equation (16). However, in inference time, the cost of AEPO and QGPO is comparable, as both adopt the same number of generation steps.\n\n\n**Table 1:** The computational efficiency comparison between AEPO and the SOTA diffusion-based model QGPO. We set the batch size to 32 and conduct experiments of hopper-m under the same hardware conditions. The GPU used is NVIDIA A10, and the CPU is Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz.\n|Method|AEPO|QGPO|\n|-|-|-|\n|training time of every updating on hopper-m (s)|0.018|0.015|\n|GPU memory usage with batch size 32 on hopper-m (M)|1135|1161|\n|inference time of every generation on hopper-m (s)|0.068|0.070|\n\n> **[4/4] W4:** Finally, the evaluation lacks diversity, focusing on a specific experimental setting rather than including real-world robotic tasks or domains with different physical properties. This narrow scope limits the assessment of the method's broader generalization and robustness, thereby reducing the stated impact of the results.\n\nThank you for your thoughtful and constructive feedback. We agree that the current evaluation is limited to virtual robotic environments, and thus may not fully reflect the method's potential generalization to real-world robotic systems or environments with diverse dynamics. Our primary motivation was to first validate the effectiveness of our method in standardized and widely-used simulation environments, which is beneficial to compare our method with previous representative methods.\nExtending the method to real-world robots or environments with significantly different physical properties is indeed an important and meaningful direction. However, we acknowledge that such an extension may involve several non-trivial challenges, including but not limited to:\n- Hardware and resource constraints: Real-world experiments require access to diverse robotic hardware, which can be costly to acquire and maintain. Additionally, the time, personnel, and safety considerations involved in such deployments often pose significant barriers.\n- Safety and deployability: Sampling-based or diffusion-driven policy outputs need to be carefully constrained to ensure stable and responsive behavior in real-world deployments.\n- Sim-to-real transferring: Offline datasets collected in one environment may not generalize well to novel states or transitions in real scenarios;\n\nWe are happy to explore these directions in future work and evaluate our method under more diverse and challenging physical conditions, including real-robot settings.\n\n\n### [II]. Explanation of Questions\n\n> **[1/2] Q1:** How is the locality and smoothness of the Q-function (Equation 14) validated, both theoretically and experimentally?\n\nWe appreciate the reviewer’s insightful review. We would like to clarify that the locality and smoothness of the Q-function are theoretical conditions required to justify the Taylor expansion used in the derivation of the intermediate energy. These are not properties that can be directly validated or falsified through empirical experiments due to the following reasons:\n- Locality and smoothness are inherently infinitesimal properties. In contrast, experiments are based on finite samples and cannot assess functional behavior in an arbitrarily small neighborhood.\n- Numerical estimation of gradients or higher-order derivatives in high-dimensional action spaces is noisy, unstable, and lacks reproducibility across different runs or data subsets.\n\nFrom a practical standpoint, the smoothness of the Q-function is an inductive bias implicitly introduced through the architecture of the neural network. In practice, the Q-function is parameterized by a deep neural network with smooth activation functions, such as SiLU, which ensures differentiability and enables local linear approximations theoretically (Kindly refer to W1 for the theoretical analysis).\n\n> **[2/2] Q2:** Can the proposed AEPO be naturally extended to a discrete action space?\n\nIf we want to apply our method to discrete action spaces, we believe the following research steps should be followed: \n1) First, the guidance formula in equation (8) has a universal formula and can be used for both continuous action spaces and discrete action spaces. \n2) When using diffusion models in disperse action spaces, the forward process is no longer in the continuous form as in equation (5), but instead uses random replacement to substitute the input x.\n3) In this case, the intermediate energy formula in RL will no longer need to transform the log-expectation into a simplified form through Taylor expansion of the Q function and moment generating function, but instead approximates the log-expectation through sampling and averaging, because actions are discrete, and integration is directly replaced by summation. \n4) Use neural networks to fit the simplified intermediate energy, just as in equation (25). \n5) Implement guided generation using the intermediate energy fitted by neural networks.\n\n\n**References**\n\n[1] Principles of mathematical analysis."}, {"review_id": "YCYZjufngf", "reviewer": "Reviewer_z6K5", "summary": "This paper proposes Analytic Energy-guided Policy Optimization (AEPO), a method designed to approximate the intermediate energy used in conditional decision generation with diffusion models. It first analyzes the limitations of existing methods that provide only inexact intermediate energy estimates, and then introduces a novel approximation technique based on Taylor expansions. Subsequently, the model approximates the Gaussian mean and covariance of the posterior distribution.\nAEPO is evaluated on the D4RL offline reinforcement learning benchmark and shows promising performance compared to 30 baseline approaches.", "questions": "Except for D4RL, are there any other benchmarks that can be used to evaluate the current approach? Is the OG-Bench (ICLR25), a goal-conditioned offline RL benchmark, a reasonable choice for the current approach?", "limitations": "yes, a limitation section has been provided in the appendix.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- The paper presents comprehensive mathematical derivations, including key theorems, proofs, and approximations. I reviewed most of the proofs and found them technically solid and convincing, demonstrating a high level of quality.\n- The experimental section is thorough, with comparisons against more than 30 baseline approaches on the D4RL benchmark, highlighting the competitiveness of the proposed method.\n- A reasonable ablation study and analysis are provided to support the understanding of individual components.\n\n# Weakness\n- No major weaknesses are identified at this stage, acknowledging the reviewer's limited expertise in energy-based methods and offline RL. However, I remain open to further discussion with other reviewers regarding potential limitations of the paper during the rebuttal phase.", "was_revised": true, "final_justification": "I recommend this paper due to its overall quality, I remain my 5 score as it is.", "rebuttal": "Dear Reviewer z6K5,\n\nWe appreciate your suggestions and respond to your concerns below.\n\n\n### [I]. Explanation of Questions\n\n> **[1/1] Q1:** Except for D4RL, are there any other benchmarks that can be used to evaluate the current approach? Is the OG-Bench (ICLR25), a goal-conditioned offline RL benchmark, a reasonable choice for the current approach?\n\nWe select D4RL to evaluate our method because it is widely adopted as a general-purpose benchmark for offline RL and widely used in subsequent literature [1]. D4RL encompasses a diverse suite of tasks across domains (e.g., locomotion, manipulation, maze navigation) and contains varying data quality levels (e.g., random, medium, expert, medium-replay), making it well-suited to examine the effectiveness of offline RL algorithms.\n\nWe are aware that OG-Bench is a recently proposed benchmark specifically designed for goal-conditioned offline RL, which is an important and emerging subdomain of offline RL [2]. OG-Bench is more specialized and does not provide a general-purpose evaluation suite like D4RL. We evaluate our method on 33 tasks across different D4RL domains, covering a broad range of difficulties and data distributions. We believe this provides a strong empirical basis for evaluating the effectiveness of our approach.\n\n**References**\n\n[1] D4rl: Datasets for deep data-driven reinforcement learning.\n\n[2] Ogbench: Benchmarking offline goal-conditioned rl."}, {"review_id": "zQAbEwPx9z", "reviewer": "Reviewer_THAc", "summary": "This paper proposes an posterior estimation method for the exact energy guidance function if Diffusion RL. There are three key estimation employed in this paper. 1) Consider Q function as a linear function within a certain area. 2) Consider diffusion posterior \\mu(\\x_0| \\x_t) as a Gaussian distribution. With these two estimations, the method can effective compute exact energy function during training, and apply energy guidance in Offline RL tasks such as D4RL.", "questions": "1. In Eq 3, Do you mean s.t. KL(\\pi|\\mu) <=  \\eta?   It seems  <= xxx is missing.\n2. Why cannot you analytically compute the gradient of energy function from Eq. 17.\n3. Besides performance, what is the advantage of AEPO compared with previous work , especially QGPO and DPS.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The paper poses a neat and in-depth analysis of existing energy guidance method. The proposed method is well theoretical-grounded. The posterior estimation method contain good mathematical insight. \n2. The experiments are extensive and supportive. This work compares with a series of related baselines to prove the effectiveness of the proposed method.\n\nWeaknesses:\n1) I think there's still room for improvement on the paper writing. E.g., What is the core motivation for proposing AEPO, what problem of QGPO(CEP) is AEPO trying to solve? This one should be highlighted in Introduction and Method Section.  Also, what is the key difference between DPS and AEPO (I think they both some kind of posterior estimation). Finally I would suggest keeping only \\pi, \\mu notation or p, q notation in introduction. Keeping both is not really necessary.\n2) Regarding (1).  I think usually the advantage of Analytic xxx Algorithm is that you can compute some quantity so that you do not need to estimate them using a network. However, AEPO seems also need to train an energy guidance model (Eq. 25), which somehow weakens its advantage.\n3) Line 177:   \"Reminding that the posterior distribution µ0|t(a0|at, s) is also a Gaussian distribution\". This is wrong.  $µ0|t(a0|at, s)$ is not Gaussian, $µ0|t(am|at, s)$ can be considered as Gaussian only if m and t are close.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer THAc,\n\nWe appreciate your suggestions and respond to your concerns below.\n\n### [I]. Explanation of Weaknesses\n\n> **[1.1/3] W1.1:** What is the core motivation for proposing AEPO, what problem of QGPO(CEP) is AEPO trying to solve? This one should be highlighted in Introduction and Method Section.\n\nThank you for your suggestions. \n\nThe core motivation of our method is that QGPO cannot solve the intractable intermediate energy with the log-expectation form, which leads to sub-optimal performance. Facing this issue, we resolve the intractable log-expectation of intermediate energy with Taylor expansion and the moment generating function, achieving better performance in 33 tasks with various domains.\n\nWe discuss the differences between our method and QGPO in detail, and we will add the contents of core motivation in the Introduction and Method.\n- **From the perspective of theory,** our method provides a theoretical solution for the log-expectation of the intermediate energy, which takes a further step compared with QGPO. QGPO only derives the formula of intermediate energy rather than the further results of the log-expectation.\n- **From the perspective of algorithmic design,** in Theorem 3.2 of QGPO, where the authors use contrastive learning loss to fit the intermediate energy, the authors mention that, under infinite data and model capacity, the contrastive loss (Equation (12) in QGPO) can perfectly approximate the intermediate energy. However, in practice, we cannot achieve infinite model capacity, and offline samples are certainly limited. Our method does not require assumptions about the sample size or model capacity.\n- **From the perspective of dataset,** QGPO requires the use of a diffusion model to generate fake action vectors for CEP, and these fake action vectors may introduce the influence of out-of-distribution actions. However, our method does not require the generation of fake actions.\n\n> **[1.2/3] W1.2:** What is the key difference between DPS and AEPO (I think they both some kind of posterior estimation).\n\nThe key difference between DPS and AEPO is: The intermediate energy formula used in AEPO is exact while that in DPS is inexact. The derivation of the exact and inexact intermediate energy can be found in Table 1 and Appendix E of QGPO. AEPO first bases on the exact intermediate energy, then solves the log-expectation that QGPO does not resolve.\n\n> **[1.3/3] W1.3:** Finally I would suggest keeping only \\pi, \\mu notation or p, q notation in introduction. Keeping both is not really necessary.\n\nSimultaneously introducing $\\pi$, $\\mu$ and p, q is intended to help readers intuitively see the direct relationship between them from a formal perspective, which can reduce, to some extent, the difficulty of understanding for readers. Following your suggestion, we will consider introducing $\\pi$, $\\mu$ in the introduction considering that this paper belongs to RL field, and then placing the relationship between $\\pi$, $\\mu$ and p, q in the preliminary section.\n\n\n> **[2/3] W2:** Regarding (1). I think usually the advantage of Analytic xxx Algorithm is that you can compute some quantity so that you do not need to estimate them using a network. However, AEPO seems also need to train an energy guidance model (Eq. 25), which somehow weakens its advantage.\n\nThe reason we use \"Analytic\" is that our intermediate energy is derived from a theoretically grounded, analytically tractable formulation based on the Taylor approximation and the moment-generating function. This is in contrast to previous methods that rely on contrastive objectives such as CEP.\n\nTo avoid potential ambiguity in the title, we accept your suggestions to remove 'Analytic', resulting in the new title 'Energy-Guided Policy Optimization for Offline Reinforcement Learning'.\n\n\n> **[3/3] W3:** Line 177: \"Reminding that the posterior distribution µ0|t(a0|at, s) is also a Gaussian distribution\". This is wrong. µ_{0|t}(a_0|a_t,s) is not Gaussian, µ_{0|t}(a_m|a_t,s) can be considered as Gaussian only if m and t are close.\n\nWe thank the reviewer for the thorough and careful review. We will modify this sentence to: \"Following previous research [1,2], we use a Gaussian distribution as the estimate for the posterior distribution $μ_{0|t}(a_0|a_t,s)$.\"\n\n### [II]. Explanation of Questions\n\n> **[1/3] Q1:** In Eq 3, Do you mean s.t. KL(\\pi|\\mu) <= \\eta? It seems <= xxx is missing.\n\nYes, you are right. We will revise the equation as $D_{KL}(\\pi|\\mu) \\leq \\eta$.\n\n> **[2/3] Q2:** Why cannot you analytically compute the gradient of energy function from Eq. 17.\n\nWe would like to clarify that Equation (17) is used to calculate the posterior mean vector of the intermediate energy rather than the gradient of the energy function. We explain the reasons as follows.\n1) From Equation (11), the difficulty in differentiating the intermediate energy lies in the log-expectation, rather than $a_0$. \n2) Equation (17) is mainly used to estimate the mean of the posterior distribution, while actually generating a_0 requires multiple steps; if we directly backpropagate through $a_0$, the computational graph would consume significant computational overhead due to the multi-step generation process.\n\n> **[3/3] Q3:** Besides performance, what is the advantage of AEPO compared with previous work , especially QGPO and DPS.\n\nWe provide further discussion about the differences between AEPO, QGPO, and DPS in detail.\n- **From the intermediate energy guidance perspective**, AEPO and QGPO both use the exact intermediate energy, while DPS uses the inexact intermediate energy.\n- **From the theoretical perspective**, our method provides a theoretical solution for the log-expectation of the intermediate energy, which takes a further step compared with QGPO. QGPO only derives the formula of intermediate energy rather than the further results of the log-expectation.\n- **From the implementation perspective**: In Theorem 3.2 of QGPO, where the authors use contrastive learning loss to fit the intermediate energy, the authors mention that, under infinite data and model capacity, the contrastive loss (Equation (12) in QGPO) can perfectly approximate the intermediate energy. However, in practice, we cannot achieve infinite model capacity, and offline samples are certainly limited. Our method does not require assumptions about the sample size or model capacity.\n- **From the dataset perspective**: Furthermore, QGPO requires the use of a diffusion model to generate fake action vectors for CEP, and these fake action vectors may introduce the influence of out-of-distribution actions. However, our method does not require the generation of fake actions.\n\n**References**\n\n[1] Score-Based Generative Modeling through Stochastic Differential Equations.\n\n[2] Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models."}, {"review_id": "JeWAnC2jek", "reviewer": "Reviewer_eV4k", "summary": "The paper introduces Analytic Energy-Guided Policy Optimization (AEPO), a novel method for offline reinforcement learning that leverages diffusion policy with analytically derived energy guidance. Inspired by the connection between guided diffusion sampling and constrained RL, the authors aim to improve action generation by analytically estimating the exact intermediate energy term, which is usually approximated or manually designed in prior work. By Taylor expansion of the Q value function and moment generating function of the Gaussian distribution, AEPO give an explicit estimation of the intermediate energy function during the diffusion sampling.", "questions": "1. The first two terms in Equation (15) assume that $\\bar{a}$ and $a_0$ are independent. However, Appendix J.2 shows that the value of $\\bar{a}$ depends on $a_0$. Does this affect the accuracy of Equation (15)?\n\n2.  AEPO seems sensitive to several hyperparameters, such as the inverse temperature, expectile weight, and guidance degree. Does this sensitivity affect its usefulness across different tasks? How should these hyperparameters be chosen based on the characteristics of a task?\n\n3. Does AEPO offer any computational efficiency improvements compared to other state-of-the-art methods?", "limitations": "Consider improving how the paper presents its motivation, especially the connection between guided diffusion sampling and constrained RL. Although Equation (1) and Equation (2) are similar, the paper should provide a clearer intuition for why the Q-value function can be used as an energy function.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n\n-\tThis paper derives a closed-form approximation of the intermediate energy used in guided diffusion policy sampling by Taylor expansion and moment-generating functions.\n-\tAEPO achieves better performance for many D4RL tasks compared with SOTA diffusion-based and non-diffusion-based offline RL methods.\n-\tThis paper is well written and easy to follow, and it also provides detail proof about the theoretical results and experimental study.\n\n### Weaknesses:\n\n-\tThe closed-form approximation of the intermediate energy in AEPO heavily relies on the Taylor expansion of the Q-value function. However, the validity of a Taylor expansion requires more than differentiability—it also assumes that the first- and second-order terms dominate, and that higher-order terms are negligible. The paper lacks sufficient discussion or analysis on whether these conditions hold in practice. If the Taylor approximation deviates significantly from the true Q-value, then the analytically derived intermediate energy remains just an estimate. In that case, it may suffer from similar biases as previous approaches that relied on inexact intermediate energy approximations.\n\n-\tWhile AEPO introduces a theoretically grounded closed-form approximation of the intermediate energy, its empirical advantage over other SOTA diffusion policy methods is relatively modest. Notably, Appendix B.3 shows that the proposed Guidance Rescaling technique for the diffusion score function yields significant performance gains. This suggests that the reported improvements may be largely attributed to this auxiliary component, rather than the core contribution of the closed-form intermediate energy itself. As a result, the experimental comparisons may not be entirely fair or isolating the effect of the main innovation. Furthermore, the paper lacks ablation studies on the impact of different Q-function learning strategies, which would be important for understanding the robustness and general applicability of AEPO.\n\n-\tThe paper omits several relevant and competitive baselines, particularly offline RL methods that incorporate uncertainty based pessimistic Q-value estimation, such as UWAC[1] and QDQ[2].\n\n[1]  Wu, Y., Zhai, S., Srivastava, N., Susskind, J., Zhang, J., Salakhutdinov, R., & Goh, H. (2021). Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08140.\n\n[2] Zhang, J., Fang, L., Shi, K., Wang, W., & Jing, B. (2024). Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model. Advances in Neural Information Processing Systems, 37, 54421-54462.", "was_revised": true, "final_justification": "Most of my concerns have been addressed by the author rebuttal!", "rebuttal": "Dear Reviewer eV4k,\n\nWe appreciate your suggestions and respond to your concerns below.\n\n### [I]. Explanation of Weaknesses\n\n> **[1/3] W1:** The closed-form approximation of the intermediate energy in AEPO heavily relies on the Taylor expansion of the Q-value function. However, the validity of a Taylor expansion requires more than differentiability—it also assumes that the first- and second-order terms dominate, and that higher-order terms are negligible. The paper lacks sufficient discussion or analysis on whether these conditions hold in practice. If the Taylor approximation deviates significantly from the true Q-value, then the analytically derived intermediate energy remains just an estimate. In that case, it may suffer from similar biases as previous approaches that relied on inexact intermediate energy approximations.\n\nWe thank the reviewer for raising this important and technically insightful review. Through the choice of activation function and network structure, we can theoretically show that the Q-function is twice differentiable and smooth with respect to the action, which forms the basis for our local Taylor expansion (Equation (14)) [1]. In order to control the error bound of Taylor expansion, we use the following method to choose $\\bar{a}$ and prove the error satisfies $R_2(a)\\leq \\frac{1}{2} \\nu^2$. We would like to clarify that we have provided the detailed proof in Appendix J.2, but to make the current content clear and complete, we have rewritten the proof below. Additionally, we will emphasize this proof in the revised version.\n\n**Proof of Taylor expansion error control:**\n\nConsidering the error bound of Taylor expansion and the stability of the training process, we use the following method to obtain $\\bar{a}$: \n$\\bar{a} = a - \\nu*\\frac{Q^\\prime_{\\psi}(s, a_0)}{||Q^\\prime_{\\psi}(s, a_0)||},$ \nwhere $\\nu=0.001$. Considering the loss term of the second-order approximation\n$R_2(a) = \\frac{1}{2}\\nabla_a^2 Q(s, c)(a-\\bar{a})^2,$\nwhere $c$ is the point between $[\\bar{a}, a]$. Thus,\n$||\\nabla_a Q(s, a)||*||a-\\bar{a}||=\\nu,$\nSo the error satisfies \n$R_2(a)\\leq \\frac{1}{2} \\nu^2.$\n\n> **[2/3] W2:** While AEPO introduces a theoretically grounded closed-form approximation of the intermediate energy, its empirical advantage over other SOTA diffusion policy methods is relatively modest. Notably, Appendix B.3 shows that the proposed Guidance Rescaling technique for the diffusion score function yields significant performance gains. This suggests that the reported improvements may be largely attributed to this auxiliary component, rather than the core contribution of the closed-form intermediate energy itself. As a result, the experimental comparisons may not be entirely fair or isolating the effect of the main innovation. Furthermore, the paper lacks ablation studies on the impact of different Q-function learning strategies, which would be important for understanding the robustness and general applicability of AEPO.\n\nThe purpose of using guidance rescale is to make our method less sensitive to $\\omega$. The main contributions are our theoretical results of intermediate energy and systematic comparison results with dozens of baselines on 33 tasks.\n\nThe results in Table 1 show that the best performance achieved via grid search under both guidance rescale and non-guidance rescale settings is comparable, indicating that guidance rescale is not the primary factor contributing to significant performance gains.\n\n**Table 1**: The best performance comparison of guidance rescale and non-guidance rescale with grid search.\n|Type|guidance rescale|non-guidance rescale|\n|-|-|-|\n|walker2d-me|109.3±0.5|109.4±0.9|\n|walker2d-mr|90.8±1.5|92.8±4.8|\n|halfcheetah-mr|43.7±1.3|43.3±1.5|\n\nWe want to clarify that regardless of which Q function training method is used, as long as the Q function learning is suitable for offline RL, the performance obtained using our method is similar. For example, with IQL-like Q function training methods and CQL-like Q function training methods, when obtaining accurately estimated Q functions, the performance of AEPO is similar. As shown by the experimental results in Table 2, this experiment used two methods to train the Q function, one being an IQL-like training method and the other using a CQL-like training method. We find that the experimental performance was similar, indicating that we just need to adopt the Q-function training method that is suitable and widely used in offline RL.\n\n**Table 2**: The influence of different training methods on the Q function. The experiments are conducted on AEPO but with different training methods of Q functions.\n|Q function training type|IQL-like Q function training|CQL-like Q function training|\n|-|-|-|\n|walker2d-me|109.3±0.5|109.8±0.7|\n|halfcheetah-me|94.4±0.9|93.8±0.9|\n|hopper-me|111.5±1.1|111.2±2.4|\n\n> **[3/3] W3:** The paper omits several relevant and competitive baselines, particularly offline RL methods that incorporate uncertainty based pessimistic Q-value estimation, such as UWAC[2] and QDQ[3].\n\nWe thank the reviewer for the valuable advice.\nWe would like to clarify that the research focus of our paper is not on Q-value estimation, but rather on the diffusion model's guided-sampling with intermediate energy. Both of these reference papers focus on Q-value estimation. Our research emphasis is distinctly different.\n\nFollowing your suggestion, we will add experimental comparisons with UWAC and QDQ in the main body. Table 3 illustrates a significant improvement in Locomotion tasks by comparing our method with QDQ. From Table 4, we can see that our method surpasses UWAC on 10/12 Adroit tasks and reaches a higher mean score than UWAC. \n\n**Table 3**: The experiments comparison on the Locomotion tasks.\n|Model|Antmaze-umaze|Antmaze-umaze-diverse|Antmaze-medium-play|Antmaze-medium-diverse|Antmaze-large-play|Antmaze-large-diverse|mean score|\n|-|-|-|-|-|-|-|-|\n|QDQ|98.6|67.8|81.5|85.4|35.6|31.2|66.7|\n|AEPO|100.0|100.0|76.7|83.3|56.7|66.7|80.6|\n\n**Table 4**: The experiments comparison on the Adroit tasks.\n|Model|pen-h|pen-e|pen-c|hammer-h|hammer-e|hammer-c|door-h|door-e|door-c|relocate-h|relocate-e|relocate-c|mean score|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|UWAC|65.0|119.8|45.1|8.3|128.8|1.2|10.7|105.4|1.2|0.5|108.7|0.0|49.6|\n|AEPO|76.7|147.0|69.3|10.3|129.7|6.4|9.0|106.5|3.9|0.8|107.0|0.6|55.6|\n\n### [II]. Explanation of Questions\n\n> **[1/3] Q1:** The first two terms in Equation (15) assume that \n $\\bar{a}$ and $a_0$ are independent. However, Appendix J.2 shows that the value of $\\bar{a}$ depends on $a_0$. Does this affect the accuracy of Equation (15)?\n \nWe want to clarify that in Equation (15), we did not assume that $\\bar{a}$ and $a_0$ are independent. In Equation (15), $a_0$ is a general notation symbol, while $\\bar{a}$ represents any constant vector. In Appendix J.2, to control the loss term of second-order approximation, we introduce $\\nu$ and use $a_0$ to calculate $\\bar{a}$.\n\n> **[2/3] Q2:** AEPO seems sensitive to several hyperparameters, such as the inverse temperature, expectile weight, and guidance degree. Does this sensitivity affect its usefulness across different tasks? How should these hyperparameters be chosen based on the characteristics of a task?\n\nIn the experiments, the values of inverse temperature $\\beta$ and expectile weight $\\tau$ are $\\beta$=1 and $\\tau$=0.5, which is suitable for most tasks. Regarding guidance degree $\\omega$, to make our method robust to $\\omega$ values, we propose guidance rescale, and experimentally demonstrate that guidance rescale can make our method robust for $\\omega$ in a wider range. Usually, $\\omega=0.1$ is appropriate for most tasks after guidance rescale.\n\n> **[3/3] Q3:** Does AEPO offer any computational efficiency improvements compared to other state-of-the-art methods?\n\nAlthough we acknowledge that AEPO does not offer explicit computational efficiency improvements over existing SOTA methods, we would like to emphasize that we provide **rigorous theoretical results regarding intermediate energy**, and have compared with **30+ baselines** on **33 tasks**, achieving SOTA performance on most tasks, demonstrating the enormous potential of our method.\n\n### [III]. Explanation of Limitations\n> **[1/1] L1:** why the Q-value function can be used as an energy function.\n\nThank you for your insightful comments. We try to address your concern by clarifying the intuition behind using the Q-value function as an energy function.\n\nIn reinforcement learning, the Q-value Q(s,a) represents the expected return of taking action a in state s. Higher Q-values indicate more desirable actions. In energy-based guided sampling, the energy function E(x) serves as a scoring function, where lower energy corresponds to higher sampling probability on x.\nBy identifying the energy function as E(x)=−Q(s,a), we make high-Q actions more likely to be sampled through the exponential weighting $\\beta$Q(s,a). This aligns with the optimization objective in constrained RL: maximizing the expected Q-value under a KL divergence constraint from a prior behavior policy.\nWe will clarify this relation and the underlying intuition more explicitly in the revised version to help readers better understand the connection between energy-function-guided diffusion models and the constrained RL problem.\n\n### [IV]. Explanation of Paper Formatting\n\nThank you for your careful review. We will place the checklist behind the appendix in the revised version.\n\n**References**\n\n[1] Principles of mathematical analysis.\n\n[2] Uncertainty weighted actor-critic for offline reinforcement learning.\n\n[3] Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model."}]

Author rebuttal:

Meta-review: This paper studies the problem of offline RL with diffusion-model-based algorithms, where a diffusion model is learned for the behavior policy and guided by an energy function derived from the Q-function. A key challenge is that the theoretically optimal guidance signal—the intermediate energy—is defined by an intractable log-expectation formulation. Motivated by the connection between guided diffusion sampling and constrained RL, the authors propose a more principled approach to estimating this intermediate energy term, which prior work has typically approximated or manually designed. Specifically, they linearize the Q-function via a first-order Taylor expansion and apply the moment-generating function of the Gaussian distribution to obtain an explicit and tractable estimation of the intermediate energy during diffusion sampling. This method, termed Analytic Energy-guided Policy Optimization (AEPO), bridges the gap between theory and practice. Empirically, AEPO demonstrates SoTA performance on most tasks in the D4RL benchmark, while maintaining computational efficiency comparable to existing approaches. 

All reviewers have recognized the technical contribution to be important. The rebuttal further strengthened the submission by (1) clarifying the validity of the Taylor approximation through the smoothness of the SiLU activation, which ensures bounded errors, and (2) demonstrating robustness to the rescaled guidance parameter. These clarifications have addressed serveral reviewer's critical concerns regarding both the theoretical foundation and the practical impact of the method. Overall, the paper makes a principled, technically solid, and well-validated contribution.
Decision: Accept (poster)