Prior-Guided Diffusion Planning for Offline Reinforcement Learning
Authors: Donghyeon Ki
JunHyeok Oh
Seong-Woong Shim
Byung-Jun Lee
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion Models
Offline Reinforcement Learning
Planning
Primary area: reinforcement_learning
TL;DR: We propose Prior Guidance, a novel guided sampling framework for diffusion-based planners in offline RL that replaces the standard Gaussian prior of a behavior-cloned diffusion model with a learnable distribution.

Abstract: Diffusion models have recently gained prominence in offline reinforcement learning due to their ability to effectively learn high-performing, generalizable policies from static datasets. Diffusion-based planners facilitate long-horizon decision-making by generating high-quality trajectories through iterative denoising, guided by return-maximizing objectives. However, existing guided sampling strategies such as Classifier Guidance, Classifier-Free Guidance, and Monte Carlo Sample Selection either produce suboptimal multi-modal actions, struggle with distributional drift, or incur prohibitive inference-time costs. To address these challenges, we propose \textbf{\textit{Prior Guidance}} (PG), a novel guided sampling framework that replaces the standard Gaussian prior of a behavior-cloned diffusion model with a learnable distribution, optimized via a behavior-regularized objective. PG directly generates high-value trajectories without costly reward optimization of the diffusion model itself, and eliminates the need to sample multiple candidates at inference for sample selection. We present an efficient training strategy that applies behavior regularization in latent space, and empirically demonstrate that PG outperforms state-of-the-art diffusion policies and planners across diverse long-horizon offline RL benchmarks. Our code is available at https://github.com/ku-dmlab/PG.

Prior-Guided Diffusion Planning for Offline Reinforcement Learning
Donghyeon Ki!
JunHyeok Oh
Seong- Woong Shim
Byung-Jun Lee
Korea University
Gauss Labs Inc.
(peoplein, the2ndlaw, ssw030830, byungjunlee) www.uneeeeeeeeee.eeeeeeae akorea.a........................ ac.kr
Abstract
Diffusion models have recently gained prominence in offline reinforcement Iearn- ing due their ability effectively learn performefor,ngg, generalizable policies from static datasets. Diffusion-baree planners facilitate long-horizon decision- making by generating high-quality trajectories through iterative denoising, guided by return-maximizing objectives. However, existing guided sampling strategies such as Classifier Guidance, Classifier-Free Guidance, and Monte Carlo Sample Se- lection either produce suboptimal multi-modal actions, struggle with distributional drift, or incur prohibitive inference- time costs. To address these challenges, we propose Prior Guidance (PG), novel guided sampling framework that replaces the standard Gaussian prior a behavior-cloner diffusion model with learnable distribution, optimized via behavior-regularizes objective. PG directly gener- ates high-value trajectories without costly reward optimization of the diffusion model itself, and eliminates the need sample multiple candidates inference for sample selection. We present an efficient training strategy that applies behavior regularization in latent space, and empirically demonstrate that PG outperforms state-of-the-art diffusion policies and planners across diverse long-horizon offline RI. benchmarks. Our code is available at suthttps:// : thub.
Introduction
Offline reinforcement learning (RL) allows agents to learn effective policies from fixed, pre-collected datasets, eliminating additional environment interaction. This paradigm is especially beneficial in scenarios where exploration prohibitively expensive, unsafe, impractical [1] A core challenge in offline RI. arises from distributional shifts, which can lead learned policies to favor out-of-distribution
(OOD) actions due to overestimated value functions. To mitigate this issue, many existing methods incorporate behavior regularization to keep the learned policy close to the behavior policy that generated the data 3
Recently, diffusion- based planners have been proposed as a promising direction for offline RL. Leveraging their strengths in modeling long-range temporal dependencies and conditioning auxiliary information [3] 6, & 9 [0]. diffusion models are well-suited for sequence modeling in complex decision-making tasks. By conditioning future return surrogates, diffusion models generate trajectories biased toward high-value behaviors, thereby demonstrating superior performance over prior approaches in long-horizon navigation and manipulation benchmarks across diverse offline
RL settings [2 1
15
Diffusion planners have employed various guided sampling strategies steer trajectory generation toward high-return outcomes. Classifier Guidance (CG) [6] leverages a learned critic to directly guide the denoising process toward high-value trajectories. Classifier-Free Guidance (CFG) [16], in contrast, achieves conditioning by interpolating between conditional and unconditional diffusion
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Behavior Diffusion Model
Prior Guidance
V(xo)
#p(x0 (s)
TTp( (xu) s)
Xg
Xo
Denoising process
Denoising process Tel [xo xr,s)
A (xx]{xrss)
XT IO,I)
xT Po(-(s)
Figure 1: Comparison between the behavior-clou clloeed diffusion model and Prior Guidance. (Left) The behavior diffusion model samples latent noise Xy from standard Gaussian and generates trajectories Xn via learned denoising process i (r, approximating the dataset distribution. (Right) Prior Guidance instead samples Xy from a learned prior pv (xr(s)) concentrated in high-value regions, leading to improved trajectory generation through the same denoising process.
models, removing the need for an additional network. While conceptually appealing, these methods face practical limitations: CG can produce suboptimal actions in settings where the optimal policy is multi-modal [17, and CFG often fails to generalize when the conditioning signal diverges from those seen during training, resulting in unreliable plans.
To address these challenges, recent high-performing diffusion planners have adopted simple yet effective alternative Monte Carlo sample selection (MCSS) [8] 19. MCSS generates N candidate trajectories and selects the one with the highest estimated return using a separately learned critic. While empirically successful, MCSS faces three primary limitations: (i) it incurs significant computa- tional overhead at evaluation time due to the need to sample and score multiple trajectories, (ii) it does not guarantee the sampling of high-quality trajectories in the dataset, potentially overlooking valuable samples with high estimated value but low sampling likelihood, and (iii) increasing N to improve coverage can lead to selecting outtof-distribution trajectories with spuriously high critic estimates.
Rather than guiding a pretrained diffusion model at inference time, an alternative approach is to directly train the diffusion model to generate high-value behaviors by maximizing learned critic under behavior regularization 20 22 23]. However, this training objective typically incurs substantial computational overhead, as it requires backpropagation through the entire denoising process. Consequently, these efforts have primarily focused on developing diffusion policies rather than diffusion planners. Moreover, since diffusion models do not offer tractable density evaluation. most existing methods have relied on simple forms of behavior regularization, such as incorporating auxiliary behavior cloning losses.
To this end, we propose Prior Guidance (PG), novel guided sampling method that trains the prior distribution that decodes into high-value trajectories. After the training of a diffusion model through behavior cloning, we replace the standard Gaussian prior with a learnalee parametrized prior distribution. This learnable prior is optimized using a behavior-regularized objective, encouraging the sampling of high-value trajectories through the diffusion model while remaining close to the behavior distribution (See Figures and
PG effectively addresses the limitations of previous methods. contrast MCSS, PG generates a single trajectory during inference, significantly reducing computational overhead. Compared end-to-end training of the diffusion model with behavior-regulariied objective, PG more efficient, as it avoids backpropagation through the full denoising process and requires optimizing only a small set of parameters. Furthermore, by modeling the target prior as a simple Gaussian distribution, PG enables more expressive and analytically tractable behavior regularization, achieving a more favorable trade-off between maximizing potential performance and mitigating the risk of out-of-distre -iloo--distriuttion action selection. Empirically, PG outperforms existing diffusion-basee methods and achieves state-of-the-art performance on long-horizon tasks the D4RL, offline RI. benchmark suite (24).

---

Denoising process
XT
X0
Start Gool
Classifier Guidance
Stort
Goal
MCSS
Stntt
Geot
Prior Guidance
Figure 2: Visualization of trajectory generation using three different guided sampling methods maze2d-large-vi Each method produces 20 planned trajectories. For MCSS, each trajectory is selected from set of 10 candidates (N 10); the unselected candidates are visualized in gray.
2 Preliminaries
Diffusion Probabilistic Models Diffusion models [5] are a class of generative models based on Markovian forward-reverse process. The forward diffusion process gradually corrupts a data sample Xu by adding Gaussian noise T steps, defined as q(x1:T x,) I (xxx (x,,)) where (xx((x(-1) Nxxx vI -, 3.1)) 3,I) and 3r E (0, 1) is a predefined variance sched- ule. The reverse process is parameterized Markov chain starting from a standard Gaussian prior p(XT) (O,, I), defined ps (Xo.T) p(xx) II Pa 1x:), where pa (xt-1|xx) (xx-11; (A (x,, l), Eg [x, ()) and both pag and Eg are predicted by a neural network.
The model is trained by minimizing variational upper bound on the negative log-likelihood. In practice, a simplified objective widely used, where the model learns predict the added noise:
((0) Exp.e,t [||e - eg (Váxxo + vI- Cigt, 1)]
where 6 I) and ar 11 3.). This training objective corresponds to denoising score matching across multiple noise levels. Once trained, the model can generate new samples by iteratively following the reverse process defined above. Alternatively, (25 proposed another generative process that shares the same marginal distribution, defined by
X1
VI (xco(xx,t) + / 1 ā,-1 of co(, + oee,
X;-1 0
Vag
where e ~ (0, I) and a, is a tunable parameter that controls the stochasticity of the generation. When 0z 0. the generative process becomes entirely deterministic, resulting in the Denoising Diffusion Implicit Model (DDIM) sampling scheme.
Offline Reinforcement Learning We assume the reinforcement learning problem under a Markov Decision Process (MDP). The MDP can be represented as tuple M = (S, A, P. R, 7, Po where S is the state space, A is the action space, P(s' (s. a) S x A A(S) is transition dynamics, R(s, a) S x A + is a reward function, Po E A(S) an imitial state distribution and y E [o, 1) discount factor. Offline RL aims to train parameterized policy (als), which maximizes the expected discounted return from a dataset D without environment interactions.
A common strategy in offline RL is to optimize policies via behavior-regularized objec- tive: max Eg- (-1a)) Q(ss. a) af (n|s)) where Q denotes the action-value func- tion of policy Tas my is the behavior policy, and f(.)) is regularization function (e.g., for f-
<p(a|s]
divergence) 4 26 27 28 However, when employing diffusion policy planner to model

---

N-1 N 10
N 50
N - 500
N 5000
TI
Figure 3: Predicted state sequences in maze2d-large-ee based on MCSS varying the number of candidate trajectories N. When N is small (N 1), MCSS often fails to sample trajectories with high value estimates, as the likelihood of capturing an optimal trajectory within a limited set is low. Conversely, when N is excessively large (N 500 or 5000), the method tends to select OOD trajectories with inflated value estimates.
Tah directly evaluating the density T becomes intractable, making it challenging to apply sophisticated regularization functions. As result, prior works have instead resorted using diffusion behavior cloning loss ( as a surrogate for behavior regularization [20] 21 22 23).
[9] generate trajectory of length H starting from the current time step т. While various design choices are available, in this work, largely follow the recommendations of [19. which are based on comprehensive experimental analysis. Specifically, a diffusion planner is trained to generate H-length sequences of current and future states x(r) = [s(]), g(T++)) ,..., ( ((++(M)1)) with the planning stride of m, where т denotes the environment timestep. To bias the generation toward high-return behaviors, trajectory critic ((x(7) E
Diffusion Planner Diffusion- based planners
13 14 13
- (() xx(r] is estimated and utilized during planning. This critic can be either the behavior value function or the target value function. After producing a high-value state sequence x()), separately trained inverse dynamies model employed to recover the corresponding action (=) from the predicted sequence. For simplicity, we overload the notation and denote the diffusion planner-based policy as iT (x(s)..
A detailed discussion of related work is provided in the Appendix A
3 Diffusion Planning with Prior Guidance
Limitations Monte Carlo Sample Selection (MCSS) Monte Carlo Sample Selection (MCSS), which selects the best sample from N candidates based a separately trained value function, has been shown to achieve superior performance and has been widely adopted in recent diffusion- based offline RL [17 18 19]. Nonetheless, discussed above, MCSS also exhibits significant limitations: (i) it incurs substantial computational overhead during inference, as it requires sampling multiple candidates from diffusion models, which already expensive sample from, (ii) it does not guarantee coverage of high-value samples within the behavioral distribution, and (iii) while a sufficiently large N is necessary for effective guidance, excessively large N increases the risk of selecting OOD samples with overestimated value estimates, leading to performance degradation Figure illustrates these limitations.
3.1 Prior Guidance
To mitigate these shortcomings, we aim to adopt a behavior regularization framework in providing guidance for diffusion planners. Concretely, the objective is formalized as:
T (xo
max Eg (-1s) V(xo)= - -af
T [ (s)
where Xo denotes a fully denoised trajectory generated by the diffusion model. This objective promotes the generation of high-value trajectories while constraining them stay close to the dataset.
'A notable variant is SBBC 29 which performs resampling by assigning weights proportional exp(aQ(s, a)) over a set of N candidate actions generated by a diffusion policy. This weighting scheme implicitly induces KL-bchavior regularization and mitigates limitation (iii). However. introduces an additional hyperparameter a, increasing the complexity of hyperparameter tuning. Moreover, due to other unresolved limitations, the method continues to incur significant inference time computational overhead.

---

However, directly optimizing this objective with diffusion models poses significant challenges: (i) it requires expensive backpropagation through the iterative denoising process, which becomes particularly costly for nigh-dimensional diffusion planners; and (ii) evaluating T (Xo (s) is non-trivial, as diffusion models do not admit tractable density estimation.
We now introduce Prior Guidance, a method for guiding diffusion models with learnable prior distribution, which is free from the abovementioned challenges. Let T. denote diffusion planner trained via behavior cloning. Given a state s, the model generates a trajectory X by first sampling an initial noise Xy p(xr) (O, I), followed by denoising process. Assuming a DDIM sampling, the conditional density T (Xo s) can be expressed as:
(s) = Ep(x)) [ō(xo 9a (xx)))]
where 6() denotes the Dirac delta function, and Is is the deterministic denoising operator of the behavior diffusion model itz for state s.
22 has demonstrated that interpolations in the latent space yield smooth and semantically meaningful trajectories in the sample space, indicating that xy captures the semantic structure of Xo. This observation motivates shifting where the learning happens from the diffusion model to the prior distribution. Accordingly, we propose to learn a parameterized Gaussian prior Pol (xTls), [s), while keeping the denoising operator ys fixed. The trajectory distribution after planning Tे can then be reformulated as:
Te (xols) = (xp((xre)) (s) [6(xo - Is (xx)))
When a sufficiently large number of discretization steps are employed, the denoising operator ys can be viewed an approximate, bijective ODE solver. This bijection allows us to re-express previously intractable behalior-regularized objective of the diffusion model in terms of a tractable objective over prior distributions:
Proposition 1. Assume that DDIM sampling employs a sufficiently large number of discretization steps ensuring that the mapping xo 9s (x(x) is bijective. If the target trajectory density Th is parametrized by placing the prior (x()))s) while keeping the denoising process ge fixed. the behavior-regularizer objective
2 is equivalent to:
max E. Ea D V (gs (XT)) - af (P((x](s)
)]
p(XT)
1
The proof is in the Appendix B Note that the behavior regularization term f(-) now evaluated between two Gaussian distributions (pe and p). rather than between intractable diffusion model densities, which enables closed-form computation under several popular formulations of f.
Avoiding Backpropagation Through the Denoising Process The reformulated objective 3 still depends Js. requiring costly backpropagation through the denoising process. To avoid this, introduce a value function V that operates directly in the latent prior space Xy rather than on the denoised output x, = Is (x7). Due to the bijectivity of Ja and the smoothness of the latent space induced by the diffusion model, the latent value function V is well-defined and straightforward to learn. It trained by minimizing the following objective:
min BxinExp-ps (Vx)(x)) - -V (o.((x))))
where (xT()))) serves as the target signal. Once trained, the latent space value function Va is used replace the original critic in the behavior-regularizee objective (3):
( (Pw(xT(s) ) p(XT)
max Eg Dx Vo((T)) -af
(5)
This training scheme completely eliminates the need for gradient flow through the denoising operator Is. Gradients are propagated only through the prior distribution Py and the value function Vg, while Is remains fixed. Consequently, PG circumvents the major computational bottleneck associated with directly optimizing diffusion models for high-value behavior generation, thereby enabling its application to diffusion planners, whereas previous approaches were constrained to diffusion policies due to their computational overhead. In practice, we alternate between minimizing the regression loss in Eq. (4) and optimizing the prior via Eq. (5). In Section 4.2, we qualitatively analyze the latent value function V and show that it accurately captures the intended value structure.
5

---

Training Time
Inference Time
Time (minutes Funai conv.
Time Deconds 33K 3Kspapless
610
2023
500
Do: (olnnee to Up
320
420
2300
20.1
330
352
1200
3.0
500
127
100
24
Mujoco
Kitchen
Antmaze
Moze
10
50
500
PG
Figure Training and inference time comparison. (Left) Training time until convergence for DQL (planner), PG with and without backpropagation through the denoising process across dif- ferent environments. DQL (planner) denotes a baseline where the diffusion model is directly trained via Eq. (2); due to the intractability of computing the model's density, Eq. T is used as a surrogate. (Right) Inference time for MCSS and PG. The inference time is measured on the antmaze menzimm palmnmplll222 with MCSS evaluated under different numbers of sampled trajectories.
MujoCo
Kitchen
Antmaze
Maze2D
464
175
Averege Score
80
80
150
125
30-
100
822
52.
60
70
50
500
PG
10
50
500
10
50
500
30
50
500
Figure 5: Average normalized score comparison between MCSS with varying numbers of samples N and PG across four D4RL tasks. PG consistently matches outperforms MCSS while requiring significantly fewer samples. Full results are provided in the Appendix D
3.2 Does PG Address the Limitations of MCSS?
We empirically validate whether PG indeed resolves the limitations of MCSS, as claimed above. To ensure fair comparison, all other experimental settings were held constant when comparing PG and MCSS.
(1) Reduced computational cost during inference. MCSS requires sampling multiple candidate trajectories at each environment step and selecting the one with the highest estimated value using a critic. Since sampling from a diffusion model is inherently computationally expensive, generating multiple candidates introduces substantial overhead, particularly high-dimensional or long-horizon tasks. In contrast, PG generates single trajectory starting from learned prior, thereby avoiding this computational burden. Although PG introduces the additional cost of training the prior, circumvents the need for costly backpropagation through the full denoising process by training latent space value function, enabling efficient training.
As shown in Figure 4 PG w/o Bp Bu-our denoising process significantly reduces training time compared PG w/ Bp. Moreover, PG achieves substantially lower inference time than MCSS, especially when the number of sampled trajectories N large, highlighting its superior computational efficiency. We also report the training time of Diffusion Q-learning when implemented as planner- method, denoted as DQL (planner). As expected, DQL (planner) significantly more con-ccon to train than PG, taking up to ten times longer in the AntMaze domain. More experimental details for DQL (planner) are provided in the Appendix C.T
our proposed method that removes backpropagation through the
(2) Tractable and stable behavior regularization. MCSS lacks an explicit mechanism to guide sampling toward trajectories aligned with the behavioral distribution, often resulting in OOD tra- jectory selections with spuriously high value estimates. PG addresses this limitation by performing regularization directly the prior space, where the divergence between the learned prior ps and the standard Gaussian prior admits a closed-form expression. This enables stable and analytically tractable behavior regularization, which was previously infeasible in diffusion models due to the intractability of their output densities.

---

Table 1: Normalized scores on the D4RL benchmark. For each dataset, the best- best-performing method highlighted in red. Prior Guidance (PG) consistently achieves strong performance, particularly on Long-horizon tasks, and outperforms DV* in nearly all environments. We compute the mean and standard error over 5 random seeds. Due to space constraints, the standard error for all algorithms are provided in the Appendix B
Gaussian polieies
Diffusion policiiss
Diffusion planners
Dataset
CQL
IQI.
SIBC
DQQI.
IDQI.
QGPO
SRPO
Diffuser
Walker2d-M
6.6
79.2
78.3
77.9
87.0
82.5
86.0
$1.4
79.5
84.4
82.5
81.0
79.5
82.3
Walker2d-M-RR
11.
26.7
73.9
65.1
95.5
85.1
84.4
84.6
70.6
84.7
75.0
84,1
83.5
83.7
Walker2d-M-F
112.7
110.7
114.0
106.9
108.2
107.1
109.0
109.4
Hopper-M
29.0
$8.0
66.3
57.1
90.5
65.4
98.0
95.5
74.3
96.5
79.3
99.3
84.1
97.5
Hopper-M-R
11.3
48.6
94.7
86.2
101.3
92.1
96.9
101.2
93.6
92.2
100.0
94.7
91.3
91.3
Hopper-M-E
108.6
111.6
110.4
HallChoctsh-M
36.1
44.4
47.4
45.9
51.1
$1.0
54.1
60.4
42.8
14.2
49.1
-6.77
50.9
45.6
HalCCheetab M-R
38.4
46.2
44,2
37.1
47.8
45.9
47.6
51.4
37.7
38.3
39.3
38.1
46.4
46.4
HalfChestah M-E
35.8
86.7
92.6
96.8
95.9
93.5
92.2
88.9
89.6
90.6
92.5
92.3
95.2
Average
31.9
75.6
83.3
83.0
84.6
Kitchen-M M
47.5
$1.0
$1.0
45.4
62.6
66.5
52.5
51.8
65.0
71.7
73.3
74.6
Kitchen.P
33.8
49.8
46.3
47.9
60.5
66.7
55.7
55.5
57.0
73.3
76.2
88.0
Average
40.7
50.4
48.7
46.7
61.6
66.6
54.1
53.7
61.0
72.5
74.8
81.3
Antmaze M.P
0.0
14.9
71.2
81.3
76.6
84.5
83.6
80.7
6.7
12.0
8.0
$0.8
87.8
Antmaze-M-D)
75.0
6.0
4.0
88.7
87.3
Antmaze-L..P
59.3
63.5
53.6
5.3
82.4
Artnase-L-
0.0
61.2
47,5
45.5
56.6
67.9
64.8
53.6
27,3
8.7
0.0
83.6
76.0
76.0
Averuge
0.0
36.4
57.1
67.0
64.6
75.2
74.7
65.7
13.3
8.0
3.0
79.9
83.4
Maxs2D-U
3.8
5.7
47.4
73.9
57.9
113.9
135.1
128.4
133.8
139.2
Mare2D-M
30.3
5.0
34,9
73.8
89.5
121.5
129.9
135.6
144.1
159.5
MaxeDDL
12.5
58.6
74.4
90.1
123
167.9
155.8
197.6
195.2
Averuge
79.2
158.5
164.6
Empirical results presented in Figure corroborate cornoooaeee this advantage: while MCSS exhibits trade-off between trajectory optimality and OOD avoidance, PG with appropriate behavior regularization consistently outperforms MCSS with the best performing N across diverse domains.
4
Experiments
Practical Implementation The target prior Py in PG is parameterized using GRU (30 to capture the temporal dependencies in x. Con- ditioned on the current state, the prior network outputs sequence of mean and log standard de- viation vectors that define independent Gaussian distributions over the latent noise dimensions. The current state is first provided as input to the GRU, and its output (prior to sampling) is fed back as input for subsequent steps. Detailed experimental settings are provided in the Ap- pendix
Algorithm 1 Prior Guidance
Input: Dataset D, Hyperparameter a Require: Planner ge, Inverse Dynamics ews Critic V Initialize: Prior Pv: Critic V.
1: // Training
2: for each iteration do
3: Sample D and xT p
Update critic Vo using Eq.
5:
Update prior pv using Eq.
6: end for
7: // Execution
8: (0)) env.reset()
9: for environment step 7 = I,...
Apart from the learnable prior, PG is built on top of Diffusion Veteran (DV) 19, following their design choices. Due to the long training time of the original implementation, we re-implemented DV and refer to this version as DV* PG inherits
10: Sample X po (T))
11:
Generate Xo using 9s
12: Predict action a using ew (xo)
13: g(++1) env.step(a (f)
14:
end for
from DV* the planner Ga. the inverse dynamics
model e.. and the critic V as described in Algorithm[1 [T. following DV, PG uses the behavior value function for its critic V. The pseudocode for DV* is provided in the Appendix D along with a detailed comparison the performance of the original DV implementation.
4.1 D4RL Benchmark
We conducted experiments on the D4RL offline RI. benchmark 24 which span wide range of domains and dataset settings. We evaluate Prior Guidance (PG) across four representative tasks:
7

---

Xg
XT
Behavior Diffusion Model
Prior Guldance
Behavior Diffusion Model
Prior Guldance
css
1000
-5
-10
92
-20
20
10
Figure 6: Visualizations of trajectories (Xx) and priors (xT) from the behavior diffusion model and Prior Guidance in maze2d- -araage- (top) and antmaze commaazeeediia
(bottom). We applied PCA 334 to project the high-dimensional trajectories into two dimensions, with the color of each sample representing its estimated value. We plotted 5,000 samples from the standard normal prior of the behavior diffusion model and 1,000 samples from the target prior of PG.
Walker2d
Hopper
Dataset
PG w/ Bp
PG w/o Bp
0.6
Correlation: 0.8149
0.6918
0.-
0.373
MuloCo
85,8+-0.3
84.6+0.2
0.4-
0.399
Kitchen
81.0+-2.4
11.
Antmaze
84.0+-1.2
83.4+0099
0.3
Maze2D 164.2+
164.6-0.9
02
0.4
0.6
0215 9.300 0,325 0.353 0,375 0.400
Total
415.0
413.9
Vgltr)
Vy(or)
(a) Correlation between V (xT) and (xx) in walker2d
(b) Average scores and without back- propagation through the denoising process.
hopper environments.
Figure 7: Experiments to evaluate whether the latent value function V accurately trained.
MuJoCo, Kitchen, Antmaze, and Maze2I). In MuJoCo tasks, the dataset types are denoted as M (medium), R (replay), and E (expert); in Kitchen tasks, M (mixed) and P (partial); in Antmaze tasks, M (medium), L (large), (play), and D (diverse); in Maze2D tasks, (umaze), M (medium), and L (large). We benchmark against BC, CQL 31. IQL [32] SIBC (29 DQL (20] IDQL (18), QGPO [33]. SRPO (28), Diffuser [I]. AD 13,, DD 12). HD [15. and DV 19 The compared methods are categorized into Gaussian policies (CQL, IQL), diffusion policies (SfBC, DQL, IDQL,
QGPO, SRPO). and diffusion planners (Diffuser, AD, DD, HD, DV). The normalized scores are summarized in Table
As shown in Table PG achieves state-of-the-ar performance on tasks that require long- long-horizon decision making. By addressing the key limitations of MCSS, it significantly improves upon DV* In MuloCo tasks where complex decision making is not required, diffusion planners generally underperform compared to diffusion policies. Although PG also exhibits relatively lower performance in these tasks, it achieves the highest scores among diffusion planners. In long-horizon tasks such as Kitchen, AntMaze, and Maze2D, PG substantially outperforms all baselines across most tasks.
4.2 Analysis
Visualization on Latent Trajectories We visualize the latent noise and corresponding trajectories from the behavior diffusion model and PG to verify whether the learned prior Pe in PG accurately captures the noise Xy that is denoised into high-value trajectories Xg. As shown in Figure 6 the prior of the behavior diffusion model spans a wide range values, whereas the learned prior of PG predominantly samples noise that leads to high-value trajectories. This demonstrates that the latent space of the learned behavior diffusion model is structured sufficiently well to cluster high- value trajectories under simple Gaussian prior. Although the target prior in PG remains close to

---

the standard normal distribution due to behavior regularization, it still captures mostly high-value trajectories.
Consequence of Using Latent-Space Value Function To avoid backpropagation through the de- noising process, PG trains a latent-space value function V directly on the prior noise x 4 Figure 7 presents our empirical evaluation of this design choice. In particular, Figure 77 depicts the correlation between learned latent values (xT) and ground-truth values V (xxo) for the walker2d-medi and hopper-mede um-v2 tasks. While a strong positive correlation is observed, a perfect learning of the mapping would yield correlation of exactly 1. The observed deviation reflects approximation errors, which could potentially affect performance when relying on the latent value function. However, our experiments show that these approximation errors do not lead to actual performance degradation. As shown in Figure 7b, PG achieves nearly identical normalized scores on the D4RL, benchmark with and without backpropagation through the denoising process, confirming that training V in latent space is sufficient. Full results corresponding to Figure 7b are provided in the Appendix G
More Expressive Target Prior To investigate the flexibility and representational capacity of PG, we extend our approach use multi- modal Gaussian mixture prior in place of the standard uni-modal Gaussian prior. This mod- ification allows us to evaluate whether a more expressive prior distribution can yield perfor- mance gains. As shown in Table 2 the results are domain-depender
Table Average normalized scores of uni-modal and multi-modal Gaussian prior on MuJoCo and Antmaze tasks.
Dataset
uni-modal
multi-modal
MuJoCo
84.61:..2
82.4:10.4
Antmaze
83.4
85.8:10.6
MuloCo tasks, the uni-modal Gaussian prior slightly outperforms the multi- modal variant, whereas in AntMaze tasks, the multi-modal prior provides marginal improvements. These findings suggest that the benefit of more expressive prior more pronounced complex, long-horizon environments such as AntMaze, while in simpler control tasks like MuJoCo, the latent space found by a diffusion model is sufficiently well structured to capture high-value trajectories only with a uni-modal gaussian distribution. Full results corresponding Table 2 are provided in the Appendix G
4.3 Ablation Studies
Regularization Function / By deriving a closed-form solution for behavior regularization, PG accommodates variety of regularization functions f. To assess the impact of different choices of f. we performed an ablation study on the D4RL benchmark. We compared fz) log (KL-
divergence), f(x) log x (reverse KL-divergence),, and f(x) (x-1)² (Pearson x² -iveeeerge In practice, we use KL-divergence in PG. Ablation results are provided in the Appendix H
Behavior Regularization Coefficient We present ablation studies evaluating the performance of PG across different values of the behavior regularization coefficient a. In practice, we sweep (50.0, 10.0, 1.0, 0.1, 0.01, 0.001) in PG. Ablation results are provided in the Appendix H
5 Conclusion
We introduced Prior Guidance, a guided sampling method for diffusion planner- based offline RL that replaces the standard Gaussian prior with a learnable, behavior-regularized distribution. Unlike previous guided sampling methods, PG avoids inference-tine overhead, reduces the risk of out- of-distribution actions, and enables stable behavior regularization in closed form. Empirically, it achieves strong performance across diverse tasks, especially in long-horizon domains.
Limitations Despite its effectiveness, PG has several limitations. First, it introduces additional complexity through the training of both a latent value function and a learnable prior. Second, the architectural choices for the prior network have not been thoroughly investigated and warrant deeper exploration. Third, our evaluation limited to offline settings and does not consider high-dimensional observations like images. Finally, the bijective mapping between latent noise and trajectory assumed by DDIM does not hold in practice due to limited discretization steps; this could be addressed by using flow-man mow-matching [35] 36), which we leave for future work.

---

6
Acknowledgements
This work was supported by the Inst(tst of Information Coummunications Technology Plan- ning & Evaluation)-ITRR (Traaoooonn Technology Research 200-r111T--222---20222----- grant funded by the Korea govemment(MMnnitry of Science and ICT) and by the IITP under the Artificial Intelligence Star Fellowship support program to nurture the best talents (IITP-2025-R5 2025-02304828) grant funded by the Korea goVer
References
[1] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0164. 2020.
[2] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11366. 2019.
[3] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-201455 2021.
[4] Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xiamyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. arXiv preprint arXiv:2303. 15810, 2023.
[5] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[6] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances neural information processing systems, 34:8780-8794 8794, 2021.
[7] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34: 119111999991 1999-17993, 2021.
[8] Xiang Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion-Im improves controllable text generation. Advances in neural information processing systems, 35:4328-4344 2022.
[9] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8646, 2022.
[10] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Text2video-zere Text-to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15954-15964, 2023.
[11] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprini 2rXii:2205.09991, 2022.
[12] Anurag Ajay, Yilun Du Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-mant making?? arXiv preprint arXiv:2211.15557. 2022.
[13] Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdif- fuser: Diffusion models as adaptive self-evolving planners. arXiv preprint arXiv:2302.01877, 2023.
[14] Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. In International Conference on Machine Learning, pages 20035-20064 PMLR, 2023.
[15] Chang Chen, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchi- cal planning with diffusion. arXiv preprint arXiv:2401.02644, 2024.
10

---

[16] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. 125555 2022.
arXiv preprint
[17] Liyunn Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang, and Amy Zhang. Diffusion-dice: In- sample diffusion guidance for offline reinforcement learning, arXiv preprint urXiv:2407.20109, 2024.
[18] Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idgl: Implicit g-learning an actor-critic method with diffusion policies. arXiv preprint WrXiv:230411057: 2023.
[19] Haofei Lu, Dongqi Han, Yifei Shen, and Dongsheng Li. What makes good diffusion planner for decision making? In The Thirteenth International Conference on Learning Representations. 2025.
[20] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06/93, 2022.
[21] Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning. arXiv preprint arXiv:2309. 2023.
[22] Suzan Ece Ada, Erhan Oztop, and Emre Ugur. Diffusion policies for out-of-distribution gener- alization in offline reinforcement learning. IEEE Robotics and Automation Letters, 9(4):3116 3123, 2024.
[23] Ruoqi Zhang, Ziwei Luo, Jens Sjölund, Thomas Schön, and Per Mattsson. reguaiz-re diffusion policy with q-ensembles for offline reinforcement learning. Advances in Neural Information Processing Systems, 37:98871 18897, 2024.
[24] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint wrXiv22004002219 2020.
[25] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
[26] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning, arXiv preprint arXiv:1910.00117, 2019.
[27] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.
[28] Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimization through diffusion behavior. arXiv preprint arXiv22310007297, 2023.
[29] Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209. 14548, 2022.
[30] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks sequence modeling. arXiv preprint arXiv: 1412.355. 2014.
[31] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 37:1 1191, 2020.
[32] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning arXiv preprint arXiv:21 10.0666,, 2021.
[33] Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning, pages 2282522285, PMLR, 2023.
11

---

[34] Ian T Jolliffe and Jorge Cadima. Principal component analysis: review and recent devel- opments. Philosophical transactions of the royal society A: Mathematical, Physical and Engineering Sciences, 374(2065):20110200) 2016.
[35] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
[36] Seohong Park, Qiyang Li, and Sergey Levine. arXiv:2502.02538,, 2025.
Flow q-learning.
arXiv preprint
[37] Dhruva Tirumala, Alexandre Galashov, Hyeonwoo Noh, Leonard Hasenelever, Razvan Pascanu, Jonathan Schwarz, Guillaume Desjardins, Wojeiech Marian Czarnecki, Arun Ahuja, Yee Whye Teh, et al. Behavior priors for efficient reinforcement learning. Journal of Machine Learning Research, 23(221): 2022.
[38] Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pages 188-204. PMLR, 2021.
[39] Lucy Xiaoyang Shi, Joseph J Lim, and Youngwoon Lee. Skill-based model-basee reinforcement learning. arXiv preprint arXiv:2207.07560, 2022.
[40] Kourosh Hakhamaneshi, Ruihan Zhao, Albert Zhan, Pieter Abbeel, and Michael Laskin. Hi- erarchical few-shot imitation with skill transition models. arXiv preprint arXiv:2107.08981, 2021.
[41] Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Of- fline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010. 1361 2020.
[42] Myunsoo Kim, Hayeong Lee, Seong-Wooe Shim, JunHo Seo, and Byung-Jun Lee. Nbdi: simple and efficient termination condition for skill extraction from task-agnostic demonstrations. arXiv preprint arXiv:2501. 12668, 2025.
[43] Sang-gil Lee, Heeseung Kim, Chachun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, and Tie- Yan Liu. Priorgrad: Improving conditional denoising diffusion models with data-dependent adaptive prior. arXiv preprint arXiv: 066006 2021.
[44] Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pages 6120-6 6130. PMLR, 2021.
[45] Woosung Kim, Donghyeon Ki, and Byung-Jun Lee. Relaxed stationary distribution correction estimation for improved offline policy optimization. In Proceedings of the AAAI Conference Artificial Intelligence, pages 13185-13192, 2024.
[46] Myunsoo Kim, Donghyeon Ki, Seong- Shim, and Byung-Jun Lee. Adaptive non-uniform timestep sampling for diffusion model training. arXiv preprint arXiv:2411.09999, 2024.
[47] Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning. arXiv preprint arXiv:2506.1570 2025.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neurul information processing systems, 30, 2017.
[49] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05101, 2017.
[50] Diederik P Kingma. Adam: method for stochastic optimization. arXiv preprint 112.6014 2014.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in the Appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section Appendix
C Appendix
D
Appendix
F Appendix
G
Appendix H
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification: Abstract, Section Appendix Guidelines:
C
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cpippi.ccppullc///uuui wwwpc//ggue////eeeeeoooc CoodeSubeiiieeee ssi asionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendix Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code. in the Appendix, or as supplemen- tal material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Appendix
Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
15

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix C.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpp//pp.....pp......ccccccaaea
Answer:
[Yes]
Justification:
We have reviewed the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper does not have societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
16

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not release new datasets models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Section Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee wwworeewwthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
17

---

Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
18

---

Justification: Core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non- stanssadd components.
Please refer to our LLM policy hattps://neuriss.. /20 for what should or should be described.
19

---

A Related Works
Learning Priors in RL Several prior works in reinforcement learning have utilized trainable prior distributions to achieve great success in multi-task learning [37 and skill learning 38 39
42]
37 introduce a behavior prior framework that learns distribution over trajectories to encode reusable movement and interaction patterns across tasks. [8 propose SPiRL, which leverages a learned skill prior to guide exploration and improve downstream task performance. [41] propose OPAL, which learns a latent prior over temporally extended primitive behaviors from offline data to improve downstream task performance in offline RI. 40 propose FIST, which learns future- conditioned skill prior from offline data and adapts it for few-shot imitation of unseen long- morizoo tasks. (39] propose SkiMo, which jointly learns a skill prior and a skill dynamics model from offline data to enable accurate long- Long-horizon planning and improve downstream task performance. [2] propose NBDI, which learns a termination prior from a state-action novelty module to guide the execution of variable-length skills in downstream tasks.
While these methods learn priors over abstract skills or trajectories, they do not operate in the context of diffusion models. To the best of our knowledge, Prior Guidance (PG) is the first to replace the standard normal prior distribution of a diffusion model with learnable prior in the context of reinforcement learning.
Learning Priors in Diffusion Models Recent studies have explored methods to improve diffusion models by replacing the standard Gaussian prior with a more informative distribution. Notably, PriorGrad [43] proposes data-dependent adaptive prior to accelerate training and improve sample quality in conditional generative tasks such as speech synthesis. While PriorGrad shares the motivation of improving the expressiveness and utility of the diffusion model's 's prior distribution, PriorGrad applies prior learning to the forward process in conditional generative models and leverages exact statistics from input conditions to define an instance-spectin learns parameterized prior distribution over the latent space of hehavior-cloned diffusion model, specifically targeting the reverse denoising process for high-return trajectory generation in offline RL.
non-lcarne prior. In contrast, PG
Behavior-Regulariied Regularized Objectives in Offline RL To mitigate policy degradation caused by over- estimated value functions, a wide range of offline RL methods incorporate behavior reeaaioorreullrizzedd objectives. [2 introduce BRAC, unified framework that employs behavior regularization to stabilize policy learning. (26) propose Srrattgg--Wighhee Regression (AWR), which formulates offline RI. as supervised regression weighted by advantage estimates. [27] present AWAC, actor-critic method that performs advantage-weighter supervised learning, enabling efficient offline pretraining and robust online fine-tuning. [44) propose OptiDICE, which optimizes behavior-regularized objec- tive over stationary distributions, avoiding the need for value bootstrapping or policy gradients. 4 introduce SQL and EQL, which leverage in-sample value regularization to address out-of-distributin (OOD) actions. (28) present SRPO, which directly regularizes the policy gradient using the score function of a pretrained diffusion model, thereby enabling sehaviorrreuulaiize learning without sampling. 45 propose POReIDICE, which relaxes the positivity constraint in f-divergence regular- ization to better handle low-advantage samples. However, integrating behavior-regularived objectives into diffusion planners poses several challenges, such as requiring backpropagation through the iterative denoising process and making it difficult to handle a wide range of regularization functions in closed form.
Diffusion Models in Offline RL Inspired by the success of diffusion models across a variety of domains [5 6 7 offline RL. One line of research replaces conventional uni-modal Gaussian policies with expressive liffusion-based policies. For example, (20] propose Diffusion-QLL which combines conditional diffusion modeling with Q-learning guidance to improve policy expressiveness and regularization. [8] reinterpret IQL as actor-critic algorithm and propose IDQL, which couples generalized IQL critic with a diffusion- fffuee behavior model. [33] introduce QGPO, which enables exact energy- guided diffusion sampling by aligning sampled trajectories with the energy landscape defined by f-function.
8, 9 10, 46), there has been growing interest in adapting diffusion models to
Another line of work leverages the temporal modeling capabilities of diffusion models to develop trajectory-level planners for long-horizon tasks. [I]] introduce Diffuser, which formulates planning as conditional trajectory generation and enables flexible behavior synthesis via classifier guided
20

---

sampling. [12] propose Decision Diffuser, which formulates decision mrriiinnnmmking as return- cruui-nonddtioeed generative modeling without relying on value estimation. [3] present AdaptDiffuser, self-evolving planner that iteratively improves its performance through reward- reward-guided trajectory generation and model fine-tuning. [14] propose HDMI, hierarchical diffusion model that generates subgoals condi- tioned on returns and performs trajectory denoising toward each subgoal. [15] propose Hierarchical Diffuser, a two-level architecture that combines jumpy subgoal generation with fine- sine-grained trajectory refinement. Finally, [19] conduct comprehensive empirical study of diffusion planners and introduce Diffussinn Veteran, a simple baseline derived from key design principles such as sampling strategy, backbone architecture, and action generation scheme. However, existing guided sampling methods in diffusion planners suffer from several limitations- such as suboptimal sample selection, distributional shift, and high computational cost hightighhgi the need for novel guidance approaches that can more effectively balance performance and efficiency.
Concurrent Works A concurrent work, DSRL 47, introduces a method that steers behavior- cloned diffusion policy by optimizing a learnable prior through reinforcement learning. While this idea of guiding generative models via learned prior shares the same high-level goal our approach, there exist both conceptual similarities and key methodological differences between DSRL and our PG framework.
Both DSRL, and PG enhance decision quality by learning a prior distribution over the noise space (XT (O, I)) that serves input to the diffusion model's denoising process. In both methods, the prior is trained to maximize a value function defined over x7, which estimates the value of the corresponding denoised sample Xg- This shared mechanism allows both methods to bias sampling toward high-value regions of the data or trajectory space.
However, there are fundamental differences. DSRL adopts a diffusion policy that directly maps noise individual actions, whereas PG employs the diffusion planner that produces entire state trajectories and subsequently reconstructs executable actions via an inverse dynamics model. This trajectory-Ievel formulation makes PG particularly effective long-horizon or sparse-reward environments, where modeling temporal dependencies is critical for performance. Moreover, PG provides a theoretical foundation absent in DSRL. Specifically, we prove in Proposition [ that, under sufficiently fine discretization, the optimization objective for the diffusion model can be equivalently reformulated as an optimization over the latent prior distribution. This equivalence arises from the bijective nature of the DDIM sampling process, which allows for an exact mapping between the data space and the latent prior. Finally, PG explicitly addresses the challenge of samples in offline reinforcement learning. While DSRL assumes the diffusion model generates only in-distribution actions and omits behavior constraints, our experiments (Figures 3 5)) reveal that diffusion models trained offline datasets still produce OOD samples that harm performance. To mitigate this issue, PG introduces behavior regularization by imposing an f-divergence penalty between the learned prior and the standard normal distribution. This explicitly penalizes deviation from the behavior distribution and induces regularization at the prior level.
(OOD)
21

---

B Proof of Proposition 1
Proposition Assume that DDIM sampling employs sufficiently large number of discretization steps ensuring that the mapping Xo 9x(x)) is bijective. If the target trajectory density Te parametrized by placing the prior P=(Xr]s) while keeping the denoising process Is fixed, the Sehavior-regularizze objective 2 is equivalent to:
max E D. xy (pb V (gs (xT)) af (Pu(X)l5) )
(3)
p(XT)
Proof. Since X = Is (xr) bijective mapping between Xy and Xo. we can apply the change-of-
variables formula:
dxg dxr det (dga(xT)
dxT
Using this, we compute the conditional density म (xols) as follows:
T s)
p(xr) -8(x0 Is (xx dxxr
xT
(8gs(xx)
p(XT) 5(x0 gss (xxx)) det
ÖxT
(gs((x))
p(XT) det
OXT
Similarly, the density To (xo (s) can be obtained in the same way:
Toe (xols) Px(xT(s) -(x0 (x(xx))d
(0gx(xT)
(x(x))s) 8(x0- = Is (s(xx)))
|det
txr
det (xg)(x))
(xT(s)
OxT
Accordingly, we can reformulate the objective
2 as follows:
Tos (Xu s))
max E D.
V(xo) - -aʃ
(xo
Pe (xT(s) det
Bgg (xT) dxr
max E
(gx(x))) - af
p(xr) /dot (99 Bxr
(xT(s)))
max F Da
V (xx(x))) - -af
p(xr)
22

---

C Experimental Details
C.1 Experiment Details of Figure
In Figure we introduce a new baseline referred as DQL (planner), which directly trains the diffusion model using behavior-regularized ob- Specifically, it employs the diffusion behavior cloning loss (1) as surrogate for be- havior regularization, resulting in the following objective:
Algorithm 2 DQL (planner)
Input: Dataset D, Hyperparameter a Require: Inverse Dynamics ew, Critic V Initialize: Planner
jective 2
1: // Training
2: for each iteration do
3: Sample D and p(xr)
max B [V(x))] (L()) (6)
Update planner सe using Eq. (6
5: end for
6: // Execution
where ((-) denotes the behavior cloning loss
7: s env.reset()
defined in Eq. This approach be seen
8: for environment step =0 l,... do
as the planner version of Diffusion Q-Learning
Sample X Tpl
(DQL) and refer it DQL (plan- ner) throughout the paper. The corresponding pseudo-code is provided in Algorithm 2 We measure the training time of DQL (planner) af- ter aligning all other settings with those of Prior Guidance ensure fair comparison.
10:
Predict action a using cw (XD)
11:
s(*+1) env.step(a
12:
end for
In Figure we compare training time of with and without backpropagation through the denoising process to specifically assess the efficiency of learning the latent value function. To this end, we exclude the time required to train the planner 9s. inverse dynamics model e and critic V. report only
which are shared across both variants in Algorithm [I] Likewise. for DQL (planner), the time taken to optimize the diffusion model for value maximization, excluding the time for training the inverse dynamies, critic, and the behavior cloning boss-which aligns with the planner training time in PG. This setup enables us to assess how much training time can be saved through latent value function learning.
C.2 Experimental Settings
We evaluate Prior Guidance on the D4RL offline RL benchmark. For MuJoCo tasks, we report the average performance over 10 evaluation trajectories for each of 5 independently trained models. For Kitchen, AntMaze, and Maze2D tasks, we average over 100 evaluation trajectories for each independently trained models. In Prior Guidance, we adopt the same experimental settings for the planner Is, inverse dynamics model c.. and critic V as used in Diffusion Veteran (DV) 19 We conducted all experiments using four NVIDIA RTX 4090 GPUs.
C.3 Statistical Details
For all experiments, we report the mean and standard error of normalized scores computed over training seeds. Additionally, in Figure Figure 7b and Table report the mean and standard error Oavg of average normalized scores across multiple environments. The standard error Gag is computed using the error propagation formula based the per-environment standard errors o,, each estimated from training seeds. Specifically.
Gavg
Σ
0.
where ō; =
Vi
with denoting the number of training seeds, the number environments, and 0; the standard deviation for environment i.
23

---

D Diffusion Veteran (DV)
The pseudo-code for Diffusion Veteran (DV), which is identical to that of DV*, is provided in Algorithm
Algorithm 3 Diffusion Veteran (DV)
Input: Dataset D, Planning horizon H, Planning stride m, Candidate num N Initialize: Planner Is. Inverse Dynamics cu. Critic V
I: Compute discounted returns: R(r) -h= (((rhh)) for every step
2: // /rainingg
3: for each iteration do
4: Sample s(7), ((7))((mm))) s(7+m),
(1++(H-1))m) and alr) R((T) from
5: Update planner 9s using Eq. ( with g(T) as imput, s(T), ..... g(T as target output
6: Update inverse dynamics eu using Eq. (T) with s(7), s((mm)) as input, a(r) as target output
7: Update critie V using MSE Loss with ...,,.,,,(T....... (H as input, R(T) target output
8: end for
9: // Exeectiio (MCSS)
10: (0)) = env.reset()
11: for environment step 0, I,... do
12:
Randomly generate N candidate trajectories using Is while fixing the first state as s(T)
13: Select the trajectory with the highest estimated value V among the candidates
Generate (7) from the best trajectory using the inverse dynamics model ew g(r+1) env.step(a(T))
16:
end for
Full results corresponding to Figure along with the comparison between the original implementation of DV and Prior Guidance, are provided in Table
Table 3: Normalized scores on the D4RL benchmark. we compare Prior Guidance (PG) with the original implementation DV and DV* under different Ns. We compute the mean and the standard error over random seeds.
N
5
10
50 (default)
500
50
walker2d-medim--2
70.8 2.2
81.1=022
80.6 0.3
79.5 0.3
79.5
0.7
82.8 0.5
82.3 0.2
40.3
0,9
84.1
83.5 2.5
8
8..0
1.0
2.11
1.8.9
109.0
109.0 0000000 0.1
108.9 + 0.2
109.2 0.0
+.1
hopper- medium--2
0.9
67.4=88
3.51
2..1
83.6
hopper medium replay-v2 v2
2..4
91.2 0.2
91.0111..11
9..3
91.3
91.9
91.3 0.3
hopper medium expert-v2
76.2
110.2+00.55
109.9
halfcheetah- medium-v2
4.6
47.5
0.0
5..9
0.1
50.4
45.6 0.5
halfcheetah medium replay-v2
46.5
0.4
wwwfchhetah-meddum--ove
82.3 + 1.4
91.8
91.6 1.2
92.3 +
87.
92.7 0.3
95.2 = 0.1
Average
60.2
80.4
81.2
83.0
83.8
83.5
84.6
kitchen-mixed-v0
$2.4 0.7
65.1 = 1.2
70.3
23.3
67.5 3.0
73.6 0.1
74.6 0.4
kitchen-partial-O0
53.4
77.7 2.0
80.6 1.8
76.2 4.3
66.8 J. 5.4
94.0 - 0.3
88.0 1: 3.1
Average
52.9
71.4
75.5
74.8
67.2
83.8
81.3
66.6
86.4
86.0
80.8
62.2 2 3.2
89.0 = 1.6
87.8 = 1.5
medium diverss----
63.6 1:::::200
72.4
2.4
71.8 4.0
82.0 3.6
84.6 1: 1.6
87.4 1: 1.6
87.3 1: 1.7
antmaze large play-v2
74.8 1.3
79.2
2.0
80.6
80.8 4: 2.8
81.0 4: 2.0
76.4 2.0
82.4 4 2.1
humuze-large-divenes
2.6
78.2
77.6
76.0
72.0
80.0 1.8
76.0 1.9
Average
71.0
79.1
79.0
79.9
75.0
83.2
83.4
maxeld umaze-vl
41.5 3.4
97.1
111.0 4: 2.7
133,8 :: 1.9
139.7 4: 1.5
136.6 1.3
139.2
maze2u-medium-vl
23.0
95.3
123.8
144.1 = 2.1
151.2 2.6
150.7
1.0
159.5 0.8
maze2d-largo-il
54.6
183.9 4: 5.3
190.8 + 5.0
197.6 = 4,2
201.8 + 5.5
203.6 = 1.4
195.2 2 2.3
Average
39.7
125.4
141.9
158.5
164.2
163.6
E Full results of Table
24

---

PG
82.320.2
83.7.1.0
109440.1
97.3
91.3.03
110-44011
45,6.1,5
46.4:00.4
95.2110.
84.6
74.620..4
88.0.13.1
81.3
87.8+1.5
87.3111.7
#2.412.1
76.0
83.44
139.2:41.3
159.5.10.8
195.2+22.5
164.6
79.5
83.5
109.0=0.1
84.142.
91.340.1
109.9110
50,94011
46.440.2
92.310066
73.3+20.6
76.2:14.
80,8+1.
82.013366
80.8+2288
76.0+22.3
133,8=1.99
144.112.1
99.6644.
DV
83.0
74.8
79.9
158.5
Table 4: Normalized scores on the D4RL benehmark. We computc the mean and standard error over 5 random seeds.
84.0 84.0200.6
84 1.2.2
107 10.
99.31103
94.7
115,3+1-1
46,7 46.7+02
38. 1-007
92.5.00.3
71.7+2.7
73.3114
88.718.1
83.625.8
12.
135.613.
155.8=2.5
DD Diffusion planners
HD
84.6
72.5
139.9
82.5+14
75.014.3
108.&+1.7
79 79.333.6
100.0
111.&11
49 19.1110
39.3+4.1
90.611..
65.0428
57.012.5
8.0=-4.6
4.0.2..8
0.0+00.0
0.0:0..0
81.8
61.0
3.0
AD
84.422.
84.71311
108 8-
96.612..7
92.21:1.
11166=2.0
44.21:06
38 28.3+00.9
89.6.1088
83.3
51.8+0088
55.5
53.7
12.0=7.5
6.0.3333
5.343.4
8.742255
8.0
135. 15..115.8
167.9=5.0
144.3
129
Difftser
79.62:066
70.611.6
106.9=0.2
74.31:4
93.6224
103.3+1.3
42.8--03
37,7+-0.5
88.910.3
77.5
52.542.
35.711.33
54.1
6.74-5.7
2.016
17.3+
27.3
13.3
113.9-31
121.5
123.0=6.4
119.5
SRPO
84.421.8
84.612.9
114.0=0099
95.5=10.8
101.2::04
100.1+5.77
60.420.3
51.4+11.4
92.21112
80.7=2.9
75.0
53.6+5.1
53.6#2.6
87.1
65.7
QGPO
86.0=...7
84.4 84.414.1
110.7+0.6
98.0+2.66
96.9426
108 00..5-2.55
54.11104
47.6+144
93.510..3
13.64444
83.8
66.6-99.8
64.85555
86.6
74.7
Diffusion policies
IDQL
82.5
85.1
112.7
65.4
92.1
108.6
51.0
45.9
95.9
82.1
66.5
66.7
66.6
84.5
84.8
63.5
67.9
75.2
57.9
79.2
DQL
87.0 $7.0=20.9
95.5.11.5
110. 100.1+0.3
90.5144.6
101.3:10.6
111.1+1.3
$1.1200
47.8+
96.
62.6:t5.1
60.5116.9
61.6
76.6=10.88
18.6
4..4
56.617..6
64.6
SBBC
77.9=2.5
65.1 15.16.666
109 0.2
57.144.1
86 06229911
108 108.6=2211
45.912.2
97.1+1177
92.6.1..5
73.6
45.4211.
47.914.11
46.7
81.3+2.6
82.013.1
59.3:+14.3
6..5
67.0
73.9-66.
7..4+
74.0
Gaussian policies
IQL
78.3
73.9
109.6
66.3
94.7
91.5
47.4
44.2
86.7
77.0
51.0
46.3
48
71.2
70.0
39.6
47.5
57.1
47.4
34.9
58.6
47.0
CQL
79.2
26.7
111.0
5800
48.6
98.7
44.4
46.2
62.4
63.9
51.0
49.8
50.4
14.9
15.8
53.7
61.2
36.4
5.7
5.0
12.5
7.7
BC
6.6
11.8
6.4
29.0
11.3
111.9
35.8
31.9
47.5
33.8
40.7
0.0
0.0
0.0
0.0
0.0
3.8
13.0
Dafaset
Walker2d-M
M-lker2d-M-R
Walker2d-M-F
Hopper-M
Hopper-M-R
Hopper-M-EE
Half HallCheetah-M
HalFCheetah-M-F
Hal/Cheettah-ME
Average
Kitchen-M
Kitchen-P
Average
M-P
Antmaze-Me M.D
Antmaze- ---
Antmaze L-D
Maze2D-U
Maze2D-M
Maze2D-L
Average
25

---

Hyperparameters
This section outlines the hyperparameters used train Prior Guidance. The planner 9., inverse dynamics model Cw. and critic V all follow the same settings as those used in Diffusion Veteran. Detailed hyperparameter settings are provided in T
Table 5: Hyperparameter settings used for Prior Guidance on the D4RL benchmark.
Settings
MuJoCo
Kitchen
Antmaze
Maze2d
Batch Size
128
128
128
128
State-Action Generation
Separate
Separate
Separate
Separate
Advantage Weighting
True
False
False
False
Inverse Dynamic
Diffusion
Diffusion
Diffusion
Diffusion
Time Credit Assignment
0.997
0.997
1.0
1.0
Planner Net. Backbone
Transformer
Transformer
Transformer
Transformer
48
Transformer Hidden
256
256
256
256
Transformer Block
2
2
2
8
Planner Solver
DDIM
DDIM
DDIM
DDIM
25
Planner Sampling Steps
20
20
20
20
Planner Training Steps
1M
1M
1M
1M
Planner Temperature
I
Planner Learning Rate
2e-4
2e-4
2e-4
2e-4
Planner Optimizer
Adam
AdamW
AdamW
AdamW
44
Planning Horizon
4
32
32
40
Planning Stride
I
1
15
25
Inverse Dynamics Net. Backbone
MLP
MLP
MLP
MLP
Inverse Dynamics Hidden
256
256
256
256
Inverse Dynamics Solver
DDPM
DDPM
DDPM
DDPM
Inverse Dynamics Sampling Steps
10
10
10
10
Inverse Dynamics Training Steps
IM
IM
IM
1M
Policy Temperature
0.5
0.5
0.5
0.5
Policy Learning Rate
3e-4
3e-4
3e-4
3e-4
Policy Optimizer
AdamW
AdamW
AdamW
AdamW
Value Learning Rate
3e-4
3e-4
3e-4
3e-4
Value Net. Backbone
Transformer
Transformer
Transformer
Transformer
Value Optimizer
Adam
Adam
Adam
Adam [50]
Prior Network
GRU
GRU
GRU
GRU 30
Prior Learning Rate
3e-4
3e-4
3e-4
3e-4
Prior Optimizer
AdamW
AdamW
AdamW
AdamW
Prior Hidden
256
256
256
256
Latent Value Learning Rate
3c-4
3c-4
3c-4
3c-4
Latent Value Net. Backbone
Transformer
Transformer Transformer Transformer
Latent Value Optimizer
Adam
Adam
Adam
Adam
Behavior Regularization
KL
KL
KL
KL
26

---

In Table 15 we present ablation studies on the choice of behavior regularization function f. The
corresponding hyperparameter listed in Table and 7
Table a used for KL and Reverse KL-divergence on the D4RL benchmark.
KL
Reverse KI.
walker2d-medium-v2
0.1
0.1
0.01
0.01
walker2d-
0.01
0.01
hopper-meeiumm--2
0.01
0.01
hopper-mediuum-replay-22
0.01
0.001
hooper-meddum--experttv2
0.01
0.01
haffchhettah-mediun--22
0.001
0.0001
mallcheetal-mmdiiu--rlaaaa
0.001
0.001
malichettalmmmdduuneepprrrtzz
0.01
0.01
Kitchen-mixed-vv0
1.0
10.0
kitchen-partial-vo
1.0
10.0
antmaaze-mcdiuu-pplyy222
1.0
1.0
antmaaze-meediun-ddivress---
1.0
0.1
antmaze-large-play--2
50.0
1.0
antmaze-large--diersse-22
10.0
10.0
maze2d-umaze-vl
1.0
10.0
maze2d-medium-vl
0.1
1.0
maze2d-large-vl
0.1
1.0
Table 7: used for Pearson 2 divergencc the D4RL benchmark.
Pearson x²
walker2d-medium-
le-4
walker2d-mmeiuu--eplaaa
1e-7
walker2dd-medium-exprrrs
le-6
hopper-medium--v2
1e-8
hopper-mediumm-eelaa--22
le-6
hopper-mediium-exxerrt-2
le-5
Malffcheetah-mediume
le-4
malicheetahmmediu--rqpuee. medium-replay-vv
le-6
halfcheetah halfchhetah--meiummeeee -exiium-exper--22
le-8
kitchen kitchen-mixed-v0
lc-6
kitchen- -itchen-partial-vO
le-7
antmaze-medium- play-v22
le-4
antmaze-meciumm-divers.
le-2
antmaze-large-play-22
le-4
antmaze-large-divere--
le-2
maze2d-umaze-vl
le-7
maze2d-medium-
1e-8
maze2d-
le-6
27

---

In 2 we report the average normalized scores MuJoCo and AntMaze tasks when using multi-modal Gaussian prior. The corresponding hyperparameter listed in Table g
Table 8: a used for multi-modal Gaussian prior on the D4RL benchmark.
multi-modal
walker2ddmedium--22
0.01
walker2d-medium-eee
0.01
0.001
hopper-medium--2
0.01
hopper-meedium--eppay--22
0.0001
hoopes--mediuu--erppettt22
0.01
haffcheetah-mediiun-222
0.0001
halfeheerda-mmodiin-----y----2
0.001
halichetubb-mmdduun--xxx-----
0.1
antiaaze-medduum-play-222
0.1
attmaze-mncduun-diveresezz
0.1
antmaze-aagg-playyv2 antmaze-large
0.1
nutmaze-arge--divess--
1.0
G Extensive Results
Full results of Figure 7b be found in Table
Table 9: Comparison Prior Guidance with and without backpropagation through the denoising process (denoted as PG w/ Bp and PG w/o Bp, respectively) on the D4RL benchmark. We compute the mean and the standard error over 5 random seeds.
PG w/ Bp
PG w/o Bp
walker2d walkerrdd-mediiu---ellaay
81,310.2
82.3+0.2
83.242.1
83.7+1.0
walker2d wallker22-medium---pperr-t
109.6:0.22
109.440.1
mopper-medium-v2
96.9+114
97.5+0.6
hoppe--medium--eplaa--22
94.5+00.7
91.3+0.3
hopper-mmdduu--expprr---
110.4++0.1
110.4+0.1
haffchhetta--medium--22
54.7++0.2
45.6+ 0.5
malfcheetah-mmedim--eee
46.6+0..1
46.4+0..4
haffchecah-mmddium-expprryy
95.1+0.2
95.2+0.11
Average
85.8
84.6
kitchen-mixed-v0
7..
74.6+0.4
-itchen-partial-v0
87.6+4.88
88.043.1
Average
81.0
81.3
antmaze-mediimm
play-v2
87.0+1..5
1.5
suimazz-medium-diieee
89.6+1.8
87.3
antmaze-large-paay-v2
82.242.4
82.4+2.1
antmare-lanee-divrree-->
77.0+3.2
76.0
Average
84.0
83.4
maze2d-umazee
137.1
139.24 1.3
maze2d-meeium-vv
159.4+3..5
159.5+00.
maze2d-large-vl
196.2+2.6
195.242.3
Average
164.2
164.6
28

---

Full results of Table 2 can be found in Table T0
Table 10: Normalized scores of PG using multi-modal Gaussian prior on MuloCo and Antmaze tasks. We compute the mean and the standard error over 5 random seeds.
multi-modal
walker24-medium--22 walker2d-meeiiu--eellay
81.3++0.2
82.5+0.22
walker2dd-mciiu--xppr-ee
108.7=0.1
hopper-mecium--22
88.9122.
hopper-mediuum-reelayy-22
96.2100.0
hooper-mediuum-expeett22
110.5
salffcheetth-meddune
44.3+2.1
halfcheetah medium-replay-ee
seffcheetah-medim---ee
43.0+0.2
halfcheetah soschhetth-mmddiu---pppree
84.9+0.5
Average
82.3
antmaze-meddiumpplaee
8..
aatmaaze-mmciium-ddiees----
93.0+1.
antmazz-large--laa-->
83.4+1.4
antmaze-lare--divereev-
79.4+1.0
Average
85.8
H
Ablation Stuides
We conducted ablation studies on the behavior regularization coefficient a of PG across MuJoCo, Kitchen, AntMaze, and Maze2D tasks. The normalized scores of PG for different values of reported in Table T1 12 13 and 14
Table 11: Ablation study of in MuloCo tasks. We compute the mean and the standard error over 5 random seeds.
0.001
0.01
0.1
1.0
10.0
walker2d-medium--v2 77.2+1..6
81.4
82.3
74.5-10.8
70.8+12.2
walkerrd-mmdiuu--epla----
61.9+110
83.7+11.2
78.3
52.3+1155
10.6+
109.440.1
109.1:0.1
108.6:0.2
108.4:0.1
108.5+0.2
2opper-medium-v2
93.5=0.4
97.5+0.6
68.910.8
54.310.5
51.420.9
hopper-medium-
menneddu---rppa--/./ replay-v2
91.3-10.
90.8+0.77
89.8=1.2
61.410.3
34.8+0..8
110.440..1
110.340.1
110.3+10.3
87.010.9
1..
malfchhettah-mddinee
45.6=0.55
40.2+0.6
43.4-10.3
43.4+0.33
43.1+00.4
halfcheetth-meeium--eee replay--v2 v2
46.4+104
45.0+0.2
44.1+:0.4
43.0+0..5
39.0++0.5
malicheetah-mediu-eepprrre
8.2+1.1
95.2+0.1
94.9-10.2
95.1+0.2
92.8+0.44
Table 12: Ablation study a in Kitchen tasks. We compute the mean and the standard error over random seeds.
0.1
1.0
10.0
kitehen-mixed-v0 kitchen-pare Dittenn-artial--0
52.4+13.5
74.6
72.5+0.77
67.8
88.043.1
82.1
29

---

Table 13: Ablation study of in Antmaze tasks. We compute the mean and the standard error over random seeds.
a
0.1
1.0
5.0
10.0
50.0
molmaaze-medimy
47.0
87.8
83.8+1.6
84.0=1.3
80.2
antmaze- ummmaz--medim-
86.0+11.4
87.3
80.3+18
72.1=2.5
68.4+2.8
antmaze- haaimazz--agge-plaa--aaaa
34.5
74.6#2.5
76.8+2.0
82.4#2.1 71.3
antmare-large-diivrse--V 21.8++3.2
70.4
69.7+
76.0+1199
68.5+
Table 14: Ablation study of in Maze2D tasks. We compute the mean and the standard error over random seeds.
0.001
0.01
0.1
1.0
10.0
maze2d-umaze-vl 113.312.5
91.913.7
126.311.7
139.2.1..3
122.2=2.1
maze2d-medium--v
155.6=00.
121.1
159.5=0.8
152.1=1.2 1.8.4+1.7
maze2d-large-vl
143.4#5.1
151.344.7
195.2+2.3
185.0+5..
183.9
We also conducted ablation studies on various behavior regularization functions f. as shown in Table
Table 15: Comparison of normalized scores of Prior Guidance across four behavior regularization functions KL, Reverse KL, and Pearson x2 x the D4RL benchmark. We compute the mean and the standard error over 5 random seeds.
KI
Reverse KI.
Pearson-A
walker2d-medium--2
82.3+0.2
82.0+0033
1..
walkerr2d-meduun--rppaa--2
83.7++1.0
83.7+0..4
82.4+-0.3
walker2o-mccduum--xpprt---
109.4:0..1
109.110.0
103.540.2
hopper-medium--22
97.5++0.6
96.5
97.0+10.1
hoppee-mcdium--relaa--22
91.3400.3
93.5+0.00
92.410.00
hopper- wwaere--miiiumeeeeeeeee
110.4+0.1
110.3+0.0
111.5+0.0
halfcheetah-mediiu-- medium-v2
45.6 0.5
48.8
43.5
halfcheetah medium medium-replay replay-v2
46.4+0.4
46.0+0.1
44.941.1
malfcheetah-medium-co -
95.2
95,4
92.5+0.1
Average
84.6
85.0
82.9
kixchen-mixed-v0
74.6:t0.4
73.3+0.3
71,210.3
sicche-partial-vee
88.043.1
85.3+1.0
75.613.7
Average
81.3
79.3
73.4
autmaaze-medium-ppay-222
1..
87,612.22
76.3
antmaz--medduumddiversss-2
87.
86.3.2.0
80.0+2.3
antmaze-larg-plla--2
82,4+2.1
79.7
85.2
untmaze-large-diie
76.0 1.9
77.0
70.9
Average
83.4
82.7
78.1
maze2d-umaze-vl
139.2++ 1.3
136.2--2.1
132.0++2.2
maze2d-medium-vl
159.5400.
150.4+1..6
170.8
maze2d-large-vl
195.2+22.3
186.612.3
193,242.0
Average
164.6
157.7
165.3
30

---

I Why PG underperform on standard MuJoCo tasks?
Table 16: Performance comparison across standard MuJoCo tasks.
DQL
DV*
PG (planner)
PG (policy)
walker2d-
87.0
79.5
82.3
84.5 + 0.26
95.5
83.5
83.7
96.4
110.1
109.0
109.4
109.8 0.23
hopper-mediun--2
90.5
84.1
97.5
+
hopper-meddium-repllyyyy
101.3
91.3
91.3
100.5 1.27
hopper-medduu--xxpertty22
111.1
109.9
110.4
111.4 0.57
halfcheectah-mccium-222
51.1
50.9
45.6
54.7 + 0.42
hallcchecaa-mmcdiuu---rppa--y
47.8
46.4
46.4
48.6 + 0.34
halfcheetah--mediuneee malicheeta--mmdiuuneeeerrryy
96.8
92.3
95.2
95.5 + 0.26
Average
87.9
83.0
84.6
88.9
PG is a diffusion planner-based method designed for long-horizon planning, where capturing tempo- rally extended dependencies essential. However, MuJoCo locomotion tasks involve short-horizon, dense-reward control focused on making agents run faster, which does not require such lookahead capabilities [19]. As result, diffusion planner-based methods, including PG, often underperform compared to diffusion policy-based approaches in these settings, as shown in Table [ While the additional structure and trajectory-level modeling of planners benefit challenging long-horizon do- mains such as AntMaze, Kitchen, or Maze2D, they provide little advantage and can even be slightly detrimental in simpler environments where policies can already achieve near-optimal performance.
To examine whether the observed performance gap in MuJoCo tasks from the use of the diffusion planner rather than limitation of the PG framework itself, conduct additional experiment by applying PG to diffusion policy. Specifically, we replaced the standard Gaussian prior of behavior-cloned diffusion policy with learnable, echavior-regularize prior, following the same training procedure used in PG.
Table 16 shows that replacing the diffusion planner in PG with diffusion policy (PG policy) consistently improved performance across MuJoCo tasks, raising the average score from 84.6 (PG planner) to 88.9 and surpassing the DQL baseline score of 87.9. This demonstrates that PG remains effective even in short-horizon environments, suggesting that the lower performance of PG in MuloCo tasks due the nature of trajectory-level planning rather than limitation of the method itself. The results for PG (policy) are averaged over 3 random seeds and reported as the mean d standard error.
31


Reviews:
[{"review_id": "duiooci9CX", "reviewer": "Reviewer_tREs", "summary": "This paper introduces Prior Guidance (PG) for diffusion planning for offline reinforcement learning. The idea is to steer a pretrained diffusion planner by learning a state-conditioned prior noise distribution that correspond to high-value trajectories. This avoids the need of multiple samples in Monte-Carlo search methods. Specifically, they parametrize the prior as a state-conditioned gaussian distribution, which is trained to maximize the (behavior) value function while staying close to the data distribution. By exploiting the bijective nature of the DDIM sampler and aliasing the sample-space value function to a noise-space value function, both objectives can be optimized in the prior space without backpropagating through the diffusion process. The paper evaluates PG on a suite of offline RL domains from the D4RL benchmark, where PG achieves strong performance compared to baselines, with a low computation cost.", "questions": "1. Why is the DDIM denoising process bijective in the limit of small discretization steps? Is there a prior work that discussed this?\n2. Equation 2 does not explain what the variables are, e.g. $V$ and $f$. It would help to explain them immediately after the equation.\n3. There is a recent related work on steering flow policies by learning the prior distribution [1]. They also train a noise-space value function to alias the sample-space value function. Can you discuss PG in relation to this work? \n\n[1] Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, Sergey Levine. Steering Your Diffusion Policy with Latent Space Reinforcement Learning", "limitations": "The authors address the limitations of the paper, stating (1) additional training cost, (2) breaking the bijective mapping assumption at practical discretization steps, and (3) no image-based experiments as the main limitations.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The paper presents a simple idea that can be applied to any pretrained diffusion planner -- just steer the diffusion planner by learning the prior distribution.\n2. The mathematical derivations for the noise-space objectives (Equations 4 and 5) are sound.\n3. The empirical results demonstrate state-of-the-art performance on the D4RL benchmark.\n4. The ablation studies and analysis experiments are quite informative. For example, Figure 5 shows a clear tradeoff between optimality and value model exploitation as the number of samples in Monte-Carlo search grows. Figure 7 shows the correlation between the real value function and the aliased noise-space value function, revealing a mismatch in the bijective assumption (though empirically the policy performance is not affected).\n5. The writing is clear and easy to follow.\n\n**Weaknesses**\n1. The D4RL results are not a significant improvement compared to the strongest baselines (DV*, DQL).", "was_revised": true, "final_justification": "The authors have sufficiently addressed my questions during the rebuttal. Overall, the paper proposes a practical method for finetuning diffusion models with RL, grounded on solid theoretical insights. Therefore I recommend acceptance.", "rebuttal": "**Dear Reviewer tREs,**\n\nWe are grateful for your thoughtful feedback. We hope the following responses address your questions and help alleviate your concerns.\n\n---\n\n**W1. The D4RL results are not a significant improvement compared to the strongest baselines (DV$^*$, DQL).**\n\nWe appreciate the reviewer for their careful reading and insightful comment regarding PG’s performance. In Table 1 of the paper, PG shows substantial performance improvements over DQL and DV* in long-horizon tasks such as Kitchen, AntMaze, and Maze2D. However, in MuJoCo locomotion tasks such as Walker2d, Hopper, and HalfCheetah, the gains over DV* are modest, and the overall performance is lower than DQL. We attribute this not to a limitation of PG itself, but to the use of a diffusion planner, which is less effective in short-horizon, dense-reward settings where trajectory-level modeling provides little benefit and can even be slightly detrimental in simpler environments where policies can already achieve near-optimal performance.\n\n|             | **DQL**        | **DV***       | **PG (planner)** | **PG (policy)**        |\n|:-----------:|:--------------:|:-------------:|:----------------:|:----------------------:|\n| Walker2d-M  | **87.0**       | 79.5          | 82.3              | 84.5 $\\pm$ 0.26         |\n| Walker2d-M-R| 95.5           | 83.5          | 83.7              | **96.4 $\\pm$ 2.49**      |\n| Walker2d-M-E| **110.1**      | 109.0         | 109.4             | 109.8 $\\pm$ 0.23        |\n| Hopper-M    | 90.5           | 84.1          | 97.5              | **98.4 $\\pm$ 1.01**     |\n| Hopper-M-R  | **101.3**      | 91.3          | 91.3              | 100.5 $\\pm$ 1.27        |\n| Hopper-M-E  | 111.1          | 109.9         | 110.4             | **111.4 $\\pm$ 0.57**    |\n| HalfCheetah-M| 51.1          | 50.9          | 45.6              | **54.7 $\\pm$ 0.42**     |\n| HalfCheetah-M-R| 47.8        | 46.4          | 46.4              | **48.6 $\\pm$ 0.34**     |\n| HalfCheetah-M-E| **96.8**    | 92.3          | 95.2              | 95.5 $\\pm$ 0.26         |\n| **Average** | **87.9**       | 83.0          | 84.6              | **88.9**                |\n\nTo verify this, we conducted an additional experiment by applying PG to a diffusion policy instead of a planner. Specifically, we replaced the standard Gaussian prior of a behavior-cloned diffusion policy with a learnable, behavior-regularized prior, following the same training procedure used in PG. The above table shows that replacing the planner with a policy (PG policy) consistently improved performance across MuJoCo tasks, increasing the average score from 84.6 (PG planner) to 88.9, surpassing not only DV* (83.0) but also the DQL (87.9). This result demonstrates that PG remains effective even in short-horizon environments and the lower performance observed in MuJoCo tasks is due to the planner architecture rather than the PG framework itself. The results for PG (policy) are averaged over 3 random seeds and reported as the mean $\\pm$ standard error.\n\nWe will include these results to further clarify that the PG framework is effective across both policy- and planner-based approaches.\n\n---\n\n**Q1. Why is the DDIM denoising process bijective in the limit of small discretization steps? Is there a prior work that discussed this?**\n\nWe thank the reviewer for requesting this clarification. The DDIM denoising process [1] becomes bijective in the limit of small discretization steps because it corresponds to solving a particular ordinary differential equation (ODE) using Euler integration. In Section 4.3 of the DDIM paper, the authors show that DDIM sampling is equivalent to numerically solving an ODE of the form. \n\n$$\nd \\bar{\\mathbf{x}}(t)=\\epsilon^{(t)}_\\theta\\bigg(\\frac{\\bar{\\mathbf{x}}(t)}{\\sqrt{\\sigma(t)^2 +1}}\\bigg) \\cdot d \\sigma(t),\n$$\n\nwhere $\\bar{\\mathbf{x}}(t)=\\frac{\\mathbf{x}(t)}{\\sqrt{\\bar{\\alpha}_t}}$ and $\\sigma(t) = \\sqrt{\\frac{1-\\bar{\\alpha}_t}{\\bar{\\alpha}_t}}$. This ODE describes a deterministic transformation from noise $\\bar{\\mathbf{x}}(T)$  to data $\\bar{\\mathbf{x}}(0)$, and vice versa. Therefore, with sufficiently fine discretization, the DDIM process approximates the ODE solution arbitrarily well. Furthermore, Section 4.3 and Appendix B of [1] explicitly note that DDIM corresponds to the probability flow ODE for the variance-exploding SDE from Song et al. (2020) [2]. \n\nThis equivalence explains why the DDIM sampling path becomes bijective when using small discretization steps. Several works leveraged this property for DDIM inversion, the process of mapping real data samples back into the latent noise space [1, 3, 4].\n\n---\n\n**Q2. Equation 2 does not explain what the variables are, e.g $V$ and $f$. It would help to explain them immediately after the equation.**\n\nWe thank the reviewer for pointing out the lack of variable definitions in Equation (2). We agree that the meanings of $V$ and $f$ should be clarified immediately following the equation to improve readability, as the original explanation in the Preliminary section is far from Equation (2) and may disrupt the reader’s flow. To address this, we will revise the main paper to include explicit definitions right after Equation (2). Specifically, we will add the following sentence in the next revision of the paper:\n\nHere, $V(\\mathbf{x}_0)$ denotes the estimated value of the denoised trajectory $\\mathbf{x}_0$ and $f(\\cdot)$ is a regularization function (e.g., for $f$-divergence) that penalizes deviation from the behavior distribution.\n\n---\n\n**Q3. There is a recent related work on steering flow policies by learning the prior distribution [1]. They also train a noise-space value function to alias the sample-space value function. Can you discuss PG in relation to this work?**\n\nWe appreciate the reviewer for bringing up this connection. A referenced concurrent work, \"Steering Your Diffusion Policy with Latent Space Reinforcement Learning\" (DSRL) [5], proposes steering a behavior-cloned diffusion policy by optimizing a learnable prior via reinforcement learning. While the high-level goal of guiding generative models through learned priors is shared with our method, there are both important commonalities and key differences:\n\n**Commonalities: Training Latent Prior and Value function**\n\nBoth DSRL and PG aim to improve decision quality by learning the prior over the noise space ($\\mathbf{x}_T \\sim \\mathcal{N}(\\mathbf{0}, \\mathbf{I})$) that feeds into the diffusion model’s denoising process, allowing for the sampling of high-value actions or trajectories. In both methods, the prior is trained to maximize the value function defined over $\\mathbf{x}_T$, which estimates the value of the corresponding denoised sample $\\mathbf{x}_0$.\n\n**Key Differences**\n\n1. **Diffusion Planner and Diffusion Policy**\n    \n    Unlike DSRL, which  employs a diffusion policy that directly maps noise to action, PG utilizes a diffusion planner that outputs entire state trajectories and uses an inverse dynamics model to reconstruct executable actions. This makes PG particularly suitable for long-horizon and sparse-reward settings, where trajectory-level modeling is essential.\n    \n2. **Theoretical Insight**\n    \n    In our work, we provide a formal result (Proposition 1) demonstrating that, with a sufficiently large number of discretization steps, the objective for optimizing the diffusion model can be equivalently reformulated as an optimization over the latent prior distribution. This equivalence is enabled by the bijective nature of the DDIM sampling process, which allows for an exact mapping between the data space and the latent prior. This reformulation is not explored in DSRL.\n    \n3. **Handling OOD Samples in Offline RL**\n    \n    DSRL assumes that the learned diffusion model generates only in‑distribution samples and therefore does not incorporate explicit behavior constraints. However, as shown in Figures 3 and 5, diffusion models trained on offline datasets can still produce out‑of‑distribution samples that degrade performance. To mitigate this issue, PG introduces behavior regularization by imposing an $f$‑divergence penalty between the learned prior and the standard normal distribution. This explicitly penalizes deviation from the behavior distribution and induces regularization at the prior level.\n    \n\nWe will revise the Related Works section to clearly position our contributions relative to DSRL and highlight both the shared foundations and the novel aspects of our approach.\n\n---\n\n**References**\n\n[1] Song, Jiaming, Chenlin Meng, and Stefano Ermon. \"Denoising diffusion implicit models.\" *arXiv preprint arXiv:2010.02502* (2020).\n\n[2] Song, Yang, et al. \"Score-based generative modeling through stochastic differential equations.\" *arXiv preprint arXiv:2011.13456* (2020).\n\n[3] Dhariwal, Prafulla, and Alexander Nichol. \"Diffusion models beat gans on image synthesis.\" Advances in neural information processing systems 34 (2021): 8780-8794.\n\n[4] Zeng, Yan, Masanori Suganuma, and Takayuki Okatani. \"Inverting the generation process of denoising diffusion implicit models: Empirical evaluation and a novel method.\" *2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)*. IEEE, 2025.\n\n[5] Wagenmaker, Andrew, et al. \"Steering Your Diffusion Policy with Latent Space Reinforcement Learning.\" arXiv preprint arXiv:2506.15799 (2025)."}, {"review_id": "C7ebKo34mo", "reviewer": "Reviewer_J4nL", "summary": "This paper introduces a novel method, Prior Guidance (PG), into the standard diffusion planning framework. Instead of sampling from a fixed Gaussian prior, the method samples from a learnable prior, which allows the policy to be optimized via a behavior-regularized objective. Experimental results demonstrate that PG performs well across multiple tasks, particularly on long-horizon problems. It also outperforms standard Monte Carlo Sample Selection (MCSS) methods in both performance and computational efficiency.", "questions": "1. Why does the proposed PG method not perform well on some datasets? For example, in Table 1, PG does not outperform some diffusion-based methods such as DQL in MuJoCo tasks. Does the method only work well on long-horizon tasks?\n\n2. How does the method achieve good performance on long-horizon tasks, consistently outperforming MCSS? It seems that prior guidance alone does not provide this capability. Does the improvement come from Diffusion Veteran (DV)?\n\n3. It is unclear whether the method modifies the training of the base Diffusion Veteran. Is the proposed prior guidance applied only at the sampling stage?", "limitations": "The limitations of this work are discussed in the paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The idea of introducing a learnable prior into the sampling process of diffusion planning is quite novel.\n\n2. Theoretical analysis is provided to justify the training objective of prior guidance and its validity.\n\n3. The paper presents comprehensive experimental results, offering a convincing evaluation of the proposed method.\n\n4. Concrete and detailed implementation details are provided in the appendix.\n\nWeaknesses:\n\n1. Although the proposed PG method performs well on some long-horizon tasks, it does not show better performance on MuJoCo simulation tasks compared to standard diffusion-based policies such as Diffusion-QL.\n\n2. It is unclear how much of the performance gain comes from the incorporation of Diffusion Veteran (DV). The authors are encouraged to include an ablation study using the standard Diffuser as the base model to better isolate the contribution of PG.", "was_revised": true, "final_justification": "I thank the reviewer for the detailed responses in the rebuttal. The results seem promising; however, the lack of an ablation study on the effect of dropping the Diffusion Veteran (DV) makes the contribution of the proposed method to the overall performance unclear. Nevertheless, the method itself appears promising, so I will maintain my current evaluation.", "rebuttal": "**Dear Reviewer J4nL,**\n\nWe appreciate your valuable comments. We hope that our responses below will clarify the points raised and resolve any concerns.\n\n---\n\n**W2. It is unclear how much of the performance gain comes from the incorporation of Diffusion Veteran (DV). The authors are encouraged to include an ablation study using the standard Diffuser as the base model to better isolate the contribution of PG.**\n\nWe thank the reviewer for highlighting the importance of isolating the contribution of PG from that of Diffusion Veteran (DV). Due to the limited time available during the rebuttal period, we were unable to conduct an ablation using the standard Diffuser as the base model. Instead, we performed a related experiment by applying PG to a diffusion policy that uses the same diffusion model architecture as DQL, allowing us to assess whether PG provides meaningful performance gains even without incorporating DV. As detailed in our response to Q1, applying PG by substituting the standard Gaussian prior of a behavior-cloned diffusion policy with a learned, behavior-regularized prior achieved an average score of 88.9, outperforming other diffusion-based policies on MuJoCo tasks. These results demonstrate that PG provides a meaningful performance gain even without DV, indicating that its benefits are not tied to a specific base model.\n\n---\n\n**W1, Q1. Why does the proposed PG method not perform well on some datasets? For example, in Table 1, PG does not outperform some diffusion-based methods such as DQL in MuJoCo tasks. Does the method only work well on long-horizon tasks?**\n\nWe appreciate the reviewer’s insightful comment on PG’s performance in MuJoCo tasks. PG is a diffusion planner-based method designed for long-horizon planning, where capturing temporally extended dependencies is essential. However, MuJoCo locomotion tasks involve short-horizon, dense-reward control focused on making agents run faster, which does not require such lookahead capabilities [1]. As a result, diffusion planner-based methods, including PG, often underperform compared to diffusion policy-based approaches in these settings, as shown in Table 1. While the additional structure and trajectory-level modeling of planners benefit challenging long-horizon domains such as AntMaze, Kitchen, or Maze2D, they provide little advantage and can even be slightly detrimental in simpler environments where policies can already achieve near-optimal performance.\n\n|                   | DQL        | DV*       | PG (planner) | PG (policy)        |\n|:-----------:|:--------------:|:-------------:|:----------------:|:----------------------:|\n| Walker2d-M  | **87.0**       | 79.5          | 82.3              | 84.5 $\\pm$ 0.26         |\n| Walker2d-M-R| 95.5           | 83.5          | 83.7              | **96.4 $\\pm$ 2.49**      |\n| Walker2d-M-E| **110.1**      | 109.0         | 109.4             | 109.8 $\\pm$ 0.23        |\n| Hopper-M    | 90.5           | 84.1          | 97.5              | **98.4 $\\pm$ 1.01**     |\n| Hopper-M-R  | **101.3**      | 91.3          | 91.3              | 100.5 $\\pm$ 1.27        |\n| Hopper-M-E  | 111.1          | 109.9         | 110.4             | **111.4 $\\pm$ 0.57**    |\n| HalfCheetah-M| 51.1          | 50.9          | 45.6              | **54.7 $\\pm$ 0.42**     |\n| HalfCheetah-M-R| 47.8        | 46.4          | 46.4              | **48.6 $\\pm$ 0.34**     |\n| HalfCheetah-M-E| **96.8**    | 92.3          | 95.2              | 95.5 $\\pm$ 0.26         |\n| Average | 87.9       | 83.0          | 84.6              | **88.9**                |\n\nTo examine whether the observed performance gap in MuJoCo tasks stems from the use of a diffusion planner rather than a limitation of the PG framework itself, we conduct an additional experiment by applying PG to a diffusion policy. Specifically, we replaced the standard Gaussian prior of a behavior-cloned diffusion policy with a learnable, behavior-regularized prior, following the same training procedure used in PG.\nThe above table shows that replacing the diffusion planner in PG with a diffusion policy (PG policy) consistently improved performance across MuJoCo tasks, raising the average score from 84.6 (PG planner) to 88.9 and surpassing the DQL baseline score of 87.9. This demonstrates that PG remains effective even in short-horizon environments, suggesting that the lower performance of PG in MuJoCo tasks is due to the nature of trajectory-level planning rather than a limitation of the method itself. The results for PG (policy) are averaged over 3 random seeds and reported as the mean $\\pm$ standard error.\n\nWe will include these results to further clarify that the PG framework is effective across both policy- and planner-based approaches.\n\n---\n\n**Q2. How does the method achieve good performance on long-horizon tasks, consistently outperforming MCSS? It seems that prior guidance alone does not provide this capability. Does the improvement come from Diffusion Veteran (DV)?**\n\nWe apologize for the lack of clarity and appreciate the reviewer for raising this important point. We would like to clarify that in Figure 5, the method labeled as MCSS with 50 samples corresponds directly to Diffusion Veteran (DV). That is, DV is implemented by sampling 50 trajectories from a behavior-cloned diffusion planner and selecting the highest-value one using a learned value function. Importantly, all methods in Figure 5, including MCSS (with different sample counts) and PG, share the same diffusion model architecture and hyperparameters, all trained using the DV procedure. The performance differences arise solely from the inference-time guidance algorithms.\n\nWhile DV (i.e., MCSS-50) achieves strong performance, MCSS inherently couples the number of samples with the degree of behavior regularization. Increasing the number of samples improves the chance of finding higher-value trajectories, but it also raises the risk of selecting out-of-distribution ones, forcing a trade-off between trajectory quality and regularization strength. Because of this coupling, there is a practical ceiling on how precisely MCSS can optimize for the best high-value trajectory under a given level of regularization. Beyond a certain point, increasing samples no longer yields better results without sacrificing regularization.\n\nIn contrast, PG completely decouples behavior regularization from the search for high-value trajectories. The $f$-divergence regularization is applied explicitly and independently, allowing the latent space prior to be optimized solely for maximizing trajectory value within the regularization constraint. This enables PG to accurately reach the behavior-regularized optimum, without approximation error from finite sampling, and without the computational cost or OOD risk that MCSS faces at large sample counts.\n\nWe will revise the paper to clearly indicate that DV corresponds to MCSS with 50 samples, and that all methods share the same diffusion model. The observed improvements thus stem from our proposed prior-guided inference, rather than differences in model capacity or training setup.\n\n---\n\n**Q3. It is unclear whether the method modifies the training of the base Diffusion Veteran. Is the proposed prior guidance applied only at the sampling stage?**\n\nWe apologize for the confusion and we appreciate the reviewer’s careful reading. To clarify, PG does not modify the training of the underlying diffusion model used in Diffusion Veteran (DV). The diffusion planner is trained exactly as in DV, using standard behavior cloning on offline trajectories. PG is applied only during inference, where it replaces the MCSS procedure used in DV. Specifically:\n\n- During training we introduce an additional learnable prior distribution over the noise variable $\\mathbf{x}_T$ that is optimized to maximize the expected value under the value function subject to an $f$‑divergence constraint maintaining closeness to the standard Gaussian.\n- This prior is trained after the diffusion model has been fixed through behavior cloning. At the sampling stage, the standard Gaussian prior is replaced with the learned prior distribution, after which the behavior‑cloned diffusion model is used for denoising.\n- The diffusion model itself is not fine-tuned or altered in any way.\n\nIn summary, PG is a post‑hoc, inference‑time guidance method applied to a fixed diffusion model. All performance gains come from learned prior distribution rather than from altering the base diffusion model’s training. We will include this clarification in the next revision of the paper.\n\n---\n\n**References**\n\n[1] Lu, Haofei, et al. \"What makes a good diffusion planner for decision making?.\" *arXiv preprint arXiv:2503.00535* (2025)."}, {"review_id": "OAK6CJ88LR", "reviewer": "Reviewer_6FdX", "summary": "This paper introduces Prior Guidance (PG), a novel method for improving diffusion-based planners in offline reinforcement learning (RL).\nthe key idea is to learn a prior conditional on the state instead the gaussian prior in order to steer the diffusion samples to high return regions. At training, the author propose to learn a latent value function (value function that depends on the noise sampled from the prior) to avoid backpropagate gradient through the denoising steps. at inference time, we need just sample from the learned prior which save runtime comparing to other guidance method (classifer guidance or Monte Carlo sample selection)", "questions": "see weakness above", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- Computational Efficiency: Reduces inference time significantly by avoiding sampling multiple candidate trajectories (unlike Monte Carlo methods).\n- Avoids backpropagation through the denoising process by using a latent-space critic.\n- Closed-form Regularization: Behavior regularization is done in the latent space where densities are tractable (Gaussian), allowing for closed-form solutions and improved stability.\n- The method builds on existing diffusion planners with minimal changes: just replace the prior and train an auxiliary value function.\n\nWeakness: \n\n- Additional Complexity: comparing to MCSS, the proposed method requires training and tuning of a latent-space value function and a learnable prior, adding architectural and training complexity.\n- in order to get a tractable regularization term, the author assumes bijectivity of the denoising scheme, which is not really satisfied.\n- Empirically speaking, while the proposed seem to work well, the performance boost is not too important. I think the performance is quite saturated in D4RL Benchmark. it would be important in include more challenging domains (e.g long-horizon tasks in ogbench, humanoid-large-maze ?)", "was_revised": true, "final_justification": "I maintain my score. My main concern is still the limited evaluation to D4RL where we see very moderate boost of performance.", "rebuttal": "**Dear Reviewer 6FdX,**\n\nThank you for your valuable comments. We hope that the following responses will answer your questions and address the concerns you have raised.\n\n---\n\n**W1. Additional Complexity: comparing to MCSS, the proposed method requires training and tuning of a latent-space value function and a learnable prior, adding architectural and training complexity.**\n\nWe acknowledge that Prior Guidance (PG) introduces additional networks, which we have also noted in the Limitations section, and that this indeed increase architectural and training complexity relative to MCSS. However, we highlight that this added complexity is both minimal and reasonable given the substantial gains in performance and efficiency. \n\n1. **Lower Training Cost**\n\n    The introduced components are lightweight and modular. The prior is parameterized by a simple GRU and the latent value function is a shallow MLP, both adding minor overhead compared to the diffusion planner backbone. Moreover, these networks are trained separately from the planner without backpropagation through the diffusion model, keeping the overall computational cost manageable (see Figure 4, left).\n\n2. **Avoidance of Costly Sampling**\n\n    MCSS requires sampling and scoring a large number of candidate trajectories at inference time, which is both computationally expensive and difficult to scale, especially in long-horizon or high-dimensional settings. PG eliminates this bottleneck by generating a single trajectory, yielding up to a 500× reduction in inference cost (see Figure 4, right).\n    \n3. **Performance Validates Overhead**\n\n    Despite the small additional training components, PG consistently outperforms MCSS across most long-horizon benchmarks (Table 1, Figure 5), demonstrating that the modest increase in training complexity is outweighed by substantial gains in both efficiency and performance.\n\n---\n\n**W2. in order to get a tractable regularization term, the author assumes bijectivity of the denoising scheme, which is not really satisfied.**\n\nWe acknowledge that exact bijectivity is not guaranteed in practice under standard DDIM sampling with finite discretization steps. This is a known limitation of DDIM inversion [1, 2], which we have also noted in the Limitations section. To fully ensure bijectivity, one could employ an ODE-based generative model or a flow-matching framework, both of which offer theoretical invertibility. \n\nNevertheless, our empirical results indicate that the bijective assumption holds approximately in practice and is sufficient for effective guidance. As shown in Figure 6, noise samples drawn from the learned prior consistently denoise into high‑value trajectories, indicating that the mapping from latent space to trajectory space is preserved well enough to enable strong performance. Consequently, as reported in Table 1, PG achieves state‑of‑the‑art results across diverse benchmarks, including long‑horizon planning tasks such as AntMaze, Kitchen, and Maze2D.\n\nThese results suggest that while exact bijectivity is not guaranteed, the approximate invertibility of DDIM is sufficient in practice for guiding diffusion models via learned priors.\n\n---\n\n**W3. Empirically speaking, while the proposed seem to work well, the performance boost is not too important. I think the performance is quite saturated in D4RL Benchmark. it would be important in include more challenging domains (e.g long-horizon tasks in ogbench, humanoid-large-maze ?)**\n\nWe thank the reviewer for this valuable suggestion. We agree that D4RL has been extensively studied and that evaluating on more challenging domains, such as long‑horizon tasks in OGBench or humanoid‑large‑maze, would further validate the generality of our approach. However, given the limited rebuttal period, conducting such large‑scale experiments is unfortunately infeasible. We believe that PG clearly outperforms other baselines in long-horizon planning tasks within D4RL, while also reducing inference time substantially compared to the previous state-of-the-art method DV, constitutes a meaningful contribution. We thank the reviewer again for this suggestion and plan to extend our evaluation to more challenging environments in future work. \n\n---\n\n**References**\n\n[1] Song, Jiaming, Chenlin Meng, and Stefano Ermon. \"Denoising diffusion implicit models.\" *arXiv preprint arXiv:2010.02502* (2020).\n\n[2] Dhariwal, Prafulla, and Alexander Nichol. \"Diffusion models beat gans on image synthesis.\" Advances in neural information processing systems 34 (2021): 8780-8794."}, {"review_id": "lAzgUFCS6l", "reviewer": "Reviewer_rn8g", "summary": "This paper introduces Prior Guidance (PG) an offline RL approach to guide the diffusion-based planner. The authors argue that current diffusion-based guidance methods including Classifier Guidance and Classifier-Free Guidance and Monte Carlo Sample Selection (MCSS) suffer from limitations such as computational inefficiency, overfitting to out-of-distribution (OOD) actions etc.\n\nPG proposes to replace the standard Gaussian prior in the difussion model with a learned, state-conditioned Gaussian prior, optimizing using a behavior-regularized objective. To avoid backpropagation through the diffusion process, the authors introduce a latent value function that operates directly in the prior space, enabling efficient training and inference.\n\nPG is evaluated empirically on long-horizon planning tasks such as D4RL, Mujoco control suit and Maze2D-U.", "questions": "- Can you clarify which specific Kitchen tasks were used? Is the reported score averaged over several tasks (e.g., kitchen-mixed-v0, kitchen-partial-v0, etc.) or based on a single one?\n- Why does PG underperform on standard MuJoCo tasks? These tasks are supposed to be easy and saturated within the RL domain. Do you attribute this to horizon length, simplicity of dynamics, or limitations of your prior formulation?\n- How is the inverse dynamics model trained and evaluated?", "limitations": "- Domain sensitivity: PG performs well on temporally extended tasks (kitchen, Maze2D-U) but fails to match baseline in standard control environments, suggesting its benefits are limited to certain settings.\n- Because the denoising model is frozen, PG may not adapt well to high-return regions discovered during training - If I understood correctly, PG freezes the diffusion model after behavior cloning. So only the prior is trained to bias generation towards higher-value trajectories. But now suppose the value function discovers a new region in latent space where higher returns can be achieved. Now if we want to shift the prior to sample more from this region but the decoder is unchanged, it may not accurately reconstruct those high-return trajectories.\n- Restricting the prior to a Gaussian form limits expressiveness", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- Reformulating the guidance objective in the latent space and optimizing the prior directly is theoretically sound and overall well-motivated.\n- PG avoids the high inference-time cost of MCSS and eliminates backpropagation through denoising network.\n- PG enables tractable, analytically stable regularization by performing directly in the prior space \n- Good performance across some of the baselines including Kitchen add Maze2D-U, which require multi-step planning and skill composition.\n- Overall paper is well-written and easy to follow.\n- The paper provides informative comparisons and ablations between PG and without denoising backpropagation as wel as existing methods like MCSS and Diffusion Q-learning.\n\n## Weaknesses\n- I have some concerns with regards to the empirical results. In Mujoco control environments like Walker2d, HalfCheetah and Hopper PG is consistently outperformed by existing baselines. This raises questions about its generality beyond long-horizon planning.\n- It was also not clear to me what constitutes the Kitchen benchmark, the paper keeps refering to \"Kitchen\" without explaining that this is the FrankaKitchen environment from D4RL? For readers unfamiliar with it, the task complexity and relevance remain unclear (and whether the single reported results is the average across multiple kitchen tasks). It might be more helpful to see performance of individual tasks since this method has shortcomings in some of these mujoco environments.\n- Theoretical justification hinges on DDIM being bijective which in parctice it may not hold (because it's an approximate ODE solver)\n- PG uses a relatively simple Gaussian prior which may not be sufficient in stochastic behavior setting\n\nOverall, I like the ideas in this paper. I would be open to raising my score if the authors can convincingly explain the reasons behind the weaker performance on standard control tasks and provide assurance that these results do not reflect a fundamental flaw in the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer rn8g,**\n\nWe appreciate your thoughtful and detailed review. We hope that the explanations provided below will address your questions and resolve any concerns..\n\n---\n\n**W3. Theoretical justification hinges on DDIM being bijective which in parctice it may not hold (because it's an approximate ODE solver)**\n\nWe acknowledge that exact bijectivity is not guaranteed in practice under standard DDIM sampling with finite discretization steps. This is a known limitation of DDIM inversion [1, 2], which we have also noted in the Limitations section. To fully ensure bijectivity, one could employ an ODE-based generative model or a flow-matching framework, both of which offer theoretical invertibility. \n\nNevertheless, our empirical results indicate that the bijective assumption holds approximately in practice and is sufficient for effective guidance. As shown in Figure 6, noise samples drawn from the learned prior consistently denoise into high‑value trajectories, indicating that the mapping from latent space to trajectory space is preserved well enough to enable strong performance. Consequently, as reported in Table 1, PG achieves state‑of‑the‑art results across diverse benchmarks, including long‑horizon planning tasks such as AntMaze, Kitchen, and Maze2D.\n\nThese results suggest that while exact bijectivity is not guaranteed, the approximate invertibility of DDIM is sufficient in practice for guiding diffusion models via learned priors.\n\n---\n\n**W4. PG uses a relatively simple Gaussian prior which may not be sufficient in stochastic behavior setting.**\n\nWe appreciate the reviewer’s insightful comment regarding the potential limitations of using a simple Gaussian prior in stochastic behavior settings. We agree that a more expressive prior model, such as a Gaussian mixture model (GMM), could be beneficial in scenarios where multiple distinct trajectories achieve similarly high returns, even under a deterministic optimal policy. To investigate this, we evaluated PG with a multi‑modal prior in our experiments (Table 2) to assess whether such an expressiveness would improve performance, and found that the improvement was not substantial. Nonetheless, PG can be easily extended to incorporate a more expressive prior, and our framework is fully compatible with such an extension if needed.\n\n---\n\n**W2, Q1. Can you clarify which specific Kitchen tasks were used? Is the reported score averaged over several tasks (e.g., kitchen-mixed-v0, kitchen-partial-v0, etc.) or based on a single one?**\n\nWe thank the reviewer for the helpful question and apologize for the lack of clarity. We clarify that the **\"Kitchen\" domain refers to the FrankaKitchen environment from the D4RL benchmark suite** [1]. In Table 1, we report performance on two standard Kitchen tasks from the D4RL benchmark suite: kitchen-mixed-v0 (denoted as Kitchen-M) and kitchen-partial-v0 (denoted as Kitchen-P). The “Kitchen” score, as reported in Figure 5, and Figure 7(b), refers to the average of the normalized scores across these two tasks. We will add detailed clarification on Section 4.1 in the final version.\n\n---\n\n**W1, Q2. Why does PG underperform on standard MuJoCo tasks? These tasks are supposed to be easy and saturated within the RL domain. Do you attribute this to horizon length, simplicity of dynamics, or limitations of your prior formulation?**\n\nWe appreciate the reviewer’s insightful comment on PG’s performance in standard MuJoCo tasks. PG is a diffusion planner-based method designed for long-horizon planning, where capturing temporally extended dependencies is essential. However, MuJoCo locomotion tasks involve short-horizon, dense-reward control focused on making agents run faster, which does not require such lookahead capabilities [2]. As a result, diffusion planner-based methods, including PG, often underperform compared to diffusion policy-based approaches in these settings, as shown in Table 1 of our paper. While the additional structure and trajectory-level modeling of planners benefit challenging long-horizon domains such as AntMaze, Kitchen, or Maze2D, they provide little advantage and can even be slightly detrimental in simpler environments where policies can already achieve near-optimal performance.\n\n|                   | DQL        | DV*       | PG (planner) | PG (policy)        |\n|:-----------:|:--------------:|:-------------:|:----------------:|:----------------------:|\n| Walker2d-M  | **87.0**       | 79.5          | 82.3              | 84.5 $\\pm$ 0.26         |\n| Walker2d-M-R| 95.5           | 83.5          | 83.7              | **96.4 $\\pm$ 2.49**      |\n| Walker2d-M-E| **110.1**      | 109.0         | 109.4             | 109.8 $\\pm$ 0.23        |\n| Hopper-M    | 90.5           | 84.1          | 97.5              | **98.4 $\\pm$ 1.01**     |\n| Hopper-M-R  | **101.3**      | 91.3          | 91.3              | 100.5 $\\pm$ 1.27        |\n| Hopper-M-E  | 111.1          | 109.9         | 110.4             | **111.4 $\\pm$ 0.57**    |\n| HalfCheetah-M| 51.1          | 50.9          | 45.6              | **54.7 $\\pm$ 0.42**     |\n| HalfCheetah-M-R| 47.8        | 46.4          | 46.4              | **48.6 $\\pm$ 0.34**     |\n| HalfCheetah-M-E| **96.8**    | 92.3          | 95.2              | 95.5 $\\pm$ 0.26         |\n| Average | 87.9       | 83.0          | 84.6              | **88.9**                |\n\nTo examine whether the observed performance gap in MuJoCo tasks stems from the use of a diffusion planner rather than a limitation of the PG framework itself, we conduct an additional experiment by applying PG to a diffusion policy. Specifically, we replaced the standard Gaussian prior of a behavior-cloned diffusion policy with a learnable, behavior-regularized prior, following the same training procedure used in PG.\n\nThe above table shows that replacing the diffusion planner in PG with a diffusion policy (PG policy) consistently improved performance across MuJoCo tasks, raising the average score from 84.6 (PG planner) to 88.9 and surpassing the DQL baseline score of 87.9. This demonstrates that PG remains effective even in short-horizon environments, suggesting that the lower performance of PG in MuJoCo tasks is due to the nature of trajectory-level planning rather than a limitation of the method itself. The results for PG (policy) are averaged over 3 random seeds and reported as the mean $\\pm$ standard error.\n\nWe will include these results to further clarify that the PG framework is effective across both policy- and planner-based approaches.\n\n---\n\n**Q3. How is the inverse dynamics model trained and evaluated?**\n\nWe apologize for the lack of explanation regarding the inverse dynamics model. In Prior Guidance (PG), the inverse dynamics model $\\epsilon_w$ is trained following the setup established in Diffusion Veteran (DV), which PG builds upon.  The model maps a pair of temporally spaced states $(\\mathbf{s}^{(\\tau)}, \\mathbf{s}^{(\\tau+m)})$ to the action $\\mathbf{a}^{(\\tau)}$ , where $\\tau$ denotes the current environment step, and $m$ is the planning stride. Then $\\epsilon_w$ is used to recover executable actions from denoised state trajectories $\\mathbf{x}_0 = \\left[ \\mathbf{s}^{(\\tau)}, \\mathbf{s}^{(\\tau + m)}, \\dots, \\mathbf{s}^{(\\tau + (H-1)m)} \\right]$ generated by the diffusion planner.\n\n**Training:**\n\n- The inverse model is trained via supervised regression from the dataset $\\mathcal{D}$. The objective is to minimize the following loss:\n    \n    $$\n    \\mathbb{E}_{\\mathbf{s}^{(\\tau)}, \\mathbf{s}^{(\\tau + m)}, \\mathbf{a}^{(\\tau)},\\epsilon,t} \\left[\\left\\Vert \\epsilon - \\epsilon_w \\left(\\sqrt{\\bar{\\alpha}_t}  \\mathbf{a}^{(\\tau)} + \\sqrt{1 - \\bar{\\alpha}_t}\\epsilon, t, s^{(\\tau)},s^{(\\tau+m)}\\right) \\right\\Vert^2\\right]\n    $$\n\n**Inference:**\n\n- At test time, after PG generates a full predicted state trajectory  $\\mathbf{x}_0 = \\left[ \\mathbf{s}^{(\\tau)}, \\mathbf{s}^{(\\tau + m)}, \\dots, \\mathbf{s}^{(\\tau + (H-1)m)} \\right]$, and the action $\\mathbf{a}^{(\\tau)}$  is computed by applying the inverse dynamics model $\\epsilon_w$ to the first two states $\\left(\\mathbf{s}^{(\\tau)}, \\mathbf{s}^{(\\tau + m)}\\right)$\n- This action is then executed in the environment, and the process repeats at the next timestep.\n\nWe will include this detailed description of the inverse dynamics model in the next revision.\n\n---\n\n**References**\n\n[1] Fu, Justin, et al. \"D4rl: Datasets for deep data-driven reinforcement learning.\" *arXiv preprint arXiv:2004.07219* (2020).\n\n[2] Lu, Haofei, et al. \"What makes a good diffusion planner for decision making?.\" *arXiv preprint arXiv:2503.00535* (2025)."}]

Author rebuttal:

Meta-review: This paper proposes Prior Guidance (PG) for diffusion-based planning in offline RL. Instead of using a fixed Gaussian prior, PG introduces a learnable, state-conditioned prior optimized with a behavior-regularized objective. The method avoids costly Monte Carlo sampling at inference time by training a latent-space value function that aliases the denoised trajectory value, thereby steering the diffusion process efficiently. The approach is evaluated on D4RL tasks, showing strong results in long-horizon environments (AntMaze, Kitchen, Maze2D) and competitive performance on others.

Reviewers agreed that the paper is clearly written, the idea is simple yet elegant, and the derivations are sound. Strengths include: (i) computational efficiency by replacing sampling-heavy MCSS, (ii) novel reformulation of guidance in latent space, (iii) tractable regularization, and (iv) comprehensive ablations and analysis.
Concerns were mainly empirical. Multiple reviewers noted that PG underperforms on short-horizon MuJoCo locomotion tasks compared to strong diffusion policies such as DQL. This raised questions about whether the benefits of PG are limited to long-horizon planning. The rebuttal addressed this point with additional experiments: applying PG to a diffusion policy (rather than a planner) improved performance on MuJoCo tasks, surpassing DQL. This clarified that the performance gap stems from the planner architecture, not PG itself. Reviewers also raised questions on assumptions (e.g., bijectivity of DDIM), the use of a simple Gaussian prior, and the reliance on DV as a base model. The authors clarified that bijectivity is approximate but sufficient in practice, PG can easily be extended to more expressive priors, and ablations confirm PG provides gains independently of DV. Clarifications on Kitchen tasks, the inverse dynamics model, and connections to concurrent work (DSRL) were also promised.

Overall, this is a technically solid and well-presented contribution. While performance gains are modest in some saturated benchmarks, PG clearly improves efficiency and shows notable advantages in long-horizon settings. The rebuttal convincingly addressed most concerns. Based on the reviewers’ feedback, I recommend that the authors include the additional ablation studies conducted during the rebuttal on applying PG to a diffusion policy, and clearly articulate the appropriate application scope of PG. In addition, the authors should provide a clearer summary of the experimental results to better highlight the benefits of PG.
Decision: Accept (poster)