Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning
Authors: Kyowoon Lee
Jaesik Choi
Venue: ICML 2025 poster (2025)
Keywords: Offline Reinforcement Learning
Trajectory Optimization
Diffusion Models
Sequential Decision Making
Primary area: reinforcement_learning->planning
TL;DR: training-free method that corrects off-manifold deviations in diffusion-based planners

Abstract: Recent advances in diffusion-based generative modeling have demonstrated significant promise in tackling long-horizon, sparse-reward tasks by leveraging offline datasets. While these approaches have achieved promising results, their reliability remains inconsistent due to the inherent stochastic risk of producing infeasible trajectories, limiting their applicability in safety-critical applications. We identify that the primary cause of these failures is inaccurate guidance during the sampling procedure, and demonstrate the existence of manifold deviation by deriving a lower bound on the guidance gap. To address this challenge, we propose *Local Manifold Approximation and Projection* (LoMAP), a *training-free* method that projects the guided sample onto a low-rank subspace approximated from offline datasets, preventing infeasible trajectory generation. We validate our approach on standard offline reinforcement learning benchmarks that involve challenging long-horizon planning. Furthermore, we show that, as a standalone module, LoMAP can be incorporated into the hierarchical diffusion planner, providing further performance enhancements.

Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning
Kyowoon Lee!
Jaesik Choi
12
Abstract
not readily available and must be learned from data. Model- based reinforcement learning (MBRL) (Sutton, 2018) ad- dresses this by coupling learned dynamics models with plan- ners, offering increased data-efficiency and the flcxibility to adapt across tasks. While MBRL offers a promising frame- work, it vulnerable to adversarial plan exploitation when the learned model is imperfect (Talvitie, 2014; Asadi et al., 2018; 8; Luo al., 2019; Janner et al., 2019; Voelcker et al., 2022).
Recent advances in diffusion-based generative modeling have demonstrated significant promise in tackling hong-horizon, sparse-reward tasks by leveraging offline datasets. While these ap- proaches have achieved promising results, their reliability remains inconsistent due to the inherent stochastic risk of producing infeasible trajectories, limiting their applicability in safety-critical appli- cations. We identify that the primary cause of these failures inaccurate guidance during the sampling procedure, and demonstrate the exis- tence of manifold deviation by deriving lower bound the guidance gap. To address this chal- lenge, we propose Local Manifold Approximation and Projection (LoMAP), training-free free method that projects the guided sample onto a low-rank subspace approximated from offline datasets, pre- venting infeasible trajectory generation. We val- idate our approach on standard offline reinforce- ment learning benchmarks that involve challeng- ing long-howe morizoor planning. Furthermore, show that, as a standalone module, LoMAP can be in- corporated into the hierarchical diffusion planner, providing further performance enhancements.
Recent progress in diffusion models provides an appealing alternative for long-horizon planning. Originally introduced as powerful generative models that iteratively reverse mul- tistep noising process (Ho al., 2020; Song al., 2021), diffusion models have demonstrated state-of-the-an sample quality across various domains (Nichol et al., 2022; Luo & Hu, 2021; Li et al., 2022). Building upon these successes, several works (Janner et al., 2022; Ajay et al., 2023; Liang et al., 2023) have leveraged diffusion to model entire tra- jectories in sequential decision-makinn tasks. By avoiding step-by-step autoregressive prediction, these approaches re- duce error accumulation and naturally capture long-range dependencies. Moreover, by leveraging guided sampling (Dhariwal & Nichol, 2021), diffusion planners can sample trajectories biased toward high-return behaviors, achiev- ing notable performance on standard offline reinforcement learning (RL) benchmarks (Fu et al., 2020).
Despite these advantages, diffusion planners struggle to guarantee reliable and feasible plans due to their inherent stochasticity. Unlike deterministic models that produce consistent outputs for given input, diffusion models gen- erate probabilistic trajectories. While this enables diverse sampling, it introduces the risk of generating physically im- plausible trajectories, which are referred to as artifacts in vision domains (Bau al., 2019; Shen al., 2020). Fur- thermore, the reward-guided sampling procedure used by unconditional diffusion planners, which jointly defines the sampling distribution and the energy function, can be hard to estimate accurately (Lu et al., 2023b). This inaccuracy may cause intermediate trajectory samples far away from the underlying data manifold during denoising, ultimately producing infeasible or low-quality trajectories and preclud-
1. Introduction
Planning over long horizons is crucial for autonomous sys- tems operating in real-world settings, where rewards are sparse and actions often entail delayed consequences (Lee et al., 2023a). When environment dynamics are fully known, methods such as Model Predictive Control (Tassa et al., 2012) and Monte Carlo Tree Search (Silver al., 2016; 2017; Lee et al., 2018) have achieved remarkable success. In most practical applications, however, the dynamics are
Kooraa Advanced Institute of Science and Technology (KAIST) "INEEJ.. Correspondence to: Jacsik Choi <jac- sho. kai>..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Codes available at gi github. com/ leekwoon/ooma

---

Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning
ing planners from being useful in safetycritical applications. To address these limitations, recent works (Lee et 2023b; Feng et al., 2024) introduced trajectory-refinemmnt strate- gies to enhance sample quality, though the challenge of ensuring fully reliable plans remains an open problem.
2.2. Planning with Diffusion Models
Diffusion planners (Janner et al., 2022) utilize diffusion probabilistic models (Sohl-Dickstein et al., 2015; Ho et al., 2020) model trajectory distribution as a Markov chain with Gaussian transitions:
In this paper, we demonstrate that manifold deviation occurs during diffusion sampling by establishing a lower bound on the guidance estimation error. To address this challenge, we introduce Local Manifold Approximation and Projection (LoMAP), a training-free method that projects guided sam- ples back onto low-rank subspace approximated from offline datasets. LoMAP operates entirely at test time, re- quiring no additional training. At each ewvrse-diffusiig step, retrieves a few offline trajectories closest the de- noised version of the current sample, corward-diffusee these neighbors, and applies principal component analysis (PCA) to span local low-dimensional subspace. The current sample is then projected onto this subspace, thereby signifi- cantly reducing off- off-manifold deviations and improving the likelihood of generating valid behaviors. Because LoMAP only adds a simple projection step after each reward-guider update, it is easily integrated into existing diffusion planners and effectively prevents manifold deviation.
M
po(T") p
II Po (T') d
(1)
where (T+ ) is standard Gaussian prior, 7 is noise-free
trajectory, and po(r"=((r') is denoising process which is learnable Gaussian transition:
Pe |+) = NN (o(''), E
(2)
This reverse process inverts a forward process that incre- mentally corrupts the data with Gaussian noise according to a variance schedule M.
q(T')-'~)) = N 1
(3)
One useful property is the ability to directly sample r from 70 any diffusion timestep i:
q(T' | T =
(1))))
(4)
where O; == I - Bz). The variance schedule is designed to respect aM R 0 so that MM becomes close to NO,, I).
Our main contributions are as follows: (1) We illustrate the manifold deviation issue in diffusion planners by deriv- ing a theoretical lower bound on the guidance estimation error. (2) We propose Local Manifold Approximation and Projection (LoMAP), training-fress plug- boug-an--lay mod- ule for diffusion planners that mitigates manifold deviation through local low-rank projections. (3) We demonstrate the effectiveness of LoMAP on standard offline RL benchmarks, particularly in challenging AntMaze task.
During the training procedure, rather than directly param- eterizing us. Diffuser trains noise-predictor model eæ to predict the noise e that was added to corrupt 7 into T (Ho et al., 2020):
((0) == E c
(5)
where T = Va,T + v 0gE and e ~ N(O..)..
2. Background
Trajectory optimization as guided sampling. To gen- erate trajectories that have high returns, the energy-guided sampling can be considered:
2.1. Problem Setting
We consider a Markov decision process (MDP) described by the tuple (8,A, A, P,r,7). Here, S is the state space, A is the action space, P S A x S [0, +00) is the transition model, r S x A + I is reward function, and E [0, 1] is the discount factor. Given a planning horizon T, the objective of trajectory optimization is to find the action sequence u that maximizes the expected return:
Pa (T())) x po(7") exp (J(T"))
(6)
Therefore, Diffuser trains a separate regression network Jo that predicts the return J(7") based on a noisily perturbed version -¹. This is accomplished through the mean-square- error (MSE) objective:
min Rii, - @
During the sampling stage, a classifier guidance (Dhariwal & Nichol, 2021) is employed, incorporating gradients of Top into the reverse diffusion process in Eq. (2). Specifically, the mean is updated as follows:
T
0g.7- arg max I(T) = arg max an.T
7rt( az),
QO:T
0=
" 1 = =N 14
(8)
where T = (su, ag. $1, ai...........................a.. ST, aT) is a trajectory, and J(T) represents the expected return of the trajectory.
where g = that controls the strength of the guidance.
and w is the guidance scale
2

---

Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning
Manifold
Mi
M
Mi
Guidance
Py (t
Reflning Guidance
M1-1
Mi-1
Mi-1
Local Manifold
Approximation
#0|l1
Tangent
Me
M.
Mo
Space
Manifold
Projection
(a) Diffuser (Janner et al., 2022) (b) RGG (Lee et al., 2023)
(c) LoMAP (ours)
Figure I: A schematic overview of our approach, contrasted with Diffuser (Janner et al., 2022) and RGG (Lee et al., 2023b). As described in Section 3.1, inexact guidance arises in high-ding -igh-dimensional settings. causing deviations from the data manifold. RGG addresses this by refining samples via an OOD detection metric but relies finely tuned guidance steps. In contrast, our LoMAP framework projects guided samples back onto a local low-rank subspace (Section 3.2), ensuring that sampling remains closer to the data manifold at each diffusion step.
2.3. Tweedie's Formula for Denoising
In this section, we formalize the phenomenon of manifold deviation, a critical limitation of diffusion planners caused by inexact guidance (Section 3.1), and propose Local Man- ifold Approximation and Projection (LoMAP), training- free method to preserve trajectory feasibility (Section 3.2).
When samples are perturbed by Gaussian noise 7 ((T, d²I), Tweedie's 's formula (Robbins, 1992) provides Bayes-optimal denoised estimate for observations:
- V log p(F).
3.1. Manifold Deviation by Inexact Guidance
where p(+) = p(+))))(7)dr_ If we consider a discrete- time diffusion model with perturbation in Eq. (4), we can get the posterior mean by rewriting Tweedie's fomula (Chung al., 2023; 2022):
Recall from Equation (6) that diffusion planners aim to ple trajectories biased toward those that have high returns by sampling from the following mnrgg--uidded distribution:
E E = va (T' (1-ai) V logp(r'))
Po( x po(+") (x)))(r)))
(10)
where (T(7 the learned trajectory distribution from the diffusion model, and J (T²) is the (negative) energy or return function to be maximized.
Vaa (T' VI aj cT())))
where the scaled score function estimated by ca: V, logp('')) -esa(i/////-----a/
Following Theorem 3.1 in (Lu al., 2023b), consider the forward process q(>' (9). Then the marginal distribution
at diffusion timestep i is:
2.4. Low-dimensional Manifold Assumption
po(T") / a(( )o) dr
High-dimensional trajectory often exhibits intrinsic low- dimensional structure. formalize this through the fol- lowing assumption:
= / / )p( ( (TO)
Z dr
Assumption 2.1. (Low-dimensional Manifold Assumption). The set of clean data Mo lies k-dimensional subspace R with k < d.
TT
po(TT) / 9(T
Z dr
Under this assumption, recent study (Chung et al., 2022) has shown that the set of noisy data ! is inherently concentrated around a (d dimensional manifold M,-
(T)())))
7
x <P((T') exp foog E Ey(to) (e)(7))
3. Manifold-Aware Diffusion Planning
[e(Tt)
3

---

Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning
Algorithm 1 Manifold-AAare Diffusion Planning
Algorithm LoMAP (T. [.T
1:
Require Diffuser pe, guide Ta, covariances E', scale w,
Imput: Noisy trajectory sample r' offline dataset
offline dataset [ n=1 number of neighbors k
{ (Tin=- number of neighbors k
2: while not done do
2: Output: Projected sample -'
3: Observe state initialize plan T. 1)0,1)
3: // denoise the current sampie
4: for i = N. N...., do
4: 4 0[(+1) Yai - ai a (6(T
-I
5:
// parameters of reverse transition
// A nearest ne neighbors
6:
μ + --((TT)))
N TopKNeighbors + O(( {T
7:
// guide using gradients return
// forward-diffuse neighbors
8:
7-1 ~ NN(# E' V WnJo (H) E))
for T E do
9:
// mani fold projection on ((OMAP)
10:
+ LoMAP(T' (TH). k)
9: T(n,)
i-1 -aiiii (mg)) + vI 0i-1 ((z)
11:
// constrain first state of plan
10: end for
12:
11: // PCA on the local neighborhood
12: U + (PCA(( ((T(-1))
13:
end for
14:
Execute first action of plan T
13: // project ento local subspace
14: r' UU -UUTT-1 -i-1
end while
15: return (-1
Hence, the exact intermediate guidance at diffusion timestep is given by the gradient
7: J.IIE (T)) be the estimated guidance via an MSE- based objective. We define the guidance gap at T by
J((T') = log ()( [xxp]]]))]))(()(()))))))))))
A guidaacce (muidance(7") = V. J((T)) V JM
(13)
By injecting this exact guidance into each reverse diffusion step, the sampled T" exactly follows the desired distribu- tion Eq. (6) by rewriting the score of
To study how inaccuracies in energy guidance grow with dimensionality, we introduce the guidance gap in Eq. (13). Proposition 3.2 shows that this gap has a lower bound on the order of vd in digensimmnsi regimes.
V e log Po(T))
V log - (+TT(TT)) (11)
guidance
Proposition 3.2. (Dimensional scaling of guidance gap.) Suppose I (79) is not constant. Given the true guidance
where the first term is approximated by the learned noise- predictor model cØ.
((T)) V log g(g E [
Guidance Gap. In practice, however, many existing dif- fusion planners (Janner et al., 2022; Liang et al., 2023; Chen et al., 2024) learn an approximate guidance function JMSE (7¹) via mean-square-error (MSE) objective:
V T((T')
E 9 T(T]]
and the MSE-based guidance
J MSE (T') E Eg(T= (r)) log
MSE
min E [I]TA(T*) - )[])
(12)
there exists a choice of T such that
where vatT" + /1 age and (0,
V+. Ji(T') V JMSE
Vd.
However, with sufficient model capacity, the optimal To under the MSE objective satisfies:
V1 as
for some constant > 0 that does not depend on d.
MSE (T*) = E g((70 (+) (T ④
< log B (+)) [T0]] Ji(T'),
Proof Sketch. The difference between the true guidance and the MSE-based guidance can be expressed as an expecta-
tion involving 5(T²) = ( (
where the inequality follows from Jensen's inequality, im- plying that MSE underestimates the desired quantity.
(())))))) [(70) times the forward-process noise c. By Jensen's inequality, 5(7") has positive mean, indicating that the MSE-based guidance un- derestimates the exponential weightee return. Exploiting
Definition 3.1. (Guidance gap). Let v J((T') denote the true intermediate guidance at diffusion step i, and let

---

Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning
the typical behavior ||e|a & vd in high dimensions and choosing r' so that ō aligns well with 6, we derive the guid- ance gap scaling on the order of vd. For the complete proof, see Appendix A.
RGG
Diffrrer
Dillowr
Mane2D Mediur
Mooe2D-Meddun
Maao2D- Large
0.6
Artffutt Ratio
0.4-
0.3
Consequently, as indicated by Proposition 3.2, this issue becomes more severe in scenarios involving long planning horizons and high-dimensional state and action spaces. The substantial guidance gap forces r' to drift from the inter- mediate data manifold M leading sampled trajectories away from the feasible manifold problem we refer to manifold deviation. Figure 4 provides empirical evidence of this issue.
0.0
15
20
25
0.0
8
15
20
of Plans
# of Plans
Figure 2: Artifact ratios on Maze2ID-Medium (left) and Maze2D-Large (right) as the number of sampled plans in- creases. The y-axis denotes the fraction of trajectories that pass through walls, making them infeasible. Across both tasks, our LoMAP augmented Diffuser" consistently pro- duces fewer artifact plans compared to Diffuser and RGG.
3.2. Local Manifold Approximation and Projection
Because each 7-11 remains close to the manifold at (ny)
As explained in Section 3.1, inaccuracies in the energy- guided update can cause the sample r' to deviate from the underlying data manifold. To mitigate manifold devia- tion caused by inexact guidance, we propose LoMAP a training-free method that projects guided samples back to the data manifold through local low-rank approximations. The key insight is that while intermediate diffusion samples T' may deviate from the manifold, their denoised estimates can guide local manifold approximation using the offline dataset.
timestep i - 1, these k samples approximate the local neighborhood M,-1- We then perform a rank-r PCA on { (- to obtain an orthonormal basis U E Rdxr, The matrix U spans an r-dimensional subspace that approxi- mates M,-- Thus,
PT,-- M.), (2) z.
which retains only the principal directions of variation sup- ported by the offiine data. In practice, r < d, and we choose r by retaining the principal components that explain at least fraction À of the total variance. In practice, we find that setting À = 0.99 works well. Pseudocode for the manifold- aware planning method is provided in Algorithm 1. Notably, our LoMAP module is entirely training-frre and can be readily integrated into existing diffusion planners by simply adding a manifold-projection step after each reward-guided update. For implementation details, including efficient man- ifold approximation and projection, see Appendix F.
Manifold-aware guidance. Given a trajectory sample T', we sample -'-1 with manifold-aware guidance in two steps:
- N [ (((()'++'''S )
(14)
(15)
where g = Vyi-1 ( JMMP((TT)) is the gradient-based guid- ance term, w is the guidance scale, and PT 1,, denotes projection onto the local manifold. Eq. (14) applies a reward- guided shift to sample 1 while Eq. (15) projects -'-1 onto the diweensiiio subspace derived from the offline dataset, mitigating drift away from feasible trajectories.
4. Experiments
In this section, we present experimental results showing that augmenting prior diffusion planners with LoMAP improves planning performance across a variety of offline control tasks. Specifically, we demonstrate (1) that LoMAP effec- tively mitigates manifold deviation and filters out artifact trajectories, (2) that it further enhances planning perfor- mance when integrated into diffusion planner, and (3) that LoMAP, as a alug-and-plar module, can be seamlessly in- corporated into hicrarchical diffusion planners, enabling successful planning in the challenging AntMaze domain. Additional details regarding our experimental setup and implementation are provided in Appendix E.
Approximating the local manifold. We estimate T Mi-1 using a local low-rank approximation from the offline dataset of feasible trajectories. To mitigate noise, we first form denoised surrogate
VI -1-ai-1 -
Vai-1
using Tweedie's formula (Eq. 9), where eg is the trained noise-prediction network. We then retrieve k nearest neighbors of f0/s-1 from the clean offline trajectories, ( using cosine similarity in trajectory space fol- lowing (Feng et al., 2024). Next, we forward diffuse these clean neighbors to timestep i 1-1:
4.1. Mitigating Manifold Deviation
To investigate whether LoMAP effectively mitigates mani- fold deviation in diffusion- based planners, we apply it to Dif-
(mn) /a-1 Thn + / a4-1 E(ny): E(n,) ~ IO.II..
5

---

Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning
Table I: Comparison on Maze2D for Diffuser", Diffuser, and prior methods. Diffuser denotes Diffuser augmented with LoMAP. We report the mean and the standard error over 1000 planning seeds.
L
(a) Diffuser
(b) RGG
(c) Diffuser" (ours)
Environment
IQL
RGG
TAT
Diffuser
Diffuser ?
U-Maze
47.4
108.8
114.5
113.9
126.01026
Visualization of 100 sampled trajectories from Diffuser, RGG, and Diffuser in Maze2D, under specified start and goal + condition.
Figure 3:
Maze2d Medium
34.9
131.8
130.7
121.5
131.0#t046
Large
58.6
135.4
133.4
123.0
151.9
Single-task Average
47.0
125.3
126.2
119.5
136.3
U-Maze
24.8
128.3
129.4
128.9
1:041
fuser (Janner et al., 2022) and refer to the resulting method as Diffuser". We conduct a quantitative evaluation of man- ifold deviation in Maze2D tasks by leveraging an oracle that identifies artifact plans, defined as trajectories that pass through walls and are thus physically infeasible. We compare Diffuser against Diffuser and a baseline variant, Restoration Gap Guidance (RGG) (Lee et al., 2023b), in tasks where the planner is given only a start and goal loca- tion. Specifically, we randomly select start and goal states and generate trajectories under increasing sample sizes. For each start-goal pair, if at least one sampled trajectory con- tains invalid transitions through walls, we mark that pair as exhibiting manifold deviation. As shown in Figure 2, Dif- fuser produces some infeasible transitions, especially in the more complex Maze2ID-Large environment. Although RGG partially alleviates this issue, Diffuser" demonstrates the highest reliability, consistently generating valid trajectories even when a larger number of plans are drawn.
Multi2d Medium
12.1
130.0
135,4
127.2
129.1 1
Large
13.9
148.3
143.8
132.1
154.71279
Multi-task Average
16.9
136.4 136.2
129.4
138.9
task settings. LoMAP effectively reduces manifold devia- tion (Section 4.1) and provides an additional performance boost, with Diffuser" achieving the best results on out of 6 tasks, showing especially strong improvements in Maze2D- Large, which features more complex obstacle maps.
Locomotion. We next evaluate LoMAP-incorporatte planners on MuJoCo locomotion tasks (Fu et al., 2020), standard benchmarks for assessing performance on het- erogeneous, varying-quality datasets. Our comparison in- cludes model-free algorithms (CQI. (Kumar et al., 2020), IQL (Kostrikov et al., 2022)), model-based algorithms (MOPO (Yu et al., 2020), MOReL (Kidambi et al., 2020)), and sequence modeling approaches (Decision Transformer (DT) (Chen et al., 2021), Trajectory Transformer (TT) (Jan- ner et al., 2021)). As baseline diffusion planners, we con- sider Diffuser (Janner et al., 2022), RGG (Lee et al., 2023b), TAT (Feng et al., 2024), and a conditional variant, Decision Diffuser (DD) (Ajay et al., 2023).
This phenomenon is further illustrated in Figure 3. Although RGG removes many artifact plans, it also reduces the di- versity of solutions, clustering trajectories near a narrower set of paths. By contrast, Diffuser" maintains high reliabil- ity and diversity, producing physically feasible trajectories without sacrificing coverage of the solution space.
As shown in Table 2. incorporating LoMAP consistently boosts average returns of Diffuser across all tasks, with particularly strong gains in the Medium dataset, which poses suboptimal and challenging distribution for learning both the diffusion planner and return estimator. Moreover, LoMAP -iccopprcoppr planners outperform other trajectory- refinement methods, highlighting the benefits of addressing manifold deviation during sampling.
4.2. Enhancing Planning Performance
Maze2D. To further demonstrate how LoMAP improves planning performance, we evaluate it on Maze2D environ- ments (Fu al., 2020), which involve navigating an agent to target goal location through complex mazes requiring long-horizon planning. Maze2D features two distinct tasks: a single-task setup where the goal location is fixed, and multi-task variant (Multi2D) in which the goal is random- ized at the start of each episode. We compare our methods against the model-free offline RL algorithm IQL (Kostrikov al., 2022) and two diffusion planners, RGG (Lee al., 2023b) and TAT (Feng et al., 2024).
4.3. Scaling to Hierarchical Planning in AntMaze
The AntMaze tasks (Fu et al., 2020) pose a substantial chal- lenge due to high- eigh-dimensional state and action spaces, long- horizon navigation objectives, and sparse rewards. Gener- ating entire trajectories often results in infeasible plans in these environments. A promising approach is to adopt a hierarchical scheme, wherein a high-level diffusion planner proposes subgoals and a low-level diffusion planner exe- cutes hooizon trajectories to move the agent from one subgoal to the next.
approaches for
As shown in Table 1, the model-free IQL suffers a notable performance drop under multi-task conditions, likely due to the challenges of credit assignment. By contrast, diffusion- based planners perform well in both single-task and multi-
6

---

Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning
Table 2: Performance comparison of Diffuser and various prior methods on MuJoCo locomotion tasks, reported as normalized average returns with corresponding standard errors over 50 planning seeds.
Dataset
Environment
BC
CQL
IQL
DT
TT
MОPO
MOReL
DD
TAT
RGG
Diffuser Diffuser
HalfCheetah
55.2
91.6
86.7
86.8
95.0
63.3
53.3
90.6
92.5
90.8
88.9
91.1.0.1
Med-Expert
Hopper
52.5
105.4
91.5
107.6
110.0
23.7
108.7
111.8
109.4
109.6
103.3
110.6:0.29
Walker2d
107.5
108.8
109.6
108.1
101.9
44.6
95.6
108.8
108.8
107.8
106.9
109.2
HalfCheetah
42.6
44.0
47.4
42.6
46.9
42.3
42.1
49.1
44.3
44.0
42.8
45.4
Medium
Hopper
52.9
58.5
66.3
67.6
61.1
28.0
95.4
79.3
82.6
82.5
74.3
93.7
Walker2d
75.3
72.5
78.3
74.0
79.0
17.8
77.8
82.5
81.0
81.7
79.6
19.9
HalfCheetah
36.6
45.5
44.2
36.6
41.9
53.1
40.2
39.3
39.2
41.0
37.7
39. 10,99
Med-Replay
Hopper
18.1
95.0
94.7
82.7
91.5
67.5
93.6
100
95.3
95.2
93.6
97.6+0.58
Walker2d
26.0
77.2
73.9
66.6
82.6
39.0
49.8
75
78.2
78.3
70.6
78.7+2..
Average
51.9
77.6
77.0
74.7
78.9
42.1
72.9
81.8
81.3
81.2
77.5
82.8
Table 3: Performance comparison of Diffuser", HDD", and prior approaches on AntMaze tasks, reported normalized average returns with corresponding standard errors over 150 planning seeds.
Dataset Eny
DD RGG Diffuser Diffuser" P HD HID P
Medium
8.0 17.3
6.7
40.7+43
42.0 92.7
Play
Large
0.0
12.7
17.3
20.7.38
54.7 74.0
Diverse Medium 4.0 25.3
2.0
36.0427
78.7 98.0 +6.1
Large 0.0 17.3
27.3
39.3425
46.0 82.0
Average 3.0 18.2
13.3
34.2
55.3 86.7
HD
(b) HDP (ours)
Building on this idea, we incorporate LoMAP into both Diffuser (Janner et al., 2022) and Hierarchical Diffuser (HD) (Chen et al., 2024), yielding Diffuser" and HDP. In HD?. a high-level diffusion model augmented with LoMAP generates subgoals for each trajectory segment. Subsequently, short-horizon diffusion model (Diffuser) translates these subgoals into lower-level actions. We com- pare Diffuser? and HD against standard Diffuser (Janner et al., 2022), Hierarchical Diffuser (HD) (Chen et al., 2024), Restoration Gap Guidance (RGG) (Lee et al., 2023b), and Decision Diffuser (DD) (Ajay et al., 2023).
Figure 4: Visual comparison of generated plans the AntMaze environment. The goal is marked by red sphere. We plot 20 sampled plans in different colors. (a) shows plans generated by Hierarchical Diffuser (HD) (Chen et al., 2024), which often produces infeasible trajectories that pass through walls. (b) demonstrates the results of HD aug- mented with LoMAP, which respects the environment ge- ometry and generates more reliable, feasible trajectories.
pronounced in larger mazes, where longer horizons and in- tricate navigation paths make adherence to valid manifold particularly critical. Consequently, LoMAP serves as a plug- and-play component that boosts planning performance even in multi-level hierarchical settings.
As shown in Table 3, Diffuser improves upon Diffuser across all AntMaze tasks, demonstrating ability of LoMAP to maintain manifold feasibility even in high-dimensional continuous control. Notably, HDd achieves the best results on every variant of AntMaze, substantially outperforming the original HD. We attribute these gains largely to the cor- rection of manifold deviation during high-level planning by LoMAP. In standard HD, subgoals generated by the high- level planner can sometimes lie off off-manifold, forcing the low-level planner to produce infeasible trajectories. As il- lustrated in Figure 4, these subgoals frequently pass through maze walls, leading to invalid paths. By applying LoMAP to refine them, HD ensures that each proposed subgoal is more feasible for the low-level planner, thereby boosting overall success rates. These improvements are especially
4.4. Generating Minority Sample
The ability to generate minority data can critical in real-world scenarios where increasing the diversity of rare- condition examples can improve predictive performance. However, any minority samples must still align with the true data distribution rather than represent artifacts. To explore whether our method can facilitate the generation of feasible minority samples in low low-density regions, we adopt minor- ity guidance (Um et al., 2024), which provides additional guidance toward low-density regions.
7

---

Local Manifold Approximation and Projection for Manifold-Aware Diffusson Planning
L TLL H
planning (Li et al., 2023; Chen et al., 2024), multi-task RL (He et al., 2023; Ni et al., 2023), and multi-agent set- tings (Zhu et al., 2024).
@
Despite these advances, diffusion planners remain suscep- tible to stochastic failures, occasionally producing trajec- tories that deviate from the feasible manifold. Although some works mitigate this issue by refining trajectories post hoc (Lee et al., 2023b; Feng et al., 2024), a robust, training- free approach to consistently maintain manifold adherence throughout the sampling process has yet to be established.
(a) Diffuser
(b) Diffuser
(c) Diffuser (ours)
minority guidance
+minority guidance
Figure 5: Sampling from low-density regions in Maze2D using minority guidance (Um et al., 2024), given a specified start and goal 4 condition.
As shown in Figure 5, Diffuser alone sometimes fails to capture alternative feasible paths, leading to poor coverage despite viable shortcuts. While minority guidance improves coverage. it also tends to introduce infeasible trajectories. In contrast, LoMAP mitigates this issue by refining these trajectories and ensuring they remain on the valid mani- fold. Consequently, combining LoMAP with minority guid- ance can help uncover feasible yet unexplored solutions that might otherwise remain inaccessible to standard diffu- sion planners. Investigating how this approach can further enhance planning is promising direction for future work.
Projections in Diffusion Models Several works in the image-generation domain have introduced projection tech- niques to mitigate off- mfnifoldo updates during diffusion sampling. For instance, MCG (Chung et al., 2022) projects measurement gradients onto the data manifold in inverse problems, guided by Tweedie's formula. DSG (Yang et al., 2024) replaces the random Gaussian step with a determinis- tic update constrained to hypersphere. This avoids deviat- ing from the intermediate diffusion manifold, allowing for substantially larger guidance steps. Meanwhile, MPGD (He et al., 2024) employs a pre-trained autoencoder to learn the data manifold and projects the sample onto the tangent space of the clean data manifold via a pre-trained autoencoder. However, the performance of MPGD is heavily depends on the expressive power of the autoencoder. As a result, it is difficult to deploy MPGD in diverse offline RL tasks, particularly where pre-trained autoencoders are unavailable. In contrast, our proposed method is entirely training-free projecting samples onto both the clean and intermediate diffusion manifolds using only local approximations from the offline dataset.
5. Related Work
Diffusion Planners in Offline Reinforcement Learning Diffusion probabilistic models (Sohl-Dickstein al., 2015; Ho et al., 2020) have recently gained prominence in rein- forcement learning (RL), particularly in the offline setting. By iteratively denoising samples from noise, these models learn the gradient of the data distribution (Song& & Ermon, 2019), bridging connections to score matching (Hyvärinen & Dayan, 2005) and energy-based models (EBMs) (Du & Mordatch, 2019; Grathwohl al., 2020). Their expres- sive power in modeling complex, high-dimensional data has led to applications as planners (Janner et al., 2022; Ajay et al., 2023), policies (Wang et al., 2023), and data synthe- sizers (Lu et al., 2023a; Wang et al., 2025).
6. Conclusion
In this work, we investigated the manifold deviation issue that arises in diffusion-based trajectory planning, where inaccurate guidance causes sampled trajectories to devi- ate from the feasible data manifold. To address this, we introduced Local Manifold Approximation and Projection (LoMAP), a training-fres method that employs local, low- rank projections to constrain each reverse diffusion step to the underlying data manifold. By ensuring that inter- mediate samples remain close to this manifold, LoMAP substantially reduces the risk of generating infeasible or low-quality trajectories. Empirical results on various of- fline RL benchmarks demonstrate the effectiveneess of our approach. Additionally, LoMAP can be incorporated into hierarchical diffusion planning for more challenging tasks such as AntMaze. Overall, our results establish LoMAP as an easily integrable component for diffusion- ners, empowering them to consistently remain on the data manifold and thereby providing safer and more robust long- horizon trajectories.
Diffuser (Janner et al., 2022) pioneered the use of diffu- sion models for planning by generating entire trajectories, demonstrating notable flexibiliiyy in long-horizon tasks. Con- cretely, unconditional diffusion model is trained on of- fline trajectories and paired with a separate network that estimates returns; this network then guides trajectory sam- ples toward high-return regions during inference (Dhariwal & Nichol, 2021). Extending this framework, Decision Dif- fuser (Ajay et al., 2023) applies classifier-free guidance, conditioning the diffusion model directly on reward or con- straint signals and thereby removing the need for a sepa- rately trained reward function. Meanwhile, AdaptDiffuser (Liang et al., 2023) progressively fine-tunes the diffusion model with high-quality synthetic data, improving gener- alization in goal-conditioned tasks. Beyond these efforts, diffusion models have also been employed in hierarchical
plan-
8

---

Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning
Acknowledgements
constraints. In Advances in Neural Information Process- ing Systems (NeurIPS), 2022.
This work was supported by Institute of Information & com- munications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-00075, Artificial Intelligence Graduate School Program (KAIST); No. 2022-0-00984, Development of Artificial Intelligence Technology for Personalized Plug-and-Play Explanation and Verification of Explanation; No. RS- RS-2024-0055788, AI Re- search Hub Project; No. RS-2024-00509258, Al Guardians: Development of Robust, Controllable, and Unbiased Trust- worthy AI Technology).
Chung, H., Kim, J., Mccann, M. T., Klasky, M. L.,, and Ye, J. C. Diffusion posterior sampling for general noisy in- verse problems. In International Conference on Learning Representations (ICLR), 2023.
Clark, K. Vicol, P., Swersky, K., and Fleet, D. J. Directly fine-tuning diffusion models on differentiable rewards. In International Conference on Learning Representations (ICLR), 2024.
Co-Reyes, J., Liu, Y., Gupta, A., Eysenbach, B., Abbeel, P., and Levine, S. Self-consistent trajectory autoencoder: Hi- erarchical reinforcement learning with trajectory embed- dings. In International Conference on Machine Learning (ICML), 2018
Impact Statement
This paper advances the field of Machine Learning through a new method for diffusion- trajectory planning. We do not identify any direct negative societal impacts that must be specifically highlighted. However, we encourage practitioners apply this work responsibly and evaluate real-world safety implications before deployment.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Dong, Z., Hao, J., Yuan, Y., Ni, F., Wang, Y., Li, P., and Zheng, Y. Diffuserlite: Towards real-time diffusion plan- ning. In Advances in Neural Information Processing Systems (NeurIPS), 2024a.
References
Ajay, A., Du, Y., Gupta, A., Tenenbaum, J. B., Jaakkola, T. S., and Agrawal, P. Is conditional generative model- ing all you need for decision making? In International Conference on Learning Representations (ICLR), 2023.
Dong, Z.,, Yuan, Y., Hao, J., Ni, F., Ma, Y., Li, P., and Zheng, Y. Cleandiffuser: An easy-to-use modularized library for diffusion models in decision making, arXiv preprint arXiv:2406.09950 2024b.
Asadi, K., Misra, D., and Littman, M. Lipschitz continuity in model-based reinforcement learning. In International Conference on Machine Learning (ICML), 2018.
Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.-E., Lomeli, M., Hosseini, L., and Jégou, H. The faiss library. arXiv preprint wxXiv:2401.0828) 2024.
Bau, D., Zhu, J.-Y., Strobelt, H., Zhou, B., Tenenbaum, J. B., Freeman, W. T., and Torralba, A. Gan dissection: Visual- izing and understanding generative adversarial networks. In International Conference on Learning Representations (ICLR), 2019.
Du, Y. and Mordatch, I. Implicit generation and model- ing with energy based models. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Cardoso, G., Idrissi, Y. J. E.,, Corff, S. L., and Moulines, E. Monte carlo guided diffusion for bayesian linear in- verse problems. In International Conference on Learning Representations (ICLR), 2024.
Fan, Y., Watkins, O., Du, Y., Liu, H., Ryu, M., Boutilier, C., Abbeel, P., Ghavamzadeh, M., Lee, K., and Lee, K. Rein- forcement learning for tne-ttunng text-to-image diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Chen, C., Deng, F., Kawaguchi, K., Gulcehre, C., and Ahn, S. Simple hierarchical planning with diffusion. In Interna- tional Conference on Learning Representations (ICLR), 2024.
Feng, L.,, Gu, P., An, B., and Pan, G. Resisting stochastic risks in diffusion planners with the trajectory aggregation tree. In Internationall Conference on Machine Learning (ICML), 2024.
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning, arXiv preprini arXiv:2004.07219 2020.
Chung. H., Sim, B., Ryu, D., and Ye, C. Improving diffusion models for inverse problems using manifold
Grathwohl, W., Wang, K.-C,, Jacobsen, J.-H., Duvenaud, D., and Zemel, Learning the stein discrepancy for training

---

Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning
and evaluating energy-based models without sampling. In International Conference on Machine Learning (ICML), 2020.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Con- servative q-learning for offiin reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off- Off-policy maximum entropy deep reinforce- ment learning with stochastic actor. In International Conference on Machine Learning (ICML), 2018.
Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved precision and recall metric for assess- ing generative models. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Hao. C., Lin, K., Luo, S., and Soh, H. Language sanguage-guided manipulation with diffusion policies and constrained in- painting. arXiv preprint Net::200609967,, 2024.
Lee, J., Yun, S., Yun, T., and Park, J. Gta: Generative trajectory augmentation with guidance for offline rein- forcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2024a.
He, H., Bai, C., Xu, K., Yang, Z., Zhang, W. Wang, D., Zhao, B., and Li, X. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Lee, K., Kim, S.-A., Choi, J., and Lee, S.-W. Deep rein- forcement learning in continuous action spaces: case study in the game of simulated curling. In International Conference on Machine Learning (ICML), 2018.
He, Y., Murata, N., Lai, C.-H., Takida, Y., Uesaka, T., Kim, D., Liao, W.-H., Mitsufuji, Y., Kolter, J. Z., Salakhut- dinov, R., and Ermon, S. Manifold preserving guided diffusion. In International Conference on Learning Rep- resentations (ICLR), 2024.
Lee, K., Kim, S., and Choi, J. Adaptive and explainable deployment of navigation skills via hierarchical deep rein- forcement learning. In 2023 IEEE International Confer- ence on Robotics and Automation (ICRA). pp. 1673-1679, 2023a.
Lee, K., Kim, S., and Choi, J. Refining diffusion planner for reliable behavior synthesis by automatic detection of infeasible plans. In Advances in Neural Information Processing Systems, 2023b.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion prob- abilistic models. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Hyvärinen, A. and Dayan, P. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Lee, K. M., Ye, S., Xiao, Q.. Wu, Z., Zaidi, Z., D' A'mbrosio, D. B., Sanketi, P. R., and Gombolay, M. Learning diverse robot striking motions with diffusion models and kine- matically constrained gradient guidance. arXiv preprint arXiv:2409. 2in::2409./5228. 2024b.
Janner, M. Fu, J., Zhang, M., and Levine, S. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Li, G., Shan, Y., Zhu, Z., Long, T., and Zhang, W. Diffstitch: Boosting offline reinforcement learning with diffusion- based trajectory stitching. In International Conference on Machine Learning (ICML), 2024.
Janner, M., Li, Q., and Levine, S. Offline reinforce- ment learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Li, W. Efficient planning with latent diffusion. In Interna- tional Conference on Learning Representations (ICLR), 2024.
Janner, M., Du, Y., Tenenbaum, J. B., and Levine, S. Plan- ning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning (ICML), 2022.
Li, W., Wang, X., Jin, B., and Zha, H. Hierarchical diffusion for offline decision making. In International Conference on Machine Learning (ICML), 2023.
Li, X., Thickstun, J., Guirajani, I., Liang, P. S., and Hashimoto, T. B. Diffusion-Im improves controllable text generation. In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2022.
Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. Morel: Model-based offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Liang, Z., Mu, Y., Ding, M., Ni, F., Tomizuka, M., and Luo, P Adaptdiffuser: Diffusion models as adaptive self-evolving planners. In International Conference on Machine Learning (ICMI., 2023.
Kostrikov, I., Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q- q-learning. In International Conference on Learning Representations (ICLR), 2022.
10

---

Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning
Lu, C., Ball, P., Teh, Y. W., and Parker-Holder, J. Syn- thetic experience replay. Advances Neural Information Processing Systems, 2023a.
Prabhudesai, M., Goyal, A., Pathak, D., and Fragkiadaki, K. Aligning text-to-image diffusion models with re- ward backpropagation. arXiv preprint arXiv:2310.03739, 2024.
Lu, C., Chen, H., Chen, J., Su, H., Li, C., and Zhu, J. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning (ICML), 2023b.
Robbins, H. E. An empirical bayes approach to statistics. In Breakthroughs in Statistics: Foundations and basic theory, pp. 388-394. Springer, 1992.
Shaoul, Y., Mishani. I., Vats, S., Li, J., and Likhachev, M. Multi-robot motion planning with diffusion models. In International Conference on Learning Representations (ICLR), 2024.
Lu, H.. Han, D., Shen, Y., and Li, D. What makes good diffusion planner for decision making? In International Conference on Learning Representations (ICLR), 2025.
Luo, S. and Hu, W. Diffusion probabilistic models for 3d point cloud generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Shen, Y., Gu, J., Tang, X., and Zhou, B. Interpreting the latent space of gans for semantic face editing. Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Luo, Y., Xu, H., Li, Y., Tian, Y., Darrell, T., and Ma, T. Algo- rithmic framework for model-based deep reinforcement learning with theoretical guarantees. In International Conferencee on Learning Representations (ICLR), 2019.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, Panneershelvam, V., Lanctot, M., et al. Mastering the game of with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
Luo, Y., Mishra, U. A., Du, Y., and Xu, D. Generative trajectory stitching through diffusion composition. arXiv preprint ar arXiv:2503.05153 2025.
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I. Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M.. Bolton, A., et al. Mastering the game of go without human knowledge. nature, 550(7676):354-3555 2017. 2017.
Mishra, U. A., Xue, S., Chen, Y., and Xu, D. Generative skill chaining: Long-horizon skill planning with diffusion models. In Conference on Robot Learning, pp. 2905- 2925. PMLR, 2023.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, Deep unsupervised learning using nonequi- librium thermodynamics. International Conference on Machine Learning (ICML). 2015.
Na, B., Kim, Y., Park, M.. Shin, D., Kang, W., and Moon, L-C. Diffusion rejection sampling. In International Con- ference Machine Learning (ICML). 2024.
Song, J., Zhang, Q., Yin, H., Mardani, M., Liu, M.-Y., Kautz, Chen, Y., and Vahdat, A. Loss-guided diffusion models for plug-and-play controllable generation. In International Conference on Machine Learning (ICML), 2023.
Ni. F., Hao, J., Mu, Y.. Yuan, Y., Zheng, Y.. Wang, B., and Liang, Z. Metadiffuser: Diffusion model as conditional planner for offline meta-rl. In International Conference on Machine Learning (ICML), 2023.
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M. Glide: To- wards photorealistic image generation and editing with text-guided diffusion models. In International Confer- ence on Machine Learning (ICML), 2022.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Song, Y.. Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021.
Park, S., Ghosh, D., Eysenbach, B., and Levine, S. Hiql: Offline goal-conditioned rl with latent states as actions. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Sutton, R. S. Reinforcement learning: An introduction. A
Bradford Book, 2018
Park, S., Frans, K., Eysenbach, B., and Levine, S. Ogbench: Benchmarking offline goal-conditioned rl. In Interna- tional Conference on Learning Representations (ICLR), 2025.
Talvitie, E. Model regularization for stable sample roll- outs. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), 2014.

---

Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning
Tassa, Y., Erez, T., and Todorov, Synthesis and stabi- lization of complex behaviors through online trajectory optimization. In 2012 JEEE/RSJ International Confer- ence on Intelligent Robots and Systems, pp. 2012.
Ziebart, B. D., Maas, A. L.., Bagnell, J. A., Dey, A. K., et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008,
Um, S., Lee. S., and Ye, C. Don't play favorites: Mi- nority guidance for diffusion models. In International Conference Learning Representations (ICLR), 2024.
Voelcker, C., Liao, V., Garg. A., and Farahmand, A.- m. Value gradient weighted model-based reinforcement learning. In International Conference on Learning Rep- resentations (ICLR), 2022.
Wang, R., Frans, K., Abbeel, P., Levine, S., and Efros, A. A. Prioritized generative replay. In International Conference Learning Representations (ICLR), 2025.
Wang, Y., Wang, L., Du, Y., Sundaralingam, B., Yang, X., Chao, Y.-W., Perez-D' Arpino, C., Fox, D., and Shah, J. Inference-time policy steering through human interac- tions. arXiv preprint arXiv:2411. 2024.
Wang, Z., Hunt, J., and Zhou, M. Diffusion policies as an expressive policy class for offline reinforcement learning- In International Conference Learning Representations (ICLR), 2023.
Wu, L., Trippe, B., Naesseth, C., Blei, D., and Cunning- ham. J. P. Practical and asymptotically exact conditional sampling in diffusion models. In Advances Neural Information Processing Systems (NeurIPS), 2023.
Yang. L., Ding, S., Cai, Y., Yu, J., Wang, J., and Shi, Y. Guidance with spherical gaussian constraint for condi- tional diffusion. In International Conference on Machine Learning (ICML), 2024.
Yang. Q. and Wang. Y.-.. Ridiff: Reverse trajectory syn- thesis via diffusion for offline reinforeement learning. In International Conference Learning Representations (ICLR), 2025.
Yoon, J., Cho, H., Baek, D., Bengio, Y., and Ahn, S. Monte carlo tree diffusion for system planning. arXiv preprint 0022022502.07202. 2025.
Yu, T., Thomas, G., Yu, L.,, Ermon, S., Zou, J. Y., Levine, S., Finn, C., and Ma, T. Mopo: Model-based offline policy optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Zhu, Z., Liu, M., Mao, L., Kang, B., Xu, M., Yu, Y., Ermon, S., and Zhang, W. Madiff: Offline multi-agent learning with diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
12

---

Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning
A. Proofs
Proposition A.I. (Dimensional scaling of guidance gap.) Suppose I (T") )
not constant. Given the true guidance
E | e 3(T Te(')(+))
V Vrllgg((7"17)
E 90(79)o)|
and the MSE-based guidance
Ti J MIE (T') E B | J(T V (.:lg))))) (=)0g((T")(()
there exists choice of T' such that for some constant c that does not depend on d.
J((T') vi -v. (
c Vd
Vd,
ai
Proof. By the forward process at in Eq. (4),
t' ='=va Vai + VI asc, c~,
We have,
((-))T)) = N( r¹ 1- Ilda) V. logg(>"(77) = v1-aa ai 6.
Vai" a₁
Let us abbreviate the distribution uu(70) = (() Then
T J((T')= BB [x V+r log [([T)]
7 To MSE = ExT(J(T)) Vn. log)((T)).
(+)))
Subtracting these yields
eT(r
Vr. Te(Ti) V (T)) = [B,(e,,(e() -((T)) V. log
Since ((7") 1 VI-ai E, we obtain
1
V J((TT) - V (T') = VI as
E[5(79)
By Jensen's is inequality, 8(7²) has positive mean whenever I is not constant. Furthermore, grows, we have ||e lel2 on the order of vd. One can then choose a r' so that 5(70) remains well-aligned with e, giving
A guidancee (T') v1 az;
v1 ai Vd.
for some constant c. Thus we can complete the proof.
B. Limitations
While LoMAP provides simple yet effective approach for mitigating manifold deviations, it exhibits certain limitations. First, the current implementation uses cosine distance for manifold approximation, which may not be optimal in very high-dimensional state spaces, such as pixel-based observations. Developing more robust manifold approximation techniques suitable for complex, digh-dimensional environments remains an important direction for future research. For instance, combining LoMAP with latent trajectory embeddings (Co-Reyes et al., 2018) could be promising approach. Second, our
13

---

Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning
method inherently encourages sampled trajectories to stay close the offline data manifold, which may restrict exploration of novel behaviors. While our primary focus in this work is ensuring safe and reliable trajectory cnneeratoonppaticllaaan beneficial for safety-critical offline RL auplications-adddesiinnn this exploration limitation remains crucial. Integrating LoMAP with complementary methods such as trajectory stitching or data augmentation (Ziebart et al., 2008; Li et al., 2024; Lee et al., 2024a; Yang & Wang. 2025), which generate diverse synthetic trajectories, could alleviate this issue and is an interesting area for future study. Furthermore, exploring how LoMAP could be effectively extended to challenging benchmarks that explicitly require stitching and long-horizon reasoning, such as OGBench (Park et al., 2025), represents an intriguing future research direction.
C. Extended Related Work
Beyond hierarchical structures (Li et al., 2023; Chen et al., 2024), multi-agent setups (Zhu et al., 2024), and post-hoc trajectory refinement methods (Lee et al., 2023b; Feng et al., 2024), recent diffusion planners have explored integrating tree search methods (Yoon al., 2025), refining trajectory sampling techniques (Dong et al., 2024a), examining critical design choices to improve robustness (Lu et al., 2025), composing short segments into long-horizon trajectories at inference time (Mishra et al., 2023; Luo et al., 2025), efficient latent diffusion planning (Li, 2024), and inference-time guided generation (Wang et al., 2024; Lee et al., 2024b; Hao et al., 2024).
While diffusion models have achieved impressive performance various generative tasks, effectively steering them toward specific objectives remains challenging. Broadly, existing methods for aligning diffusion models can be categorized into two groups: fine-tuning methods and guidance- based methods. Fine-tuning methods, such as reinforcement learning-base -based tuning (Fan et al., 2023) or direct gradient optimization (Clark et al., 2024; Prabhudesai et al., 2024), directly update model parameters to maximize target objectives. Despite their effectiveness, these methods tend excessively focus on reward optimization, often compromising the diversity and fidelity of generated outputs (Clark al., 2024). Conversely, guidance-bart based methods offer simpler inference- time alternative that preserves the pretrained model distribution. Among these, classifier guidance (Dhariwal & Nichol, 2021) involves training an auxiliary classifier to guide the sampling process toward target conditions, but the additional training overhead can be costly. Recent training-free guidance approaches circumvent this by directly utilizing pretrained classifiers or reward predictors via approximate inference (Chung et al., 2023; Song et al., 2023; He et al., 2024). In particular, these methods commonly rely on Tweedie-based denoising (Robbins, 1992), which provides predictions of clean data given noisy samples. However, inaccuracies inherent to Tweedie's approximation limit its effectiveness, especially in accurately aligning diffusion samples with target objectives. Sequential Monte Carlo (SMC)-based approaches (Wu et al., 2023; Cardoso et al., 2024) address inaccuracies in guidance through principled probabilistic inference. Although these methods provide asymptotic exactness, their practical efficiency under limited sampling budgets remains sigmificant challenge.
D. Additional Results
Realism score evaluation. To further validate the effectiveness of LoMAP, compute the Realism Score (Kynkäänniemi al., 2019), which measures how closely generated trajectories lie to the true manifold defined by the offline dataset. Specifically, we approximate the true manifold using k-nearest neighbor (K-NN) hyperspheres constructed from 20,000 offline trajectories, and evaluate the average realism score over 100,000 sampled trajec- tories. As shown in Table 4, applying LoMAP consistently yields higher realism scores compared to diffusion sampling without LoMAP, demonstrating that LoMAP effectively produces trajec- tories closer to the true data manifold.
Table 4: Comparison of Realism Scores on Maze2D tasks. Higher realism scores indicate samples closer to the true data manifold.
Environment
Diffuser
Diffuser
Maze2D U-Maze
1.23
1.30
Maze2D Medium
1.40
1.56
Maze2D Large
1.36
1.47
Dynamic consistency evaluation. Additionally, we assess trajectory feasibility for MuJoCo locomotion tasks using the dynamic mean squared error (Dynamic MSE), defined as:
Dynamic MSE = (s, -s'13,
where f represents the true environment dynamics. As shown in Table 5, LoMAP consistently achieves lower Dynamic MSE compared to diffusion sampling without LoMAP, clearly indicating improved adherence to the true dynamics of the
14

---

Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning
environment.
Table 5: Dynamic MSE comparison on MuJoCo locomotion tasks. Lower Dynamic MSE indicates better adherence to true environment dynamics.
Environment
Diffuser
Diffuser P
halfcherrah-medduneepprr
0.363
0.295
hopper-meddium-epper
0.027
0.020
walker2d-meddum-eepprr
0.391
0.293
halfcheetahhmmedimm
0.352
0.285
hopper-medium
0.024
0.021
walker2d-medium
0.395
0.293
maffcheetta-meedium-ree
0.710
0.555
hopper-mediimm-replay
0.049
0.045
walker2d-medium- mediim--replay
0.829
0.506
Additional comparison with inference-time guidance methods. We further provide comparative evaluations against re- cent inference-time guidance methods, including stochastic sampling (Wang et al., 2024), constrained gradient guidance (Lee al., 2024b), and inpainting optimization (Hao et al., 2024).
For stochastic sampling (Wang et al., 2024), we adapted goal- conliiondii via MCMC sampling, tuning the number of sampling steps [2, 4, 8). To implement constrained gradient guidance (Lee et al., 2024b), we approximated maze walls as multiple spherical constraints following Shaoul et al. (2024), defining sphere- sphere-based based cost:
M
H1
Jx(T)
max (r dist(T,, Pm),
172.00
where H is the planning horizon, Pm the center of sphere constraints, and their radius. We tuned the guidance scale within the range (0.001, 0.01, 0.05, 0.1]. To compare with inpainting optimization (Hao al., 2024), emulated vision-language model (VLM)-based keyframe generation by training a high-level policy using Hierarchical Implicit Q-Learning (HIQL) (Park et al., 2023), The policy generated optimal subgoal sequences (keyframes), with k 25 steps, aligning with the official implementation provided by ogbench (Park et al., 2025).
Table 6 presents artifact ratio comparisons. LoMAP consistently achieves the lowest artifact ratio, significantly outperforming all inference-time guidance baselines. Even the constrained gradient approach (Lee al., 2024b), despite explicitly modeling maze walls, performed worse, likely due to gradient- brased projections struggling with nonconvex constraints. Both stochastic sampling (Wang et al., 2024) and inpainting optimization (Hao al., 2024) improved over Diffuser but still exhibited higher artifact ratios than LoMAP.
Table Artifact ratio comparison with inference-time guidance methods in Maze2D-Large. Lower values indicate fewer infeasible trajectories.
# of Plans
Diffuser (LoMAP, ours)
Diffuser
(Wang et al., 2024)
(Lee et al., 2024b)
(Hao et al., 2024)
10
0.35
0.50
0.42
0.49
0.43
20
0.35
0.62
0.44
0.54
0.46
30
0.38
0.66
0.47
0.61
0.49
Visual comparisons. We provide additional rollout visualizations. Figure 6 depicts rollouts executed by Diffuser (Janner et al., 2022) and our Diffuser on the Hopper-Medium dataset, demonstrating effectiveness even in suboptimal and challenging data distribution. Meanwhile, Figure offers side-by-side comparison Diffuser and HD" on AntMaze- Large-Diverse. We observe that, while standard Diffuser frequently produces trajectories that collide with maze walls or fail reach the goal, our hierarchical extension with LoMAP (i.e., HDP?) maintains more coherent routes and significantly
15

---

Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning
increases the likelihood of reaching the target (marked by the red sphere). In both examples, projecting intermediate diffusion steps locally approximated manifold substantially mitigates stochastic failures, highlighting the effectiveness of our approach for long-horizon, high-dimensional control tasks.
Diffuser (Janner et al., 2022)
Diffuseer" P (ours)
Figure 6: Visual comparison of rollout trajectories from Diffuser and Diffuser the Hopper-Medium task.
E. Experimental Details
E.1. Environments
Maze2D, Maze2D environments (Fu al., 2020) require an agent to undertake long-horizon navigation, moving toward distant goal location. The agent accrues no reward unless successfully reaches the goal, which point it receives reward of 1. The three available layouts- Sy-uuss---Maze Medium, and Large-vary vary in complexity. Further, Maze2D features two distinct tasks: single-task variant with a fixed goal and multi-task option (Multi2D), which randomizes the goal location at the beginning of each episode. A summary of key details can be found in Table 7.
Table Environment details for Maze2D experiments.
Maze2D-Large
Maze2D-Medium
Maze2D-UMaze
State space S
R²
R4
R4
Action space A
R
R
IR2
Goal space G
R²
IR2
IR2
Episode length
800
600
300
Locomotion. Gym-MuJoCo locomotion tasks (Fu et al., 2020) serve as widely recognized benchmarks for assessing algorithm performance on heterogeneous datasets of varying quality. The Medium dataset consists of one million sam- ples gathered from an SAC (Haarnoja et al., 2018) agent trained to roughly one-third of expert-level performance. The Medium Relllaaayll dataset contains all experiences accumulated throughout the SAC training process up to that same per- formance threshold. Finally, the Med sum-Exppert dataset created by combining expert demonstrations and suboptimal
16

---

Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning
Diffuser (Janner al., 2022)
HD (ours)
Figure 7: Visual comparison of rollout trajectories from Diffuser and HD? in the AntMaze-Large-Dieee Diverse task. We plot each trajectory at 50-step intervals for clarity. The goal is marked by red sphere, and 20 rollout trajectories are shown in different colors.
data in equal amounts. Details on the state/action spaces and episode lengths can be found in Table 8.
Table 8: Environment details for Locomotion experiments.
Hopper-*
Walker2d-*
Halfchectah-*
State space S
R1
R
R17
Action space A Episode length
R3
R
R
1000
1000
1000
AntMaze. AntMaze tasks (Fu et al., 2020) involve guiding an 8-DoF Ant robot through intricate maze layouts using MuJoCo for physics simulation. The environment uses a sparse reward structure, granting a reward only upon reaching the designated goal, thus posing challenging long-horizon navigation problem. Further difficulty arises from the offline dataset, which contains numerous trajectory segments that do not succeed in reaching the goal. We measure performance by the success rate of the agent reaching the endpoint. A summary of the key environment details is provided in Table 9.
Table 9: Environment details for AntMaze experiments.
AntMaze-
State space S
R229
Action space A Episode length
R
1000
E.2. Implementation Details
Below, we summarize the key implementation details and hyperparameters used throughout our experiments:
Network architecture. We build on the Diffuser framework (Janner et al., 2022), employing a temporal U-Net with repeated convolutional residual blocks to parameterize Eg.
Planning horizons. For Maze2D and Multi2D tasks, the planning horizon is 128 in U-Maze, 256 in Medium, and 256 in Large. For MuloCo locomotion tasks, the horizon 32, and for AntMaze it is 64.
Diffusson steps. We use 256 steps for the diffusion process in Maze2D//Multi2D Large and Medium, 128 in Mave2D//Mult2D U-Maze, and 20 other environments.
17

---

Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning
Guidance scales. For AntMaze tasks, we select the guidance scale w from the set (5.0, 3.0, 1.0, 0.01, In MuloCo locomotion tasks, we select w from {0.3, 0.1, 0.01, 0.000,000000 during planning.
Local manifold approximation. We tune the number of neighbors E {5, 10,200 20} in our local manifold approximation procedure.
Hierarchical Diffuser in AntMaze. For the high-level and low-level planners, we follow Chen et al. (2024) and train each component separately using trajectory segments randomly sampled from the D4RL offline dataset. Specifically, the high-level planner generates state-space trajectories with a planning horizon of 226 and temporal jumps of 15. During execution, the corresponding actions are inferred through a learned inverse dynamics model (Ajay et al., 2023).
F. Practical Implementation
Manifold approximation. A straightforward h-nearest-neighhor retrieval from the entire offline dataset at each diffusion step can be prohibitively expensive. To mitigate this cost, we employ an inverted file (IVF) index from the Faiss library (Douze et al., 2024), which partitions the dataset into a set of coarse centroids and restricts each query to only a few relevant clusters.
Concretely, IVF uses k-means to learn Riist centroids (CI...., (Cmee.. } across the dataset of dimension d. Each data point x mapped to its nearest centroid, forming an inverted list. query vector q is matched to its closest centroids, after which the search proceeds solely within the corresponding clusters. This design significantly reduces the number of distance computations relative an exhaustive linear scan. Although coarse clustering introduce minor inaccuracies, we have found this approach to be effective in our experiments.
In LoMAP, the IVF-based approximate neighbor search operates on denoised sample * retrieving to k neighbors from the offline dataset. We then forward-diffun these neighbors to timestep and perform a rank-r grincipa--copooner analysis to approximate the local manifold of feasible trajectories. Projecting T' onto this manifold helps correct off- manifold drift caused by inexact guidance. Crucially, limiting the search to relevant clusters enables LoMAP to handle large datasets and high-dimensional state-action spaces, making it practical for long-horizon offline reinforeement-leamingg tasks.
Manifold projection. We observed that applying manifold projection selectively, rather than uniformly across all diffusion steps, can significantly reduce computational costs and even enhance overall performance. Specifically, we found that projection is particularly beneficial when applied during intermediate to later stages of the reverse diffusion process. We hypothesize two main reasons for this phenomenon: first, Tweedie-based denoisers inherently exhibit biases toward majority or high-density features, as noted by Um et al. (2024), making early stage projections less impactful. Second, the discrepancy between the learned reverse transitions of the diffusion model and the true data distribution becomes most pronounced at intermediate diffusion steps, consistent with observations reported prior studies (Na et al., 2024). Consequently, concentrating projection efforts on these critical stages not only improves computational efficiency but also effectively mitigates manifold deviation, resulting in improved sampling quality.
G. Baseline Performance Sources
G.1. Maze2D
The reported IQL scores come from Table in Janner et al. (2022), RGG scores from Table 2 in Lee et al. (2023b), and TAT scores from Table 2 in Feng et al. (2024).
G.2. Locomotion
We obtain scores for BC, CQL, and IQL from Table I in Kostrikov et al. (2022); DT from Table 2 in Chen et al. (2021); TT TT from Table I in Janner et al. (2021); )); MOPO from Table I in Yu et al. (2020); MOReL from Table 2 in Kidambi et al. (2020); and Diffuser from Table in Janner et al. (2022). Scores for RGG and TAT are drawn from Table 3 in Feng al. (2024). while DD scores come from Table 1 in Ajay al. (2023).
G.3. AntMaze
Scores for DD in the AntMaze domain are taken from Table in Dong et al. (2024b).
18


Reviews:
[{"review_id": "qxVPIMiMEe", "reviewer": "Reviewer_XEyQ", "summary": "This paper addresses the limitation in diffusion-based trajectory planning for RL tasks. Previous works in diffusion models often produce infeasible trajectories due to \"manifold deviation\" during the sampling process, so the authors proposed a novel method LoMAP, which is a training-free framework. It projects diffusion model samples onto locally approximated low-rank manifolds derived from offline datasets to prevent infeasible trajectory generation caused by guidance errors during the sampling process. The experiments on Gym achieved good results compared to the baseline.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and constructive feedback on this work. We especially appreciate the insightful questions regarding the accurate manifold approximation. Please find our detailed answers below.\n\n- **\"I do not fully understand why LoMAP can help with the issue of diffusion sampling.\"**\n\nWe thank the reviewer for this important question. To clarify, the core intuition behind LoMAP is grounded in the observation that feasible trajectories from offline datasets lie on an intrinsically low-dimensional manifold embedded within a high-dimensional trajectory space. Diffusion-based sampling methods can deviate from this manifold due to inaccuracies in guidance (as analyzed in Section 3.1), leading to infeasible trajectories. \n\nLoMAP addresses this problem by iteratively projecting guided diffusion samples back onto an approximated local manifold derived directly from offline trajectories. Specifically, at each diffusion step, we first compute a denoised estimate of the current trajectory sample using Tweedie's formula. We then retrieve the $k$-nearest neighbors from the offline dataset, which naturally represent trajectories that closely adhere to the true data manifold. By forward-diffusing these neighbor trajectories to the current diffusion timestep, we approximate the local structure of the intermediate manifold ($\\mathcal{M}_{i-1}$). \n\nPCA then provides a convenient and effective way to approximate the local manifold around this neighborhood. Because PCA identifies principal directions of variance, it naturally captures the major local geometric structures of the feasible manifold represented by nearby trajectories. By projecting the diffusion sample onto this PCA-derived subspace, we remove the off-manifold components, effectively correcting the artifact trajectories. \n\n- **\"I'm wondering if the author can provide some evidence this method actually helps with more accurate manifold approximation.\"**\n\nThanks for the important question. To provide further evidence, we computed the Realism Score [1], which quantifies how closely generated trajectories lie to the true manifold defined by the offline dataset. Specifically, we approximated the true manifold using k-NN hyperspheres constructed from 20,000 offline trajectories and generated 100,000 trajectory samples to measure the average realism score. As shown below, applying LoMAP consistently yields higher realism scores compared to diffusion sampling without LoMAP, clearly demonstrating that our method effectively produce samples closer to the true data manifold.\n\n| Environment | **w/o LoMAP** | **w/ LoMAP** |\n| --- | --- | --- |\n| Maze2D U-Maze | 1.23 | **1.30** |\n| Maze2D Medium | 1.40 | **1.56** |\n| Maze2D Large | 1.36 | **1.47** |\n\n- **“I suggest adding more DT-related baseline methods.”**\n\nThank you for suggesting additional DT-related baselines. Following your advice, we now include comparisons with QDT [2], LPT [3] (guidance-based methods), and WT [4] (waypoint-based method). For MuJoCo locomotion tasks, we used the QDT implementation from the d3rlpy library and the official implementation provided by the authors for LPT, while WT results are taken directly from the original paper [4]. As shown in the table below, despite its simplicity, LoMAP consistently achieves the best average performance across these benchmarks. We will incorporate these additional comparisons into the final version of the paper.\n\n| Environment | **QDT** | **LPT** | **WT** | **ours** |\n| --- | --- | --- | --- | --- |\n| halfcheetah-med-expert | 89.8 ± 0.7 | 90.8 ± 0.19 | **93.2 ± 0.5** | 91.1 ± 0.23 |\n| hopper-med-expert | 109.4 ± 2.3 | **111.4 ± 0.31** | 110.9 ± 0.6 | 110.6 ± 0.29 |\n| walker2d-med-expert | 108.8 ± 0.7 | 109.1 ± 0.04 | **109.6 ± 1.0** | 109.2 ± 0.05 |\n| halfcheetah-med | 42.3 ± 0.4 | 43.5 ± 0.08 | 43.0 ± 0.2 | **45.4 ± 0.13** |\n| hopper-med | 66.5 ± 6.3 | 63.8 ± 1.47 | 61.1 ± 1.4 | **93.7 ± 1.54** |\n| walker2d-med | 67.1 ± 3.2 | **81.1 ± 0.33** | 74.8 ± 1.0 | 79.9 ± 1.21 |\n| halfcheetah-med-replay | 35.6 ± 0.5 | **40.7 ± 0.12** | 39.7 ± 0.3 | 39.1 ± 0.99 |\n| hopper-med-replay | 52.1 ± 20.3 | 89.9 ± 0.61 | 88.9 ± 2.4 | **97.6 ± 0.58** |\n| walker2d-med-replay | 58.2 ± 5.1  | 75.7 ± 0.34 | 67.9 ± 3.4 | **78.7 ± 2.2** |\n| **average** | 69.9 | 78.4 | 76.8 | **82.8** |\n\nReferences \n\n[1] Kynkäänniemi et al., Improved precision and recall metric for assessing generative models. In NeurIPS, 2019.\n\n[2] Yamagata et al., Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL. In ICML, 2023.\n\n[3] Kong et al., Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inference. In NeurIPS, 2024.\n\n[4] Badrinath et al., Waypoint Transformer: Reinforcement Learning via\nSupervised Learning with Intermediate Targets. In NeurIPS, 2023.", "claims_and_evidence": "The authors make a clear point on the problem of the guidance gap in diffusion models and provide theoretical proof to understand the issue. However, I'm a bit lost on why KNN and PCA can solve this problem, and I feel section 3.2 is a bit isolated from the discussion of manifold deviation. I hope the authors can explain more about this part and expand this section since it is the key part of this paper.", "methods_and_evaluation_criteria": "The method and analysis are clear and simple, which is a plus. The whole framework is lightweight, but I do not fully understand why LoMAP can help with the issue of diffusion sampling. \n\nThe experiment part seems solid and the performance is good. I suggest adding more DT-related baseline methods. See the below sections for references.", "theoretical_claims": "The theoretical analysis in section 3.1 looks good to me. And the introduction of Diffuser and related literature is clear and concise.", "experimental_designs_or_analyses": "The experimental designs follow the most commonly used environment and baselines. \n\nThis whole paper talks about the manifold projection, and I'm wondering if the author can provide some evidence this method actually helps with more accurate manifold approximation.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "Although this paper focused on diffusion models, I suggest the author include more related works based on Decision Transformers (DT) as baseline or related works, especially those with classifier or classifier-free guidance in planning. Here are some papers: \n[1]: Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inference\n[2]: Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL", "other_strengths_and_weaknesses": "I like to narrative of the paper. The weaknesses are discussed above.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See the above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rGcArdkWB1", "reviewer": "Reviewer_AXnQ", "summary": "Classifier guidance can introduce distribution shift during diffusion sampling. This paper proposes a training-free method to constrain guided diffusion within a learned manifold by projecting noisy samples onto a local low-dimensional manifold, approximated using nearest neighbors from the training set at each diffusion step.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and constructive feedback on our work. Please find our detailed responses below.\n\n- **Not a strong improvement over chosen baselines**\n\nWe appreciate the valuable feedback provided by the reviewer. However, we respectfully disagree with the subjective judgment that there was “not strong” improvement. We believe the performance improvements demonstrated by our method are meaningful. Specifically, our approach consistently outperforms all baselines in terms of average performance across three established benchmarks, despite the simplicity of our method.\n\n- **Missing baselines**\n\nWe sincerely thank the reviewer for pointing out several interesting works. However, we would like to clarify why the suggested baselines are not directly suitable for comparison in our setting:\n\n- **[1]** assumes human-provided guidance in the form of sketches or demonstrations, which fundamentally differs from our scenario where no such human intervention is assumed. Additionally, [1] relies on a diffusion-based policy rather than a diffusion-based planner, making direct comparisons inappropriate. The reviewer's comment, \"It also enables minority guidance without introducing constraint violations, as demonstrated in your Figure 5,\" is also not entirely accurate. The trajectories in [1] were generated using a diffusion policy trained with random walks rather than being goal-conditioned as in our experiments.\n- **[2]** assumes the availability of differentiable constraints. For instance, applying [2] directly to the maze environment would require prior knowledge about the exact location of walls, making the comparison unfair.\n- **[3]** focuses specifically on manipulation scenarios guided by language instructions, which significantly deviates from the tasks and settings considered in our work, making it an unsuitable baseline.\n\n---\n- **Figure suggestion**\n\nThank you for the suggestions! We will incorporate them into the final version of our paper.\n\n- **What is the intuition behind the LoMAP**\n\nPlease see response for reviewer XEyQ for the details.\n\n- **Impact of the number of PCA components**\n\nWe have conducted additional experiments by varying the number of PCA components ($K$). As shown in the results below, LoMAP demonstrates robustness provided that K is above a certain threshold. We select $K=10$ as it offers computational efficiency comparable to $K=20$, yet achieves similar performance.\n\n| Environment | **Diffuser** | **K=5** | **K=10** | **K=20** |\n| --- | --- | --- | --- | --- |\n| halfcheetah-med-expert | 88.9 ± 0.3 | 91.2 ± 0.3 | 91.1 ± 0.2 | 90.9 ± 0.2 |\n| hopper-med-expert | 103.3 ± 1.3 | 108.9 ± 2.9 | 110.6 ± 0.3 | 110.9 ± 0.2 |\n| walker2d-med-expert | 106.9 ± 0.2 | 107.8 ± 0.2 | 109.2 ± 0.1 | 108.9 ± 0.1 |\n| halfcheetah-med | 42.8 ± 0.3 | 44.5 ± 0.1 | 45.4 ± 0.1 | 44.9 ± 0.1 |\n| hopper-med | 74.3 ± 1.4 | 90.16 ± 2.0 | 93.7 ± 1.5 | 94.1 ± 1.7 |\n| walker2d-med | 79.6 ± 0.6 | 79.17 ± 1.8 | 79.9 ± 1.2 | 81.4 ± 1.1 |\n| halfcheetah-med-replay | 37.7 ± 0.5 | 38.2 ± 1.2 | 39.1 ± 1.0 | 38.9 ± 0.7 |\n| hopper-med-replay | 93.6 ± 0.4 | 96.4 ± 0.2 | 97.6 ± 0.6 | 96.3 ± 0.1 |\n| walker2d-med-replay | 70.6 ± 1.6 | 74.7 ± 3.1 | 78.7 ± 2.2 | 78.9 ± 1.8 |\n- **“Have you tried applying LoMAP only in early steps or only in later steps?”**\n\nYes, we explored applying LoMAP exclusively in either early or later stages of the diffusion sampling process. We observed that applying LoMAP only in early steps was not particularly effective. In contrast, applying LoMAP during intermediate to later diffusion steps yielded the most significant improvements. This indicates that the mismatch between the model’s reverse transition and the true data distribution becomes most pronounced during these intermediate steps, aligning with similar observations reported in prior study [4].\n\n- **“Computational overhead of finding nearest neighbors”**\n\nPlease see response for reviewer wazP for the details.\n\n- **“Does your algorithm require access not only to the diffusion model but also to the dataset the model was trained on?”**\n\nLoMAP, as currently presented, assumes the existence of an offline dataset, consistent with other offline RL baselines. For instance, Decision Diffuser (DD) retrains a conditional diffusion planner from scratch using offline datasets, and RGG trains an out-of-distribution (OOD) score predictor utilizing offline datasets. Unlike these methods, our approach leverages the dataset directly without additional training, making it simple and straightforward to apply.\n\nReferences:\n\n[1] Shi et al., Inference-Time Policy Steering through Human Interactions. In ICRA, 2025.\n\n[2] Lee et al., Learning Diverse Robot Striking Motions with Diffusion Models and Kinematically Constrained Gradient Guidance. arXiv, 2024.\n\n[3] Hao et al., DISCO: Language-Guided Manipulation with Diffusion Policies and Constrained Inpainting. arXiv, 2024.\n\n[4] Na et al., Diffusion Rejection Sampling. ICML, 2024.", "claims_and_evidence": "The claim that guided diffusion can lead to manifold deviation is supported by theoretical analysis. However, the assertion that LoMAP, the proposed projection method, effectively bridges the guidance gap is less convincing, as evidenced by its lack of significant improvement over baselines like DD and TAT in Table 2.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "See below the essential reference section.", "essential_references_not_discussed": "Many recent works have explored diffusion sampling that satisfies constraints [1-3]. In particular, [1] introduced a training-free, plug-and-play four-line modification to guided diffusion that directly addresses the issue shown in your Figure 3. It also enables minority guidance without introducing constraint violations, as demonstrated in your Figure 5. Comparing your proposed method against this simpler alternative would help verify whether it offers a meaningful improvement over existing approaches in addressing the same challenge.\n\n[1] Inference-Time Policy Steering through Human Interactions\n[2] Learning Diverse Robot Striking Motions with Diffusion Models and Kinematically Constrained Gradient Guidance\n[3] DISCO: Language-Guided Manipulation with Diffusion Policies and Constrained Inpainting", "other_strengths_and_weaknesses": "Strength – The paper provides a theoretical demonstration of manifold deviation.\n\nWeakness – The results do not show a strong improvement over chosen baselines such as DD or TAT. Additionally, the study lacks comparisons with directly relevant baselines, as mentioned above.", "other_comments_or_suggestions": "Figures 1(a) and 1(b) do not appear significantly different, making it difficult to highlight the differences. Additionally, Figure 1(c) does not clearly convey the idea of your method. It may be helpful to refine the figure to better illustrate the key improvements and make the distinctions more visually compelling.", "questions_for_authors": "1. What is the intuition behind the idea that projecting onto the subspace approximated by PCA of nearest neighbors will reduce manifold deviation during diffusion? Why does PCA, in particular, help? Does the number of PCA components impact performance? \n\n2. Could this projection (an approximation) at later diffusion steps introduce constraint violations? Have you experimented with applying LoMAP only during the early diffusion steps or, conversely, only during the later diffusion steps, rather than applying it across all diffusion steps?\n\n3. What is the computational overhead of finding nearest neighbors from a large offline trajectory dataset at each diffusion step? Does your algorithm require access not only to the diffusion model but also to the dataset the model was trained on?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NMF1MMxn35", "reviewer": "Reviewer_wazP", "summary": "The authors tackle the problem of approximate energy guidance with diffusion policies in the context of offline RL. Partial energies naively trained through MSE loss such as in Diffuser are only lower bounds to the true energy, and so using their gradients for conditional sampling can push trajectories off the manifold supported by the offline dataset. The authors propose a simple to implement technique to project the latent trajectories back into a low rank approximation of the K nearest neighbors to the dataset, noised to the current diffusion timestep. The K nearest neighbors are obtained with the expected denoised trajectory from the current diffusion step using Tweedie's formula. The authors demonstrate that using this correction improved performance across tasks in the D4RL suite and specifically highlight generation of valid trajectories in Maze2D and AntMaze tasks, where Diffuser often generates infeasible paths that pass through walls.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive review and constructive feedback. Please find our detailed response below.\n\n- **“With the IVF method for faster nearest neighbors as described in Appendix D, what was the wall clock time to generate a single trajectory for a task like AntMaze? How expensive is the procedure with naive nearest neighbors?”**\n\nThank you for highlighting this important practical aspect. We measured the wall-clock time required by Hierarchical Diffuser (HD) with LoMAP in the AntMaze task, averaging the generation time per trajectory using a single NVIDIA A10G GPU, with the number of neighbors set to $K=10$.\n\n| **Method** | **Total wall-clock time per trajectory (sec)** | **KNN-search time (sec)** |\n| --- | --- | --- |\n| Naive KNN | 21.62 | 21.02 |\n| IVF-based approximate KNN (**ours**) | 0.53 | 0.01 |\n\nAs shown above, the naive KNN search is prohibitively expensive, requiring over 21 seconds to generate a single trajectory, with the vast majority of time dedicated to the KNN search itself. In contrast, our IVF-based approximate KNN significantly reduces runtime, with only a negligible fraction of the total time spent on the nearest neighbor search.\n\n- **“Could you include QGPO to the results tables? It is a technique which trains a time dependent energy model for unbiased guidance, so perhaps can serve as a sort of upper bound in performance.”**\n\nThank you for suggesting QGPO as an additional baseline. We agree that including QGPO can indeed serve as a valuable upper-bound comparison for performance. We appreciate this insightful recommendation and will incorporate the QGPO results into the final version of our paper.\n\n- “**Cosine distance metric function may not be good heuristics for other problems (especially with high dimensional state spaces).**”\n\nAs the reviewer correctly points out, cosine distance may not be suitable for certain high-dimensional state spaces, especially pixel-based environments. Exploring effective manifold approximation methods for more complex, high-dimensional state spaces remains an important direction for future work. We believe one promising approach would be to combine LoMAP with latent trajectory representations [1, 2]. We will clarify this limitation explicitly in the final version.\n\n- **\"An additional references to asymptotically unbiased guidance strategies (perhaps in the appendix)\"**\n\nThank you for this valuable suggestion. We will discuss asymptotically unbiased guidance strategies further in the related work section of the final version.\n\nReferences \n\n[1] Co-Reyes et al., Self-consistent trajectory autoencoder:\nHierarchical reinforcement learning with trajectory embeddings. In ICML, 2018.\n\n[2] Jiang et al., Efficient Planning in a Compact Latent Action Space. In ICLR, 2023.", "claims_and_evidence": "The claims in the paper are clear, and evidence is convincing", "methods_and_evaluation_criteria": "D4RL, specifically the maze tasks make sense for evaluation of this approach. However, I will note that D4RL is a very saturated benchmark, and going forward I hope OGBench is used instead. Since this work must have been done concurrently with the introduction of OGBench, I do not consider this a weakness.", "theoretical_claims": "The theoretical claims are valid, and I checked the only major proof in Appendix A.", "experimental_designs_or_analyses": "The experiments seem to be sound, I did not notice any specific issue.", "relation_to_broader_scientific_literature": "The paper is relevant to the application of diffusion models for offline RL, which is an active area of research. Contrastive Energy Prediction [1] and Relative Trajectory Balance are quite relevant to this work, as hey similarly focus on the problem of inaccurate energy guidance in the context of offline RL. However in that paper, the authors introduce a contrastive training strategy to learn an unbiased guidance function, whereas this paper instead introduces a simpler to implement (but perhaps less general) training-free guidance method. The problem of training-free approximate/asymptotically unbiased guidance for diffusion models is heavily explored outside the context of offline RL, as explored in this survey paper [3].\n\n[1] Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning, https://arxiv.org/abs/2304.12824\n\n[2] Amortizing intractable inference in diffusion models for vision, language, and control, https://arxiv.org/abs/2405.20971\n\n[3] Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Review, https://arxiv.org/abs/2501.09685", "essential_references_not_discussed": "I think it would improve the paper to include references (and perhaps even a comparison) to asymptotically unbiased guidance strategies such as Twisted Diffusion Sampler (TDS) [1] and some other SMC based methods listed in the above survey paper, and broadly discuss this line of work (perhaps in the appendix).\n\n[1] Practical and Asymptotically Exact Conditional Sampling in Diffusion Models, https://arxiv.org/abs/2306.17775", "other_strengths_and_weaknesses": "### Strengths\n1. The paper is clearly written, with useful illustrations.\n2. The proposed method is novel to my understanding.\n\n### Weaknesses\n1. K-nearest neighbors computation with the entire dataset could be expensive, especially with growth in dimensionality of states and trajectory length. This is especially problematic for any sort of real world control tasks.\n2. The approach used to compute K-nearest neighbors from the dataset, with Tweedie's denoised estimate as the anchor and cosine distance the metric function may not be good heuristics for other problems (especially with high dimensional state spaces). Perhaps some kind of abstract trajectory latent representations could be used instead?", "other_comments_or_suggestions": "Could you include QGPO (from the contrastive energy prediction paper) to the results tables? It is a technique which trains a time dependent energy model for unbiased guidance, so perhaps can serve as a sort of upper bound in performance.", "questions_for_authors": "1. With the IVF method for faster nearest neighbors as described in Appendix D, what was the wall clock time to generate a single trajectory for a task like AntMaze? How expensive is the procedure with naive nearest neighbors?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Llv6bTVjg1", "reviewer": "Reviewer_bsde", "summary": "The authors introduce formally and investigate mathematically the manifold deviation issue due to approximate guidance in the context of trajectory-planning (for reward maximization) via diffusion models. They provide a lower bound on this error, and to address this issue they introduce LoMAP, a training-free method that performs projections sequentially by leveraging offline data along the reverse sampling process to constrain the generated samples to the underlying data manifold. Ultimately, they show extensive experimental validation for the proposed method.\n\n## update after rebuttal\nAfter reading the authors rebuttal I decided to keep my original score as the content of the rebuttal did not alter significantly my beliefs regarding positive aspects as well as (inherent) limitations of this work.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and constructive feedback on our work. We especially appreciate the reviewer pointing out the clarity issue regarding our claims. Please find our detailed response below.\n\n- **“The claim \"The current sample is then projected onto this subspace, thereby remaining on the manifold of valid behaviors.\" at line 69 seems not precise.”**\n\nWe fully agree with the reviewer that our LoMAP method does not necessarily guarantee the validity of behaviors generated by diffusion models. Accordingly, we will **tone down** this claim to avoid ambiguity and overstatement. Specifically, we will revise:\n\n> \"thereby remaining on the manifold of valid behaviors.\"\n> \n\nto a more accurate and precise description:\n\n> \"thereby significantly reducing off-manifold deviations and improving the likelihood of generating valid behaviors.\"\n>\n\n- **“The claim \"Consequently, combining LoMAP with minority guidance can help uncover feasible yet unexplored solutions that might otherwise remain inaccessible to standard diffusion planners.\" at line 403 seems not precise.”**\n\nWe apologize for the confusion caused by this statement. Our original intention was to highlight that standard diffusion planners typically require a large number of samples to generate trajectories located in low-density regions of the data distribution, thus limiting exploration within a restricted sampling budget.\n\nWe aimed to clarify that one way to overcome this limitation—though not our contribution—is through minority guidance [1], which explicitly guides sampling towards low-density regions. Our contribution, LoMAP, further enhances this process by ensuring that trajectories sampled under minority guidance remain feasible and closer to the data manifold, allowing efficient generation of trajectories from low-density regions without extensive sampling.\n\nWe appreciate the reviewer highlighting this ambiguity and will clarify this point carefully in the final version of the paper.\n\n- **“what is $n$ in line 161 defined?”**\n\nWe apologize for this confusion. This was a typo. To clarify, the original sentence:\n\n> \"… $\\boldsymbol{\\tau}^i$ is inherently concentrated around a ($n-d$) dimensional manifold $\\mathcal{M}_i$.\"\n> \n\nshould be corrected to the following description:\n\n> \"… $\\boldsymbol{\\tau}^i$ is inherently concentrated around a ($d-k$)-dimensional manifold $\\mathcal{M}_i$.\"\n> \n\nThank you for pointing out this typo.\n\n- **“In line 225 the paper claims that Prop. 3.2 indicates that the manifold dev. problem is stronger for long-horizon tasks. How should I infer this from the mathematical result within the proposition?”**\n\nProposition 3.2 states that the guidance gap scales at least proportionally to the square root of the dimensionality $d$ of the trajectory representation:\n\n$$\\Delta_{\\mathrm{guidance}}\\bigl(\\boldsymbol{\\tau}^i\\bigr) \\ge \\frac{\\,c\\,}{\\sqrt{1-\\alpha_i}}\\sqrt{d}$$\n\nWe clarify that the dimensionality $d$ here directly relates to the length of the planning horizon. Specifically, in diffusion-based planning, trajectories are represented by concatenating states and actions across the entire planning horizon. Therefore, the dimension $d$ grows linearly with the length of the planning horizon.\nConsequently, the lower bound provided by Proposition 3.2 indicates that as the planning horizon increases, the guidance gap inevitably grows, which implies a higher likelihood and severity of manifold deviation in longer-horizon tasks.\n\n- **“My main concern regarding the paper regards the risk of ending up obtaining a sampler excessively regularized from offline data as explained within my point (2) of Claims And Evidence section.”**\n\nThank you for raising this important point. We acknowledge that our approach inherently regularizes trajectories towards the offline data manifold. However, our primary objective in this work is to ensure the generation of feasible, high-return trajectories for offline RL tasks particularly in settings where safety and reliability are crucial. This highlights a fundamental trade-off between trajectory feasibility and exploration freedom.\n\nTo clarify this explicitly, we will revise the manuscript to discuss this limitation in detail, clearly identifying scenarios where LoMAP provides the most substantial benefits (e.g., safety-critical domains or tasks prioritizing feasibility over aggressive exploration).\n\nFurthermore, we believe LoMAP can be effectively combined with trajectory stitching and data augmentation techniques [2, 3], which enrich offline datasets with diverse, synthetic trajectories, which we leave for future work.\n\nReferences:\n\n[1] Um et al., Don't Play Favorites: Minority Guidance for Diffusion Models. In ICLR, 2024.\n\n[2] Li et al., DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching. In ICML, 2024.\n\n[3] Chen et al., Extendable Long-Horizon Planning via Hierarchical Multiscale Diffusion. arXiv, 2025.", "claims_and_evidence": "1 - The claim \"The current sample is then projected onto this subspace, thereby remaining on the manifold of valid behaviors.\" at line 69 seems not precise. Diffusion models, as well of offline data, capture an implicit notion of validity only approximately. As a consequence these models can generate data points that are invalid, and the procedure presented within the paper, although useful to improve the chances of generating valid points, does not seem to imply that the generated points are valid. In order to achieve this, the diffusion model would need to interact with an available validity checker or certain assumptions would have to be made about the data space.\n\n2 - The claim \"Consequently, combining LoMAP with minority guidance can help uncover feasible yet unexplored solutions that might otherwise remain inaccessible to standard diffusion planners.\" at line 403 seems not precise. To the best of my knowledge, training a diffusion model consists in learning a function approximator for the score function (i.e., a score network) that can induce a final marginal density (on the data space) which modes (i.e., significantly positive density) in regions where no data are present. It seems to me that the proposed method would instead prevent the diffusion model from sampling these modes, while it would let the model sample low-density modes where enough offline data are present. From the experiments presented within Sec. 4.4 this distinction is not clear and therefore the claim above seems not supported, especially the 'unexplored solutions' part.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Checked the derivations in the main paper and seem correct.", "experimental_designs_or_analyses": "Checked exp. in Sec. 4.1 and seems convincing to me. Regarding exp. in Sec. 4.4 I mentioned a doubt within point (2) of the Claims and Evidence section.", "relation_to_broader_scientific_literature": "- The paper tackles an important problem that arises in the context of trajectory-based planning, a very timely research topic, tackled via guidance. To the best of my knowledge, the paper properly mentions related works and references and makes a convincing positive comparison with works within this literature stream.\n\n- The problem of reward-guided sampling can be solved in a multitude of ways beyond guidance. Examples include using RL[1] or control-theoretic[2] formulations to fine-tuning a diffusion model, or inference-time techniques beyond guidance [3]. It is not clear how the ideas presented in this work extend to these settings, as the current formulation seems quite specific to guidance. \n\n[1] Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review, Uehara et al.\n[2] Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control, Domingo-Enrich et al.\n[3] Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Review, Uehara et al.", "essential_references_not_discussed": "I am not aware of important references not mentioned.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper tackles a timely open problem and presents a formally clear formulation.\n2. Proposes a solution that seems of practical relevance (i.e., easy to use) and well-performing on relevant problems.\n3. The paper is well written, clear, and easy to read.\n\nWeaknesses:\n1. My main concern regarding the paper regards the risk of ending up obtaining a sampler excessively regularized from offline data (i.e., overfitting to offline data) as explained within my point (2) of Claims And Evidence section.  This might be problematic towards using the introduced machinery in RL problems, where the goal is often to discover new strategies not contained within offline data.", "other_comments_or_suggestions": "- what is $n$ in line 161 defined?", "questions_for_authors": "- Did I misinterpret something within the weakness point that I mentioned ?\n- In line 225 the paper claims that Prop. 3.2 indicates that the manifold dev. problem is stronger for long-horizon tasks. How should I infer this from the mathematical result within the proposition?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Diffusion-based generative models for RL have inconsistent reliability due to stochastic risk of infeasible trajectories. Via identifying inaccurate sampling guidance as the cause and showing manifold deviation, the paper proposes LoMAP, a training-free method projecting guided samples onto a low-rank subspace from offline data to prevent such infeasible trajectories. I think the paper addresses a key problem in diffusion-based RL methods and will be quite helpful for further research in this direction. Thus I recommend an acceptance for this paper.
Decision: Accept (poster)