Tree-Guided Diffusion Planner
Authors: Hyeonseong Jeon
Cheolhong Min
Jaesik Park
Venue: NeurIPS 2025 poster ()
Keywords: diffusion-based planning
zero-shot planning
training-free guidance
decision making
Primary area: reinforcement_learning

Abstract: Planning with pretrained diffusion models has emerged as a promising approach for solving test-time guided control problems. Standard gradient guidance typically performs optimally under convex, differentiable reward landscapes. However, it shows substantially reduced effectiveness in real-world scenarios with non-convex objectives, non-differentiable constraints, and multi-reward structures. Furthermore, recent supervised planning approaches require task-specific training or value estimators, which limits test-time flexibility and zero-shot generalization. We propose a Tree-guided Diffusion Planner (TDP), a zero-shot test-time planning framework that balances exploration and exploitation through structured trajectory generation. We frame test-time planning as a tree search problem using a bi-level sampling process: (1) diverse parent trajectories are produced via training-free particle guidance to encourage broad exploration, and (2) sub-trajectories are refined through fast conditional denoising guided by task objectives. TDP addresses the limitations of gradient guidance by exploring diverse trajectory regions and harnessing gradient information across this expanded solution space using only pretrained models and test-time reward signals. We evaluate TDP on three diverse tasks: maze gold-picking, robot arm block manipulation, and AntMaze multi-goal exploration. TDP consistently outperforms state-of-the-art approaches on all tasks. The project page can be found at: https://tree-diffusion-planner.github.io

Tree-Guided Diffusion Planner
Hyeonseong Jeon'
Cheolhong Min'
Jaesik Park
'Deparmeent of Computer Science & Engineering. "Interdisciplinary Program of Al Scoul National University
Abstract
Planning with pretrained diffusion models has emerged as a promising approach for solving test-time guided control problems. Standard gradient guidance typically performs optimally under convex, differentiable reward landscapes. However, it shows substantially reduced effectiveness in weal-world scenarios with non-convex objectives, eindiffeerentible constraints, and multi-reward structures. Further- more, recent supervised planning approaches require task-specific training or value estimators, which limits test-time flexibility and zero-shot generalization. We propose a Trreeguided Diffusion Planner (TDP), a zero-shot test-time planning framework that balances exploration and exploitation through structured trajectory generation. We frame test-time planning tree search problem using bi-level sampling process: (1) diverse parent trajectories are produced via training- rrieing-frree free particle guidance encourage broad exploration, and (2) sub-trajectories are fined through fast conditional denoising guided by task objectives. TDP addresses the limitations of gradient guidance by exploring diverse trajectory regions and harnessing gradient information across this expanded solution space using only pretrained models and test-time reward signals. We evaluate TDP on three diverse tasks: maze gold-picking, robot arm block manipulation, and AntMaze multi-goal exploration. TDP consistently outperforms state-of-the-ar approaches all tasks. The project page can be found at: tree-diffusion patprsiiio-pllaan.gtthbiiiiiii
Introduction
Diffusion models offer a data-driven framework for planning, enabling the generation of coherent and expressive trajectories learning from offline demonstrations [14. Compared to single-step model-free reinforcement learning (RL) methods () diffusion planners are more effective for long-horizon planning by generating temporally extended trajectories through multi-step prediction. Without task-specific dynamics models, pretrained diffusion planners can be adapted for test-time planning through guidance functions that provide numerical scores for user requirements such as state conditions or physical constraints. Prior studies in diffusion guidance algorithm (e.g.. classifier guidance 7 have been successfully incorporated to generate conditional trajectory samples given test-time reward signals [8.. These guidance algorithms present exploration-exploiatiii trade- off (9), balancing adherence to the pretrained model for feasibility against maximizing the external guide score, which may require exploring out-of distribbttiinno trajectories.
As the main bottleneck in guided planning lies in the limited quality of trajectory samples produced by the pretrained models, prior works have primarily focused on improving general sample quality and mitigating planning artifacts, while the guidance algorithms themselves remain relatively un- derexplored. The majority of recent studies on diffusion planners emphasize advancing supervised planning capabilities 2 10. They are categorized into sequential [TO] [1] hierarchical B 4. and fine-tuning 12 [2] approaches. These works improve the modeling of the underlying system dynamics from the static offline RL benchmarks. Recent works successfully solve challenging offline
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

benchmarks by reformulating the training scheme, learning value estimator, and test-time scaling, Another recent research direction in diffusson planners aims enhance zero-shot planning capabili- ties [3]. Test-time tasks are shifted from the training distribution, and planners are given access to a pretrained model along with dense reward signals to adapt to these unseen tasks effectively.
However, test-time planning capabilities are often evaluated in relatively simple optimization tasks, such as minimizing the distance to optimal trajectories or matching the outputs of pretrained classifiers trained on expert demonstrations 4 and predominantly within in-distribution trajectory settings [2] 4. Typical benchmarks include maze navigation tasks [0] where agents minimize distance to single goal, or block stacking tasks [ where unique target configuration maximizes reward. These tasks generally involve convex optimization problems, where a unique global optimal trajectory maximizes smooth guide function. In addition, similar to model-free RL methods (14], guided planning often relies on pretrained value estimator trained on supervised trajectory data; however, collecting optimal trajectories for each new task is often impractical or infeasible, particularly for tasks with complex dynamics or limited simulation control.
In this work, we address a fundamental challenge of existing diffusion- based planners: while they excel generating low-level action sequences, most real-world planning tasks require decision- making over high-level abstractions. These abstractions often introduce non-convex guide functions or -inddifferenti constraints, making them incompatible with conventional test-time guidance methods that assume smooth, convex optimization landscapes. For example, maze navigation with ninidfffrretiable rule into the planning process. In reward
intermediate goal bypassing (2] introduces
multi-reward block stacking, the agent must reconcile multiple signals to determine the most favorable block arrangement. These scenarios underscore the need for guided planning algorithms that flexibly accommodate complex test-time specifications while ensuring both trajectory feasibility and task-specific fitness, Recently, several training-fiee free guidance algorithms have been proposed in the image domain (15H88) but their capabilities are typically restricted to smooth differentiable guide functions. Since test-time guide functions can take any form, there is need for a flexible planning algorithm that can handle broad class of guide functions.
We propose Tree-guided Diffusion Planner (TDP), which formulates test-time planning as a tree search problem that balances exploration via diverse trajectory samples and exploitation via guided sub-trajectories. While pretrained diffusion planners model underlying system dynamics, TDP samples high-reward (i.e., high guidance score for the test task) solution trajectories conditioned on the learned dynamics in a zero-shot manner. TDP equips the pretrained diffusion planner model with the ability to reason over higher-level objectives. TDP outperforms state-of-the-art planning methods on challenging tasks with non-convex guide functions and nonedifferentiable constraints across all test scenarios. TDP enables flexible task-aware planning without requiring expert demonstrations.
2 Related Work
Existing Diffusion Planners. Despite strong performance on standard offline benchmarks, existing diffusion planners face limitations in zero-shot planning:
Sequential approaches [10 explore one action at a time, which works well in single-goal convex tasks but struggles with multi-goal tasks where different goals have varying test-time priorities. In such settings, they often converge local optima rather than discovering distant, high-priority goals. Furthermore, sequential approaches like MCTD [1] typically require training task-specific value estimators to guide their single-step decisions, limiting applicability to zero- shot scenarios where no task-specific training is available. In contrast, TDP performs multi-step exploration through diverse bi-level trajectory sampling, which better handles challenging multi- goal scenarios without requiring additional training components beyond the pretrained planner.
4 rely on training-time supervision to learn sub-goal distri- butions. They perform well when both initial and goal states are given, but struggle on unlabeled zero-shot tasks such as the test-time gold bold-picking task (2). In standard maze navigation bench- marks, Hierarchical Diffuser 4 tends to generate shortest-path trajectories when initial and goal states are specified. However, the gold-picking task poses fundamentally different challenge: the goal is hidden and often misaligned with the shortest path.
Hierarchical diffusion planners
3
Diffusion model predictive control (D-MPC) [2] adapts to changing dynamics via few-shot fine-tuning with expert demonstrations, but struggles with unseen long- long-horizon tasks and complex
2

---

behaviors as standard dynamics models p(s(a) struggle to capture long- cong-context reward structures effectively. In contrast, TDP models the joint distribution p(s, a) to enable solving long- morizon and multi-goal tasks and is fully zero-shot planner that operates without test-time demonstrations.
Training-free Guidance. Training free guidance methods leverage structural priors and domain knowledge for control without additional learning. Classical planners (e.g., A*, potential fields) [[9 23] compute feasible trajectories through graph search or geometric reasoning. In continuous domains, trajectory optimization and model predictive control [24 (25] refine actions iteratively using known dynamics. Local search techniques (e.g. . hill climbing), guided policy search, and reward shaping [26] 28] serve strong baselines for structured, domain-specifie control problems but typically lack the flexibility to generalize beyond narrow task settings. In contrast, diffusion-basse test-time planning largets complex tasks that need out-of-distribution generalization and adaptability.
Tree-based Decision Making. Tree structures naturally represent hierarchical sequential decisions. Trajectory Aggregation Tree (TAT) (29] mitigates artifacts in diffusion -eeiirrte trajectories by aggregating similar states near the initial state, but it struggles with complex long-horizen dependen- cies due to its limited aggregation depth early in the trajectory. Monte Carlo Tree Search [30 explores full-horizon trajectories via stochastic roll-outs, but its reliance on discrete actions and reward heuristics limits scalability in nigh-dimensional continuous control tasks with external guidance. In contrast, TDP's bi-level tree framework integrates gradient-basee -aased guidance at both parent and child levels, enabling structured and adaptive planning under complex test-time objectives.
3 Background
3.1 Problem Setting
We consider the test-time reward maximization problem discrete-time dynamics system 8:+1 [St. a.) via a pretrained planner model, where the agent has access to the user-defined guide function I (T), which indicates the fitness of the generated trajectory т. As per-timestep reward does not guarantee the optimality of low-level action (e.g., non-convex reward landscape), planning capability based on exploration is required to find the optimal trajectory + that maximizes J. The agent must find an action sequence that maximizes the guide score within a limited number of steps:
+ = d1.
arg max I (80. 01:T) subject to
Tpro S <T TTax
(1)
T, aL.T
Planning horizon Tred determined by the choice of planner model. Model-free RL. methods with predict a single action at each timestep so Tpes = 1, whereas diffusion planner [I] predicts a sequence of actions di:T,ea at once. As diffusion planners predict more future states, they benefit from capturing longer-term contextual information. For example, in long- horgzoorioon multi-goal navigation task [34], an agent may encounter several intermediate suboptimal goals, but reaching farther goal yields substantially larger reward, requiring planning several steps rather than greedy pursuit of nearby rewards. Planning over longer horizons can enhance performance on more challenging tasks, particularly when future rewards more significant [35] 36).
single step execution [ 6
3.2 Test-time Guided Planning with Diffusion Models
The standard approach to guide diffusion planning test time is use naïve gradient guidance 18] which progressively refines the denoising process by combining the score estimate from the uncondi- tional diffusion model with the auxiliary guide function 4 It approximates the reverse denoising process as Gaussian with small perturbation if the guidance distribution li(T)) is sufficiently smooth and the gradient of the guide function is -ndeeeddeenn
p(T:-I/T;) Po(T._I7T,)((TT) NT
gZ E
where g , log h(T:) is the gradient of the guidance distribution 37,, is guidance strength,
and μ,, Σ are the mean and covariance of the pretrained reverse denoising process. On the other hand, classifier guidance (CG) 7 and classifier-fres guidance (CFG) (38 are also broadly used in guided planning 3 39]. However, they require access to expert demonstration data train either a time-dependent classifier model or conditional diffusion planner model based on trajectory rewards
3

---

Diffuser
Diffuser
TDP
Rollout Trajectories for the Gold-Picking in Maze2D. the Maze2DD-Large environment 40) the agent must collect additional gold objective (yellow) positioned off the shortest navigation path. Trajectories are generated by Diffuser [ (with gradient guidance), Diffuser" 29 and our method. See Sec. 5.2 for details.
Figure 1:
for given task. Despite their simple yet powerful architecture, it is expensive to extend CG and CFG test-time as it requires collecting expert demonstrations for each new task and retraining the model. The ultimate goal of test-time guided planning is adaptive planning, identifying trajectories that satisfy user requirements using a pretrained diffusion planner without additional expert supervision.
3.3 Challenges in Guided Planning
Guided planning has primarily been evaluated on simple tasks where expert demonstrations are available the guide function is convex and differentiable. As a result, prior work has focused on improving the sampling quality of the pretrained diffusion model, which was the main bottleneck for test-time guidance in these simplified settings 4T. Despite the importance of improving trajectory sampling quality, there has been limited investigation into how guidance algorithms themselves adapt increasing task complexity and respond various forms of test-time objectives. In this section, we first study a fundamental challenge in guided planning: the conpiitttioo-eplaaatttooo trade-off, and then discuss the limitations of naive gradient guidance, standard test-time planning approach.
Iaporaatoon-Exploitattoon Trade-off. Test-time guided planning employs two distinet score functions: score esti- mate from the pretrained diffusion model and user-defined guide Not only generate feasible trajectory but also to maximize its fitness, the agent is encouraged to bal- ance the exploitation of the pretrained model and the ex- ploration of novel trajectories. Gradient- baadd guidance typically requires selecting guidance strength a bal- ance adherence to the guide signal and trajectory fidelity. However, is highly dask-depen,nnt, and exhaustive tuning across tasks introduces significant overhead during evalu- ation. For example, Fig. [ illluttrates several bold-picking tasks within fixed maze map. The optimal value of a varies across tasks, depending on how far the gold location deviates from the unconditional navigation trajectory, which favors the shortest path (see the supplement for more details).
Riow
Rnigh
X
data
ideal
Gradient Daldance
b.
intermediate goal
Figure 2: Limitation of Gradient-based Guided Planning. In-distribution pref- erence. b. Naïve gradient guidance is in- compatible with non-differentiable rules.
In-distribution Trajectory Preference. Diffusson models capable of generating compositional behaviors | 42
but often get stuck in local optima due to insufficient explo-
ration of the trajectory space. Most previous works on guided planning did not adequately address the exploration-explee exploitation trade-off, as they typically assume convex (or concave) guidance with single optimal trajectory that globally maximizes the reward 43 However, real-world tasks often involve multiple objectíves with different priorities, where the agent must explore the trajectory space avoid sub-optimality. Pretrained diffusion planners tend to favor generating in-distribution trajecto- ries that align with previously seen data, rather than discovering novel compositional solutions [4.

---

Consequently, gradient- masdeent-based guidance algorithms do not effectively address this dilemma, as they often prioritize local optimal trajectories within the learned distribution (see Fig. 2a).
Non-differentiable Rule. Gradient- guidance algorithms face significant challenges in plan- ning tasks with non-differentiable constraints. Since diffusion planners are trained on trajectories with fixed start and end states, they struggle to generate feasible paths that incorporate additional intermediate goals. For example, described in Fig. 2 -b, navigation trajectories conditioned to test-time intermediate goal often result in suboptimal (left) or infeasible (right). This introduces eon-differentiable constraint from the planner's 's perspective, as the requirement to pass through a specific state imposes discrete structural condition not reflected in the training distribution. Similar challenges arise in other domains of diffusion- based generation, such as enforeing chord progression in music generation 44 or satisfying chemical rules in molecule generation [45]. Both introduce hard constraints that are difficult to optimize with standard gradient-basee methods.
Method
We introduce Tree-guided Diffusion Planner (TDP), zero-shot test-time planning framework leveraging pretrained diffusion planner for adaptive trajectory generation. While naïve gra- dient guidance often converges to local optima due to limited gradient signals, TDP addresses this by combining diverse trajectory samples (ex- ploration) with gradient-guides sub-trajectories (exploitation) to identify optimal solutions (see Fig. This tree structure enables coverage a broad range of reward landscapes. By branching into diverse regions, TDP increases the chance of finding globally high-reward solu- tions that naïve gradient methods may miss.
Parent Branching
Sub-Tree Expansion
gradient guidance
(O.D)
Data space
VJ
Figure 3: Tree-guided Diffusion Planner (TDP). TDP constructs a trajectory tree combining diverse parent trajectories (navy) and guided sub-trajectories (red) in the 2D data space. The 3D surface represents the reward landscape, with peaks indicating high-reward regions.
Appendix A outline the overall TDP pipeline and present the full algorithms. We detail the core modules of TDP: state decomposition (Sec. [.1), parent branching (Sec. 4.2). and sub- tree expansion (Sec. 4.3
4.1 State Decomposition
Given guide function for the test-time task, states are autonomously decomposed based on gradient signals: observation states receive non-zero gradients, while control states receive none. This gradient- based criterion enables scalable and domain aggnosic decomposition at planning time. Observation states are directly steered by the guide function, whereas control states are unaffected by the guide function but govern the underlying system dynamics that support high-level objectives.
For instance, the KUKA robot arm environment provides state vectors containing multiple features such as robot joint angles and block positions. Since TDP operates as zero-shot planner, does not have prior knowledge of the state category for each feature of the state vector. Given test-time block stacking task with distance-based guide function, TDP autonomously categorizes each feature value in the state vector using Algorithm De. as an observation state; if zero, it is classified as control state. Consequently, features related to robot physics detected control states. In contrast, block position (xy) features detected as observation states, because the block-stacking guide function is only affected by the block positions.
[ It evaluates whether the gradient of the guide function with respect to the ith feature (i.e., 8.7 is zero or non-zero. If non-zero, the äth feature is classified
4.2 Parent Branching
In the first phase of our bi-level planning framework, control states are applied to fixed-potential particle guidance (PG) 46 to explore diverse control trajectories. PG promotes diversity among generated samples within a batch. For instance, when moving block to target position, multiple
5

---

control trajectories can accomplish this task. Standard gradient-based approaches often suffer from in- distribution bias and limited exploration, constraining trajectory diversity. In contrast, TDP enhances exploration through this procedure, generating what we term parent trajectories.
Specifically, fixed-potential PG is implemented using the gradient of a radial basis function (RBF), denoted V which can be computed directly from all pairwise distances between control trajectories within a batch. Unlike conventional gradient guidance methods that pull samples toward high-reward regions, PG introduces repulsive forces that push samples apart in the data space. This leads to a broad coverage of dynamically feasible trajectories independent of task objectives. Although fixed -eteentttt PG incurs some inference overhead from computing kernel values between all trajectory pairs, it is significantly more sample-efficies than -otenntatttt variants [46], while remaining training-fres and modular. A single denoising step for parent branching is denoted as:
M control f Pecmtrot a2 EVV''
Miae + a V E'V' ((aba))
Mob
where sootrol and Hobs denote the control and observation components of the predicted mean of the denoising trajectory at timestep Particle guidance (( introduces repulsive updates among control states to diversify denoising paths. In contrast, gradient guidance ( steers observation states toward task-relevant regions defined by the guide function. This enables wide exploration in the control state space, which helps to discover diverse observation state configurations and exposes the planner to richer gradient signals from the guide function J.
Notably, TDP formulates single joint conditional distribution with an integrated guidance term. As shown in Eq. 2 the overall guidance distribution is defined as h(T,) (Ti) (h where hgg(() (-) denotes the gradient guidance component, and hgg (.) denotes the particle guidance component. Since both components jointly condition the same pretrained reverse denoising process, the perturbed reverse denoising process is approximated as ((T IT.) ~ WT + @TDP 9TDP, Σ') where
gTDP = V log (T.) (pp((T))) = V log hgg (T:) + V log hp (T.) :) = Iex + 9r8. Therefore,
μ i 1 + TTDP 9TDP
denotes the integrated guidance term from Eq. where -²= [ M Mobs and ØTDP 9ps
Jog amo (ap V (H(orra)) + ag ( TDP employs single integrated guidance term, which is the sum of the gradient guidance part (geg) and the particle guidance part (Opg).
4.3 Sub-Tree Expansion
In the second phase, we apply fast denoising with reduced steps N, < N, where N the original number of diffusion steps. to refine parent trajectories using task gradient signals. For each parent trajectory, we select a random branch site and generate a child trajectory by denoising from a partially noised version of the parent, conditioned on the preceding segment. Sub-tree expansion proceeds as:
T child qNy (T purent, C) where C {sx 1 and b ~ f(0 Tpeed)
(5)
where C' denotes the parent trajectory prefix, qN, is the partial forward noising distribution with N, N₁ denoising steps, and child is the partially noised trajectory from which the child trajectory is denoised during sub-tree expansion. These child trajectories enable fine-grained local search around the parent branch, improving alignment with the guide signal. Sub-tree expansion offers two key advantages:
Enhance dynamic feasibility of parent trajectories: Diverse parent trajectories benefit exploration, but perturbing the control states may lead to dynamically infeasible plans. During sub-tree expansion, perturbed control states are refined by pretrained diffusion denoising process.
Efficient Local search: Sub-Tree expansion refines observation states of parent trajectories with gradient guidance signal. Parent trajectories serve as initial points to guide child trajectories. Since parent trajectories intended to cover a broad region of search space, local search conditioned on the parent trajectories is an efficient way to find better local optima.
Why is bi-level sampling necessary? We investigate the role of our bi-level sampling framework in handling multi-reward structures, as illustrated in Fig. 3 We characterize problems with both local and global optima and demonstrate that bi-level trajectory generation avoids local optima. Consider

---

trajectory data in a learned subspace, where the pretrained diffusion planner maps Gaussian noise back data space via the reverse denoising process. Given a test-time guide function defined as the sum of Gaussian components (as defined in Proposition IT the following statements hold:
Proposition 1. Initialization problem in gradient guidance with diffusion planner. Assume that the trajectory data X E IH. follows the Assumption 1 in (18V, and given guide J = (X) +
Jx(X) where Ti(X) = exp
I - 10-11 I2(X) = exp (- 20 |X w 11²) W 1 207
(A), U1 is the first eigenvector of A A, 01 < 02 and Em~N < En- [2(
a. Xo NN( 1), then the guided sample Xr converges to local optimal solution.
b. X. ~ qN, (XT, Σ²) where XT an unconditional sample with small perturbation, then the guided sample XT converges to global optimal solution.
The proof is provided in Appendix C Guided sampling initialized from standard Gaussian noise can be steered away from the subspace, converging to off-subspace local optima [88, as illustrated in Proposition a These samples may locally optimize the task objective but lie outside the learned data manifold, resulting in unrealistic or infeasible trajectories. This occurs because standard gradient guidance modifies the pretrained denoising process without preserving the underlying data structure, causing the sampling to drift toward regions that satisfy the guidance objective but violate the training distribution constraints. In practice, many test-time planning problems exhibit off- subspace optima- for example, in maze navigation, unseen obstacles can block the learned shortest path, making pretrained trajectory preferences suboptimal. In contrast, unconditional samples naturally remain close to the learned data subspace since they follow the original training distribution. When guided sampling is initialized from these on-subspace points, it leverage both data structure and gradient information to converge to the global optimum, as demonstrated Proposition T-b.
TDP's bi-level planning framework enables flexible exploration through structural search over trajectories. During parent branching, gradient guidance can be optionally applied to steer the observation states, depending on task characteristics. Unconditional PG supports broad exploration of the data space, effective for discovering diverse solutions In contrast, conditional PG directs exploration toward regions aligned with the guide function, useful for convex objectives where a single high-quality solution is desired (e.g.. PNP tasks in Sec. (5.3). Gradient guidance in parent branching provides a prior for sub-tree expansion but may also limit the breadth of exploration in trajectory space. Each strategy offers complementary henefits: unconditional PG promotes diversity without task-specific priors, while conditional PG enables targeted search.
specifeed or multi- goaa settings.
5 Experiments
We evaluate TDP across diverse zero-shot planning tasks featuring non-convex and don-differentiable objectives. Our experiments assess zero-shot planning capabilities and robustness when addressing unseen test-time objectives. All experimental hyperparameters are reported in Appendix D
While existing diffusion planners (e.g., MCTD 11,, Hierarchical Diffuser 4 excel on standard offline benchmarks, they suffer from zero-shot scenarios discussed in Sec. Consequently, focus comparisons on recent zero-shot planning approaches, particularly Trajectory Aggregation Tree (TAT) [29], Monte-Carlo sampling [43, and stochastic sampling [3] We design our benchmarks to challenge planners with unseen test-time objectives, in contrast to offline benchmarks that only test learned dynamics aligned with the training distribution. For completeness, we provide supplementary comparisons with sequential approaches (i.e., MCTD [II, Diffusion Euffusion-Forcing (IO) on standard maze benchmarks in Appendix ! Notably, despite being designed specifically for zero-shot planning, TDP still surpasses these sequential approaches on standard benchmarks.
5.1 Baselines and Ablations
Diffuser [I] Diffusion- approach that plans by iteratively denoising complete trajectories.
AdaptDiffuser 2
Fine-tune pretrained Diffuser with synthetic expert demonstrations.
Diffuser" (TAT) 29]: Aggregate diffusion samples into a tree structure, bounding trajectory
artifacts. Although TAT equips the pretrained Diffuser with better performance in offline RI. tasks,
its zero-shot planning capability is strictly constrained by standard diffusion sampling.
7

---

Table 1: Result of Mazeld Gold-picking, Mean performance of our method and baselines across 20 tasks (5
random seeds each). Four types of test maps are depicted Appendix
= denotes standard error.
Environment
Diffuser
Diffuser"
MCSS
MCSS+SS
TDP (we dhild)
TDP (wi)
TDP
Maze2d
Medium
10.1 25
12,3 a
17.2 3.
17.4 32
19.0 3:35
39.1 4:42
39,8 4:42
Maze2d
Large
4.3 = 18
26
25.0 + 88
21.2 15
30.4 2
41.1 4.4
47.6 =41
Single-task Average
6.8
10.8
21.1
19.3
24.7
40.1
43.7
Multi2d
Medium
2.
8.6
32.3 +
29.2
35.3
75.9 3030
74.7
Multi2d
Large
9.9
23.1 a
57.5
58.0 33
59.1
64.9 30
70.0
Multi-task Average
8.8
15.9
44.9
43.6
47.2
70.4
72.4
Monte-Carlo Sampling with Selection (MCSS) 43]:
Sample multiple trajectories from the pretrained Diffuser and select the best trajectory based on the guidance score. Monte Carlo sampling methods effectively explore the solution space and approximate optimal trajectories [47].
Stochastic Sampling (MCSS+SS) 13 MCMC-based, training-fiee freee guided planning method requiring M times more computation than MCSS, where M is the number of iterations in the inner loop of diffusion sampling. Following (13), set M 4 in all our experiments.
TDP (wle childy: Remove the Sub-Tree Expansion phase from TDP, relying solely on parent trajec- tories generated by conditional PG sampling without further refinement through sub-trajectory sampling. This isolates the contribution of parent trajectories to overall planning performance.
TDP (wo PGY Ablate the particle guidance step in Parent Branching, relying solely on gradient guidance to generate parent trajectories. This highlights PG's s role in producing diverse parents that serve as initialization points for Sub-Tree Expansion.
5.2 Maze2d Gold-picking
We extend the single gold-picking example [2] in the Maze2D environment 440 to multi-task benchmark. The agent is initialized at a random position in the maze and has to find the gold at least once before it reaches the final goal position. As discussed in Sec. 3.3] the gold-picking task is planning problem with a test-time non-differentiable constraint, where the agent must generate a feasible trajectory that satisfies an initial state, a final goal state, and an intermediate target (the gold position). In addition, the task is black-box problem that requires inferring the intermediate goal (i.e., gold) location using only distance-based guide function, without access the gold's exact position. To address this setting with diffusion planners, prior work 2 proposes an approximate,
Σird ||s; &gald| where &guld denotes the gold position. While this approximate function is used for the gradient guidance sampling, the true guide function min ||s; ggulal is used for selecting the best one from the generated candidates. (T) xe(1... F
distance-based guide function J(T)
We report the performance of TDP and baselines in Table 1 TDP consistently outperforms both Diffuser" and MCSS across single- and multi-task settings. Notably, even TDP (win child) achieves approx. a 7% performance improvement over MCSS overall. TDP generates farther sub-trajectories through sub-tree expansion, allowing the planner to localize the gold position better and collect stronger gradient signals from the surrounding region to guide the trajectory effectively. This bi-level trajectory sampling approach enables the discovery the hidden gold location within the map.
5.3 KUKA Robot Arm Manipulation
We evaluate the test-time planning performance of TDP and baselines on robotic arm manipulation tasks. Diffuusin planners are pretrained on arbitrary block stacking demonstrations collected from PDDLStream (88 and are typically evaluated on downstream tasks such as conditional stacking M and bick-and-place (2) where test-time goals are specified to the planner. Both tasks aim to place randomly initialized blocks into their corresponding target locations in predetermined order. The lok-and-place task is more challenging because it requires placing each block at unique target without access to expert demonstrations. To better isolate test-time planning capability, we evaluate a variant of the conditional stacking task [2] without using the pretrained classifier on expert data. We refer to this variant as PNP (stack), and the original task is denoted as PNP (place). For both tasks, the pretrained diffusion planner is guided to generate sequential manipulation trajectories,
8

---

one for each block, toward its target location Starget using naive guidance objective defined as ((T)=-X) E; ||s; Starget |- As shown in Table TDP achieves an average improvement of 10% over MCSS and 20% over TAT in two PNP tasks, demonstrating strong generalization to diverse task configurations. Notably, TDP (wo PG) outperforms MCSS by 18% on PNP (place), underscoring the role of our sub-trajectory refinement mechanism in such out-of-distributin planning scenarios.
Moreover, we carefully design a more challeng- ing test-time manipulation task which extends PNP (place), namely pick-and-wi and-where-too to-pe-e (PNWP). As shown in Fig. 4 the agent must find global op-
Reward
Locool
timal trajectory given non-concave guide function [(T = E.c0 ||s; smaal - ||s; miad + e3 ||8; Sglobal where c1, c22 C3 are positive constants and Stecal, 8mid: &global are the local, middle, and global target positions, respectively. Both &global and 8tecal equidistant from the robot arm's 's attachment point. Since the local optimum has a wide peak while the global optimum has a narrow peak, agents easily get trapped in local optima without sufficient exploration. While PNP requires fitting blocks into target config- uration, WN challenges planners to distinguish be- tween globally optimal and suboptimal arrangements.
glabbal
Xiocal
gloobal
Figure Pick-and- Where-to- Place (PNWP). PNWP evaluates the agent's 's exploration capacity in the robot arm manipulation environment. The agent must infer suitable placement locations for each block based on the reward distribution and plan corresponding pick-and- actions.
Table 2: Result of Robot Arm Manipulation. Mean performance of baselines, ablations, and our method. Diffuser uses sample, while others samples E {6, 12, 18, 24) with 100 seeds. denotes standard error.
Enviromment
Diffuser
Diffuser"
AdaptDifluser
MCSS
MCSS+SS
TDP (wl-
TDP (ma
TDP
34.72
35.53
66.63
66.81
PNP (stac)
51.5 si
60.08 1016
60.54 1aw
59.91 1aw
56.8 1.36
60,00 0.1
59.42
1:00
61.17 121
PNP (place)
32.19
36.94
PNP Average
36.41
40.76
48.36
45.64
46.15
46.10
48.18
49.06
We report the performance of TDP, baselines, and ablations Table 2 Mono-level guided sampling methods (i.e., AdaptDiffuser, MCSS(+SS), TAT, and TDP (wro child)) tend converge to local optima, often stacking all blocks at a single position, since a landscape of local optima is spread out in a broader range as shown in Fig. In contrast, bi-level sampling approaches (i.e., TDP and TDP (wlo PG)), which combine parent branching and sub-trajectory refinement, are better able to identify globally optimal placements consistently. Notably. TDP outperforms AdaptDiffuser both the standard benchmark (PNP) and on the custom task (PNWP). This demonstrates that TDP's s rest-time scalability and generalization enable zero-shot planning that surpasses the raining--pertask approach.
Pairwine trop distaxe
b.
X4
Atooat
x giobat
giobal
MCSS TAT
TOP TOP
TOP
MCSS
TDP
Figure 5: Diverse Trajectory Generation. a. Mcan pairwise distance computed over 32 trajectories, averaged across 100 planning sceds in PNWP. Error bars indicate standard error. b. Visualization of trajectory generation process and rollout results of MCSS and TDP.
The key advantage of our method over the baselines comes from employing unconditional PG in the parent branching phase, enabling broad exploration of the trajectory space. Both TDP and its ablations produce diverse trajectories (see Fig. a): however, TDP (wlo child) often fails to find the global optimal placement due to the use of conditional PG in parent branching, which can bias samples toward local optima. The mean pairwise trajectory distance decreases when PG is combined with sub-tree expansion, as the generated gub-trajectories share segments with their corresponding parent trajectories. Unconditional PG enables the generation of diverse, gradient-free parent trajectories. When combined with sub-tree expansion guided by the objective, this bi-level strategy supports compositional solutions that successfully stack blocks into global optimal placement (see Fig. 5 More detailed information on the evaluation metries among these tasks is available in Appendix

---

5,4 AntMaze Multi-goal Exploration
t=4
Figure 6: AntMaze Multi-goal Exploration. A priority-aware multi-goal exploration designed to evaluate exploration in AntMaze locomotion planning. A diffussion planner predicts the next Ieed 64 steps (bright areas on the map) using a net guide signal from multiple goals, with maximum horizon of 2000 steps.
We finally evaluate test-time multi-goal exploration capability on AntMaze [40], which more challenging than Maze2D due to its nigh-dimensional observation space for controlling the embodied agent. complexity causes the pretrained diffusion planner to predict shorter horizons than the full trajectory length [49]. We design a new multi-goal task in which the agent must visit all goal positions in the correct order of priority, as specified by the guide function J(T) Eges hg
exp (-EssT ||s; sn /oz) where is the set of goal positions and fgg og are parameters for gaussian guide function each goal g E G. For example, the agent in Fig. first visits goal Ø2 at t= 13. If it subsequently visits (I 94, and I3 after i = 1z. it successfully reaches all four goals in the sequence 92 + 91 + 94 + 93- However, two precedence rules are violated (g2 -> 91 94 -> g3) while the remaining four (92 + 94, 92 + 93. 91 - 94. and 91 + 93) are satisfied. In this scenario, the agent achieves a goal completion score 4/4 but only 4/6 priority sequence match accuracy. Maximum accuracy of 6/6 can only be achieved by visiting all goals in the correct prioritized order.
Table 3: Result of AntMaze Multi-goal Exploration. Mean performance of baselines, ablations, and our method. Diffuser uses 1 sample, while others samples E (32, 64, 128, 256) with 100 sceds. We report three metrics: numher of found goals, sequence match score, and average timesteps goal. denotes standard erro.
Metric
Diffuser
Diffuser
MCSS
MCSS+SS
TDP mil child
TDP (wlo PG)
TDP
# found goals f sequence match f
1.5 a3
22s
61.3 =0
62.8
65.8 1
64.6
66.1 =
1.56
1.2 222
30.2 1.
31.2 13
33.8 15
32.7 1 1.4
33.8 1.4
# timesteps per goal
5333.3
4020.1
612.1
604.4
574.3
$78.3
558.4
We report the performance of TDP, baselines, and ablations in Table 3 using three metrics, with MCSS in both the
detailed definitions in Appendix TDP achieves about 11% improvements number of found goals and the sequence match score, while also reducing timesteps per goal. The ablations highlight complementary roles the components: when PG removed (TDP (w/o PG)), all three metrics degrade, confirming that PG is essential for both goal discovery and sequence alignment. In contrast, removing child branching (TDP (wlo child)) maintains performance on the first two metrics but requires more timesteps per goal, suggesting less efficient exploration in the environment.
6 Conclusion
In summary, we propose TDP, a flexible test-time planning framework that leverages pretrained diffusion planner via bi-level snnjectory-sampliig process without training. By balancing trajectory diversity and gradient-guie addresses key limitations of conventional test-time- test-time-uuided planning. Empirical results across both structured and compositional manipulation tasks demonstrate consistent performance gains over existing baselines, particularly in scenarios that demand -idtr--diittiittion generalization. Our experiments also highlight the robustness of the framework handling non-convex, multi-objective guidance and difdeifferentiable constraint problems, where naïve gradient guided methods often fail.
refinement via branching structure sampled trajectories, our method
Limitation and Future Work. While TDP outperforms existing planning approaches across a suite of challenging test-time control tasks, our bi-level trajectory generation process incurs additional computational cost due to the expanded search in trajectory space and the pairwise trajectory distance calculations. We analyze the additional computational time required for the two PNP tasks and the PNWP task in Appendix G Future work may explore more efficient search strategies or learned priors to reduce overhead while ensuring sufficient exploration and preserving planning performance.
10

---

Acknowledgements
This work was supported by HTTP grant (RS-2021-1121 AT Graduate School Program at Seoul National Univ. (5%) and RS-2022--25442333 AI Star Fellowship Support Program (Scoul National Univ.) (60%)) and NRF grant (No.2023R1AIC200771111 (35%)) funded by the Korea government (MSIT).
References
[1] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205. 00999999,,, 2022.
[2] Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni. Masayoshi Tomizuka, and Ping Luo. Adaptdiffuser: Diffusion models as adaptive self-evolving planners. arXiv preprint arXiv:2302.011877 022.002..1877.. 2023.
[3] Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. In Andreas Krause, Emma Brunskill, Kyungbyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference Machine Learning (ICML), volume 202 Proceedings of Machine Learning Research, pages 20035-20064. PMIR, 23-29 Jul 2023.
[4] Chang Chen, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchical planning with diffusion. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[5] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems (NeurlPS), 13:1179-1191, 2020.
[6] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2/10006/16 2021.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances Neural Information Processing Systems (NeurIPS), 34:8780-8794, 2021.
[8] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-makingg? arXiv preprint 175.57221..16657. 2022.
[9] R.S. Sutton and A.G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, 105) 1054, 1998. doi: 10.1 N0.1109/TNN.190 (wT.TNN...99..1111 1998.712192.
[10] Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full- uoll-sequence diffusion. Advances in Newral Information Processing Systems (NeurIPS), 37:24081- 121-2008-22415,,, 2024.
[11] Jaesik Yoon, Hyconsco Cho, Doojin Back, Yoshua Bengio, and Sungjin Ahn. Monte carlo tree diffusion for system 2 planning. arXiv preprint arXiv:2502.0723 2025.
[12] Guangyao Zhou, Sivaramakrishman Swaminathan, Rajkumar Vasudeva Raju, J Swaroop Guntupalli, Wolfgang Lehrach, Joseph Ortiz, Antoine Dedieu, Miguel Lázaro-Gredilla, and Kevin Murphy. Diffusion model predictive control. arXiv preprint arXiv:2410.08: 05344, 2024.
[13] Yanwei Wang. Lirui Wang, Yilun Du, Balakumar Sundaralingam, Xuning Yang. Yu-Wei Chao, Claudia Pérez-D' Arpino. Dieter Fox, and Julie Shah, Inference- policy steering through human interactions. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 15626-15633 IEEE, 2025.
[14] Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois. Shunshi Zhang, Guodong Zhang, Picter Abbeel, and Jimmy Ba. Benchmarking model-hased reinforcement learning. arXiv preprint 10XXx:1907.00000 2019.
[15] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. The Eleventh International Conference on Learning Representations (ICLR), 2023.
[16] Jiaming Song. Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming- Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug and-aan-ppp and-play controllable generation. Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202 of Proceedings of Machine Learning Research, pages 18483-32499 PMLR, 23 29 Jul 2023.
11

---

[17] Jiwen Yu, Yinhuai Wang. Chen Zhao, Bernard Ghanem, and Jian Zhang, Freedom: Training- frreinng-reeeeeoe energy- guided conditional diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 23174-23184, 2023.
[18] Yingqing Guo, Hui Yuan, Yukang Yang, Minshuo Chen, and Mengdi Wang. Gradient guidance for diffusiin models: An optimization perspective. Advances in Neural Information Processing Systems (NeurIPS),
2024.
[19] Peter E. Hart, Nils J. Nilsson, and Bertram Raphael. A formal basis for the hcuristic determination of minimum cost paths. IEEE Transactions Systems Science and Cybernetics, 4(2):100-100 1968. doi: 10.1109/TSSC.196 196..30116
[20] Edsger Dijkstra. A note on two problems in connexion with graphs. Numerische mathematik, 1(1): 269-271, 1959.
[21] Lei Tang, Songyi Dian, Gangxu Gu, Kunli Zhou, Suihe Wang, and Xinghuan Feng. A novel potential field method for obstacle avoidance and path planning of mobile robot. In 2010 3rd International Conference Computer Science and Information Technology, volume 9, pages 633-637, 2010. doi:
10.1 109/TCCSTT221155555009
[22] Roland Geraetts and Mark H. Overmars. comparative study of probabilistic roadmap planners. In Jean- Daniel Boissonnat, Joel Burdick, Ken Goldherg, and Seth Hutchinson, editors, Algorithmic Foundations of Robotics V. pages 43-57. Springer Berlin Heidelberg. Berlin, Heidelberg, 2004. ISBN 978-3-540-45058-0 doi: 10.1007/978-3 1..1007/978--550-49550 3-540-45058-0- 4.
[23] Juan Cortés and Thierry Siméon. Sampling-based motion planning under kinematic loop-closure constraints. In Michael Erdmann, Mark Overmars, David Hsu, and Frank van der Stappen, editors, Algorithmic Foundations of Robotics VI. pages 75-90. 90. Springer Berlin Heidelberg, Berlin, Heidelberg, 2005. ISBN 188-3-540-315000 doi: 10.1007/10991541..
[24] Brandon Amos, Ivan Jimenez, Jacob Sacks, Byron Boots, and Zico Kolter. Differentiable mpe for end-to-end planning and control. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.
[25] Max Schwenzer, Muzaffer Ay, Thomas Bergs, and Dirk Abel. Review on model predictive control: An engineering perspective. The International Journal of Advanced Manufacturing Technology. 117(5): 1327-1349, 2021.
[26] Sergey Levine and Vladlen Koltun. Guided policy search. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning (ICML), volume 28 of Proceedings of Machine Learning Research, pages 1-9, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
[27] Alexander Trott, Stephan Zheng, Caiming Xiong, and Richard Socher. Keeping your distance: Solving sparse reward tasks using self-balancing shaped rewards. Advances in Neural Information Processing Systems (NeurlPS), 32, 2019.
[28] Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach reward shaping. Advances in Neural Information Processing Systems (NewrIPS), 33: 13931-11959,, 2020.
[29] Lang Feng. Pengjie Gu, Bo An, and Gang Pan. Resisting stochastic risks in diffusion planners with the trajectory aggregation tree. arXiv preprint arXiv:2405.170 17879, 2024.
[30] Maciej Swiechowski, Konrad Godlewski, Bartosz Sawicki, and Jacek Maádziuk. Monte carlo tree search: review of recent modifications and applications. Artifccial Intelligence Review, 56(3):2497-2562, July 2022. ISSN 1573-7462. doi: 10. 00.1007/11462-000 0222100422--22222...
[31] Daniel Hennes and Dario Izzo. Interplanetary trajectory planning with monte carlo tree search. In International Joint Conference on Artificial Intelligence (IJCAI), pages 769-775, 2015.
[32] Yann Labbé. Sergey Zagoruyko, Igor Kalevatykh. Ivan Laptev, Justin Carpentier, Mathieu Aubry, and Josef Sivic. Monte-carlo tree search for efficient visually guided rearrangement planning. IEEE Robotics and Automation Letters (RA-L), 5())33115-372,, 2020. doi: N0.11009RAA.2220229880
[33] Xiaoxiao Guo, Satinder Singh, Honglak Lee, Richard Lewis, and Xiaoshi Wang. Deep learning for real-time atari game play using offline monte-carlo tree search planning. In Z. Ghahramani, M. Welling. C. Cortes, N. Lawrence, and K. K.Q. Weinberger, editors, Advances in Neural Information Processing Systems (NeurIPS), volume 27, Curran Associates, Inc., 2014.
12

---

[34] Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Learning (ICML). pages 7750-7761. PMLR, 2020.
[35] Lukasz Kaiser, Mohammad Babacizadch, Piotr Milos, Blazej Osinski, Roy Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based rein- forcement learning for atari. arXiv preprint 1003::99330000 2019.
[36] Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839);604-609 December 2020. ISSN 1476-4687. doi: (0.1038/441586-00 026-00000000---44
[37] Jascha Sohl-Dickstein Eric Weiss, Niru Mahoswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. International Conference on Machine Learning (ICML), pages 2256-2265. pmlr, 2015.
[38] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12590 2022.
[39] Siyuan Zhou, Yilun Du, Shun Zhang, Mengdi Xu, Yikang Shen, Wei Xiao, Dit-Yan Yeung, and Chuang Gan. Adaptive online replanning with diffusion models. In Thirry-sevnnth Conference on Neural Information Processing Systems (NeurIPS), 2023.
[40] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072) 2020.
[41] Kyowoon Lee, Seongun Kim, and Jaesik Choi. Refining diffusion planner for reliable behavior synthesis by automatic detection of infcasible plans. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023.
[42] Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation and inference with energy based models. arXiv preprint arXiv:2004.06083 2020.
[43] Haofei Lu, Donggi Han, Yifei Shen, and Dongsheng Li. What makes a good diffusion planner for decision making? In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
[44] Yujia Huang. Adishree Ghatare, Yuanzhe Liu, Ziniu Hu, Qinsheng Zhang, Chandramouli S Sastry, Siddharth Gururani, Sageev Oore, and Yisong Yue. Symbolic music generation with non- non-difeerentiable rule guided diffusion. arXiv preprint arXiv:2402. 14285 2024.
[45] Yuchen Shen, Chenhao Zhang, Sijie Fu, Chenghui Zhou, Newell Washburn, and Barnabas Poczos. Chemistry-inspired diffusion with non-differentiable guidance. The Thirteenth International Con- ference on Learning Representations (ICLR), 2025.
[46] Gabriele Corso, Yilun Xu, Valentin De Bortoli, Regina Barzilay, and Tommi S. Jaakkola. Particle guidance: non-i.idd. diverse sampling with diffusion models. In The Twelfth International Conference Learning Representations (ICLR), 2024.
[47] Alexander Shapiro. Monte carlo sampling methods. In Stochastic Programming, volume 10 Handbooks in Operations Research and Management Science, pages 353-425. Elsevier, 2003. doi: https://doi.org// 10. (16650927-0007(03)1000000)
[48] Caclan Reed Garrett, Tomás Lozano- Lozano-Pérez, and Lestie Pack Kaelbling. Pddlstream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning. In Proceedings of the international conference on automated planning and scheduling, volume 30, pages 440 -48,, 2020.
[49] Zibin Dong, Yifu Yuan, Jianye HAO, Fei Ni, Yi Ma, Pengyi Li, and YAN ZHENG. Cleandiffuser: An easy-to-use modularized library for diffusion models in decision making. In The Thirty-eight Conference on Neural Information Processing Systems Datosets and Benchmarks Track, 2024.
[50] Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal-gradient methods under the polyak-lojasiewic condition. In Joint European conference on machine learning and knowledge discovery in databases, pages 795 811. Springer, 2016.
[51] Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Oghench: Benchmarking offline goal-conditioned rl. arXiv preprint arXiv:2410.20092 2024.
[52] Cameron B. Browne, Edward Powley, Daniel Whitehouse, Simon Lucas, Peter Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and Al in Games, 4(1): 43, 2012. doi: 10. (0.11999CIAAG.2222222188888 2012.2186810.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction explicitly outline the claims and contributions, which are consistent with our experimental findings.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed limitation in the conclusion section. (see Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
We provide the proof of our proposition in Appendix C
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
We provide comprehensive implementation details through step-by-step by-s-p guide- lines in Appendix A B which outline TDP's complete pipeline. Additionally, all hyperpa- rameters for every experiment reported in the main paper are documented in Appendix D ensuring full reproducibility our results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Code with instructions to reproduce the main results is available on the project website.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
oliccy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccpublle wwwppppgpuuiiieeeee /otpc//uuuieee/oeeuubiiiiii
(CodeSubiisse wwtauubmmsssooni ssioniolccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental details are provided in the appendix/suppleeenntary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report the performance with their standard errors.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources required for the experiments are provided in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Research in this paper conform with the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
The paper poses no such societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original papers that produced any code data used in our paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

A Tree-guided Diffusion Planner
Step I. Parent Branching In the first phase of bi-level sampling, -N parent trajectories generated via Algorithm
N denotes the number of trajectories sampled in batch.
Step 2. Sub- Tree Expansion In the second phase of bi-level sampling, N child trajectories are generated via Algorithm 3 Each child trajectory is generated from a parent trajectory, and the branching site is randomly chosen among intermediate states in the parent trajectory.
Step 3. Leaf Evaluation Now construct trajectory tree consisting of parent and child trajectories. The root node the tree corresponds to the initial state, and the tree has 2.V leaf nodes. Each leaf node represents complete trajectory and is associated with guide score of the path from the root. The path to the leaf with the highest score is selected as the final solution trajectory.
Step 4. Action Execution TDP supports both open-loop and closed-loop planning. In open-loop planning, the agent executes the entire solution trajectory as planned. In closed-loop planning, the agent executes only the first action of the planned trajectory, then replans by repeating Step 1-3 at each timestep.
B Algorithms
Algorithm State Decomposition (SD)
1: Input: J, trajectory т
2: == number of features state vector
3: (2,,8 .w)
4: leontoll f- 0. Johs +
5:
for = to W do
6: if 0.7 0 then
Ds:
7:
Append 8; to coontol
8: else
9: Append 8; to lobs
10: end if
11:
end for
return [control, lotss)
Algorithm 2 Parent Branching
Algorithm 3 Sub-Tree Expansion
1: Input: μtg Di, N, J. scales (Ctp, 0g), parti- cle guidance kernel Φ(-). condition C'
1: Input: μa. Σi, N, J agg, C, parent trajecto-
ries T parent: number of samples B, planning horizon T. fast diffusion steps Ny < N
2: Initialize plan N (0,
3: for = N to do
2: for j = - to B do
4: n +- ( (prmm)
3: // random branch site
5: // state decomposition
4: bi Uniform((()
6: W Wounroor M SD(J,
5: Extract first by states from Tparent.j
7:
particle guidance
6:
C+ {s
8:
Poonmll + Poomtol + a, (1
7:
// perturb parent traj.
9:
1 gradient guidance
Nr
10:
Pohs +- M + +ag E
8:
T Tchid, + (N C)
11:
Mpurent + 1-1 Motss
1-1
1-11
10:
9:
for i= to do
// fast planning
12:
T 2-1
parent (H )
11:
1d + me((Tihaaa))
Constrain -/-1 parent with C
12:
7-11 ~N(n ag a childy
14:
end for
13: Constrain Tchldd with C
ri-1
15:
return parennt
14: end for
15: end for
16: return 7 7
21

---

C Proof of Proposition 1
Given guide ((X) Jx()) (X), orthogonal gradient V VI](X) the subspace spanned by A is calculated as follows:
VLJ(X) o (1- AA (I-
J/ w1)Ja(X)
(6)
02
We examine the orthogonal reverse process applying the orthogonal gradient guidance:
dx = 12 h(T -t) X dt + V f( + (T AA )aW,
where the conditional distribution X, Xo in forward process Gaussian, i.e., N (((()) h(t)T) where h(t) 1 (t) = - exp(-v [8]. First we consider the case when Xo ~ (0, 1). The orthogonal gradient can be approximated
(X) R - (X w (x(( by assumption. As T2(X) 2 Ex, (xx,)) =: bo > and taking expectation, it is derived by linear ODE dE[x,,1]
[7>0)) // E[X,,1]+ + 2 w_ dt. Solving this ODE, we get lower bound the expectation of displacement of the final state XT.1 2 (xp(-((T,)) 0))
hg ?3
Jo exp (0 0))
|w where
( s) S (7(w) 3g du. Since the coefficient of direction vector wi is a tractable positive constant, the final state is always off-subspace. The final state converges linearly as the given guide function satisfies the PL condition and the Lipschitz property [50]. However, the global optimal solution lies in the subspace spanned by A, guided sample converges to local optimal solution which is orthogonal to the subspace.
Second, we consider the case when Xo IN, (XT,, >T)) As the small perturbation is added to the unconditional sample in the subspace, the orthogomal gradient can be approximated as V (X) &
- - AA (X A = - X (X)). From Eq. we get linear ODE dE [X =
[7000- -300) [[ dt. As [[[ O, we have [[ = 0, which means the final state is on-subspace. Similar to the previous case, the final state converges lincarly to the global optimal solution lying on the subspace as the given guide function satisfies the PL condition and Lipschitz property [50]
(exp(=x²)
jexpi (exp(=x))(exp((-14 4)2)
Local Maximum
Global Maxinum
0.4
0.3
0.2
0.1
0.0
10
Figure 7: Example of Local& optimum existing reward problem. When gradient guidance initialized at -2, it converges to a local maximum at 0. This example highlights the importance using multiple initial points when applying gradient-base guidance, which can improve the probability of reaching the global optimum under test-time guide function.
22

---

D Hyperparameters
In Maze2D gold-picking tasks, we use the same number of diffusion steps for both parent trajectory generation and sub-tree expansion, any state within a valid maze cell can serve as feasible trajectory starting point. For fair comparison, we evaluate MCSS and TAT with twice the number of samples (256) to account for their single-stage sampling design. All experiments were conducted using a single NVIDIA GeForce RTX 3090 GPU.
Table 4: Hyperparameters of three tasks.
Task
Name
Value
maze2d-medium planning horizon gred mazc2d-mcdium maximum steps Tmx
256
600
maze2d-large planning horizon T geed
384
maze2d-large large maximum steps T Tmxx
800
Maze2D Gold-picking
Threshold distance
gradient guidance strength og
62.5
particle guidance strength Op diffusion steps N 2
0.1
256
Number samples
128
planning horizon Timei maximum steps Timas
256
800
guide function parameters C1, C2. C3.
1.0. 1.5, 2.0
peak width ratio (m]smi Stecal || ||smst Sglobal |])
3:1
Threshold distance (PNP (stack), PNP (place))
0.2
Kuka Robot Arm Manipulation
Threshold distance (PNWP)
0.4
gradient guidance strength Og particle guidance strength op (PNP (stack))
100
0.25
particle guidance strength Og (PNP (place))
0.50
diffusion steps N fast diffusion steps N₁
1000
100
planning horizon T Frod maximum steps Tima
64
2000
Threshold distance gradient guidance strength Og particle guidance strength Op first goal configuration (hin ogy
0.1
0.1
0.1
AntMaze Multi-goal Exploration
(4.0, 0.05)
second goal configuration (ltgg dap)
(2.0, 0.15)
third goal configuration (hag: T
(0.5, 0.2)
fourth goal configuration (hour 0g.)) diffusion steps N fast diffusion steps N₁
(0.25, 0.25)
20
23

---

E
Maze2D Gold-picking Test Map
Figure 8: Maze2d-Medium Single-Tast Task Test Map.
Figure 9: Maze2d-Medium Multi-Task Test Map.
Figure 10: Maze2d-Large Single- Task Test Map.
Figure 11: Maze2d-Large Multi-Task Test Map.
Evaluation Metrics
We explain the evaluation metrics used in Table and Table 3
KUKA Robot Arm Manipulation. PNP (stack), the agent gets point whenever it stacks block on the existing block. The maximum points the agent can get is 3 points since there are blocks to stack (one is fixed). PXP (place), the agent gets point whenever moves a block to desired space. The maximum points the agent can get is 4 points since there are 4 blocks to move. In PNWP, if the agent moves a block region of global optimum, it will get point, while can get only half point if it moves block region of local optimum. Therefore, the maximum score that the agent can achieve in this task 4 points, as guided to move 4 blocks in total.
AntMaze Multi-goal Exploration. The number of found goals denotes the number of goals that the agent has found until the task ended. The number timesteps per goal indicates the timesteps consumed to reach each goal. Sequence match evaluates how closely an agent follows predefined

---

sequence of visits. For example, if the defined sequence is + - + 4, the agent earns points based on the number of ordered pairs (i.e., {(1, 2). (1, 3), (1, (2, (2, (2, (3, 4)}) included in its
path. The maximum score an agent can obtain is points (:' (() 6). If the agent's visiting order is 2 + 3 + 4 + 1, then the ordered pairs (2,3), (3,4), and (2,4) match the order in the defined sequence, resulting in a score points.
G Time Budget Analysis
We analyze the extra time cost of our method 1) to calculate pairwise distance of trajectory samples for particle guidance and 2) to expand sub-tree trajectories by fast-denoising, shown in Table 5
Table 5: Planning Time Comparison in KUKA Robot Arm Manipulation. The results averaged over 100 planning seeds per trajectory generation. All test results are measured on single NVIDIA GeForce RTX 3090 GPU core. The unit is second (s).
Task
Diffuser
Diffuser
MCSS
TDP (wlo child)
TDP (w/o PG)
TDP
PNP (stack-place)
16.28
16.73
16.39
17.25
18.02 17.87
19.08 18.31
PNWP
16.66
17.53
16.89
17.59
H
Hyperparameter Selection
TDP relies on three key hyperparameters: gradient guidance scale (tg, particle guidance scale ap, and fast denoising steps Ny. We provide an empirical analysis of these hyperparameters the KUKA tasks (PNP (place) and PNWP), demonstrating the robustness of TDP across broad range of values for each hyperparameter. Each result used 18 samples per planning. Bolded values indicate the defaults used in the main experiments, and the rest of the hyperparameters are fixed to their default values denoted in Appendix D
Table 6: Effect of N, on TDP Performance. dcnotes standard error.
50
100
200
400
PNP (place)
37.1 au8
+ 0.99
38.1 + 0.8
38.0 0.07
PNWP
65.7 + 0.14
67.1 + 0.10
68.3 + 0.09
68.8 0.10
Table 7: Effeet a, on TDP Performance. = denotes standard error.
Og
0.1
0.25
0.5
1.0
PNP (place)
37.4
36.6
37.8
35.5
PNWP
66.5
66.1
67.1 + 1010
64.8
Table 8: Effect of og on TDP Performance. denotes standard error.
Qg
25
50
100
200
PNP (place)
38.1 1 + ou8
38.6 # oar
37.8 + 0.09
36.0
0.12
PNWP
67.8 au9
68.2 0.11
67.1 + 0.10
64.0
Based the ablation study, we provide practical guidelines for selecting each hyperparameter for unseen test tasks.
Ny: This can be selected based on the environment before planning time. A common heuristic is set Ny between 10% and 20% of the original diffusion steps used by the pretrained diffusion planner.
On: In practice, On, typically lies within the range [0.1, 0.5). This range is compatible across stan- dard environments, where state spaces are normalized (e.g. to [-1, IJ). Increasing op encourages greater diversity in parent trajectory sampling.
Og: This should be empirically tuned based on the characteristics of the test-time task and guide function.
25

---

I Ablation Study: Number of Samples
Our framework aggregates multiple trajectory samples during test-time planning, where the number of samples significantly impacts performance. While the main paper reports averaged results across different sampling budgets, here we provide detailed ablation to show how performance varies with the number of samples.
I.1
Kuka Robot Arm Manipulation
Table Result of PNP (stack). denotes standard error.
samples
Diffuser
MCSS
TDP (wlo child)
TDP (wlo PG)
TDP
0.10
61.3+ 0.30
62. 02.0+ o.8
56.0# 0.10
62.3 0.09
12
61.0+
58.7 0.09
57.0+ o.s
59.7 0.10
56.0=
18
58.71:000 0.09
58.0: 0.10
62.01
59.71
66.0
24
61.3
61.7:
59.0.t 0.10
62.3 0.10
60.3
Table
Result PNP (place). + denotes standard error.
# samples
Diffuser
MCSS
TDP (wlo
TDP PG)
TDP
6
208
30.2 0.08
31.8 one
34.0= 0.08
35.2
12
21.0= om
31.2+ 0.08
31.0#
37.2 0.08
35.5
18
21.2+ 006
32.2+ 0.07
33.24 0.r
37.5 0.07
37.8
24
22.8+ 0.00
31.8 0.08
32.8+ 0.07
39.0# 0.09
39.2+ 0.08
Table 11: Result of PNWP. + denotes standard crror.
# samples
Diffuser
MCSS
TDP (w/e
TDP (wle PG)
TDP
6
a0s
34.8 nas
34.4+ 0.s
64.5 0.11
65. 15.1 0.11
12
33.6+
35.2+
36.0#
65.6
65.0=
18
36.0:
35.5 0.05
34.9. 0.05
67.61 0.10
67.1
24
35.8 008
37.21 0.04
36.9. 0.04
68.8 0.10
70.0 0.10
1.2
AntMaze Multi-goal Exploration
Table 12: Number of Found Goals denotes standard error.
samples
Diffuser"
MCSS
TDP (wlo
TDP PG)
TDP
32
11.5
53.5 17
53.8 1
59.8 37
57.5
64
13.5 13
56.5 441
73.8
61.0 38
73.8
128
13.5 19
66.0 133
67.3 17
67.3 =
63.5 336
256
11.3 1.6
69.3 + 39
70.8 4.6
70.3 + 39
69.5 3.8
Table 13: Sequence Match Score. + denotes standard error.
samples
Diffuser
MCSS
TDP (wlo child)
TDP (wlo PG)
TDP
32
0.8 0.22
25.3 1.47
24.3a 1.59
27.7 1.50
27.
64
1.2 0.99
27.8 1.68
35.0.: 1.81
30.5 1.70
36.5 1.45
128
2.0 047
33.7 1.71
38.5a 1.74
34.8 1.62
33.8 1.87
256
0.8 0.22
34.0# 1.52
37.34 188
37.8 1.78
37.5
Table 14: Number of timesteps per goal.
denotes standard error.
# samples
Diffuser
MCSS
TDP culs child)
TDP (wie PG)
TDP
4347.5 8
801.1
773.4
661.6
708.4
64
3703.7
667.5
598.7
626.7
462.0
128
3703.7
549.0
545.2
564.3
594.1
256
4444.5
481.1
467.4
478.7
504.0
26

---

TDP performance on the standard maze benchmark
TDP surpasses sequential approaches (i.e., MCTD Diffusion-Forring 10) on the standard maze offline benchmarks [5
pointmaze-(meddiun, large, giant) -navigate--- 100 + 0
antmaze-(meddum,, -avigat-- 100 =
autmaze-giantnnavigateeoo 98 6
TDP has advantages over sequential approaches even in single-task scenarios, in terms of trajectory optimality. For instance, in standard maze benchmarks, the shortest-path trajectory between initial and goal states represents the globally optimal solution, while longer trajectories correspond to suboptimal (local optimal) solutions 44. As maze environments scale up, identifying globally optimal solutions requires evaluating combinatorially diverse trajectory candidates with varying structures.
Single-step exploration methods like MCTD optimize actions one step at a time (via guided vs. non-guided action selection), which tends to lead to locally greedy decisions (52) and results in longer trajectories than the shortest path. In contrast, TDP employs a multi-step exploration framework via bi-level search, enabling per-step action selection based on multi-step future rewards evaluated across diverse-branched trajectory candidates. This approach allows TDP to be able to assess long-term action consequences and avoid the local optimal path.
K
Learned PG Fixed PG
Learned potentials [46] offer slight improvements in both sample diversity and overall performance. The results below compare TDP using learned versus fixed potentials on the KUKA benchmarks.
pairwise trajectory distance: Learned PG (17.54 + a00) vs. Fixed PG (17.38 + 0.05)
performance:
- -NN (stack): Learned PG (61.28 + 0.2 vs. Fixed PG (61.17 0.24)
PNP (place): Learned PG (37.04 201)) vs. Fixed PG (36.94 1013) - PNWPP: Learned PG (67.23 a.s) vs. Fixed PG (66.81 2
However, training a task-specific particle guidance potential model requires expert demonstrations, making it impractical for unseen tasks.
27


Reviews:
[{"review_id": "PWkUorgg8f", "reviewer": "Reviewer_k36W", "summary": "In this work, the authors propose a Tree-guided Diffusion Planner (TDP) that addresses challenges posed by non-convex objectives, non-differentiable constraints, and multi-reward scenarios. The method first decomposes the state into observation and control components. It then combines particle guidance with gradient-based guidance and employs a bi-level sampling scheme, where multiple child trajectories are sampled at each timestep. This design increases the likelihood of reaching global optima, avoiding the local traps that commonly hinder traditional gradient-based methods. Experimental results demonstrate that TDP outperforms state-of-the-art diffusion planning approaches.", "questions": "1. The examples of non-convex objectives or non-differentiable constraints in the supplementary material appear solvable via subgoals or intermediate rewards using hierarchical diffusion. Does TDP offer specific advantages over such hierarchical approaches? Or are there cases where hierarchical methods fail but TDP works?\n\n2. The paper does not provide sufficient detail on how particle guidance is computed. How is the particle guidance calculated? Is it applied only to the control states, or to the original state?\n\n3. How are observation and control states separated? Are they identified through automated method (e.g., gradient-based analysis) or specified manually?", "limitations": "The limitations are addressed in the conclusion section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The authors propose a novel tree-guided method that increases the probability of reaching the global optimum compared to traditional gradient-based approaches.\n\n2. Theoretical analysis is provided to support the main proposition, offering insight into the method’s effectiveness.\n\n3. The experimental results are comprehensive, covering multiple tasks and clearly demonstrating the method’s effectiveness and reliability.\n\n4. Detailed implementation information is provided in the appendix, which enhances reproducibility.\n\nWeaknesses:\n\n1. I strongly recommend the authors to split the method section into separate subsections to improve clarity and better highlight the contributions of each component.\n\n2. The sub-tree expansion procedure seems very similar to conventional MCSS method,  which may reduce the novelty of the proposed approach.", "was_revised": true, "final_justification": "I thank the authors for the detailed responses during the discussion phase and for outlining clear plans to improve the paper. I remain positive about its acceptance. However, since I cannot evaluate the extent of the planned revisions, I am unable to increase my score and will maintain my current evaluation.", "rebuttal": "We greatly appreciate Reviewer **k36W** for the thoughtful comments. Below, we address each of the raised concerns.\n\n> I strongly recommend the authors to split the method section into separate subsections to improve clarity and better highlight the contributions of each component.\n\nAs the reviewer's suggestion, we will revise the method section (Sec. 3) into three subsections (State Decomposition, Parent Branching, Sub-Tree Expansion) to clarify the contributions of each component of TDP. Additionally, we provide a detailed summary of each component’s contribution.  \n**Sec 3.1 State Decomposition**: The *observation*-*control* state decomposition is a **task-aware** and fully **automated** procedure performed at planning time. Given a guide function for the test-time task, states are decomposed based on gradient signals: *observation* states are those receiving non-zero gradients, while control states receive none. Since this decomposition relies solely on the task definition, it requires no prior knowledge or manual specification of the state components.  \n**Sec 3.2 Parent Branching**: In this first procedure of our bi-level sampling, control states are applied to particle guidance in order to **explore diverse *control* trajectories**. For example, when we want to move a block to a target position, there are many possible *control* trajectories to succeed in this task. While the traditional gradient-based approaches meet challenges (i.e., in-distribution preference, insufficient exploration) to generate diverse trajectories, TDP enables enhanced exploration via this procedure. We refer to these trajectories as parent trajectories.  \n**Sec 3.3 Sub-Tree Expansion**: In this second procedure of our bi-level sampling, parent trajectories are partially refined via fast-denoising. Each child trajectory is generated from a parent trajectory, and the branching site is randomly chosen among intermediate states. We refer to these trajectories as child trajectories. Sub-tree expansion offers two key advantages:  \n- Enhance **dynamic feasibility** of parent trajectories: Diverse parent trajectories benefit exploration, but perturbing the *control* states may lead to dynamically infeasible plans. During sub-tree expansion, perturbed *control* states are refined by a pretrained diffusion denoising process.  \n- Efficient **Local search**: Sub-Tree expansion refines *observation* states of parent trajectories with gradient guidance signal. Parent trajectories serve as initial points to guide child trajectories. Since parent trajectories are intended to cover a broad region of search space, local search conditioned on the parent trajectories is an efficient way to find better local optima.\n\nThe contribution of each component of TDP is empirically validated across multiple environments, where TDP consistently outperforms both ablated variants (TDP (w/o child) and TDP (w/o PG)) on all test tasks.\n\n> The sub-tree expansion procedure seems very similar to conventional MCSS method, which may reduce the novelty of the proposed approach.\n\nWe would like to clarify that TDP's sub-tree expansion procedure is distinct from conventional MCSS methods in the following two key aspects.\n- Novel **initialization scheme** based on theoretical analysis: TDP leverages diverse parent trajectories as initialization points for generating child trajectories via a diffusion process. This enables sub-tree expansion to perform local search from multiple, sparsely distributed points across the state space. In contrast, MCSS typically uses zero-centered noisy initializations, which are more likely to converge to local optima, as highlighted in Proposition 1-a.\n- **Temporal exploration** via random branching: Sub-tree expansion selects a random branching site for each trajectory. As a zero-shot planner, TDP receives no expert demonstrations and must instead infer optimal behavior through the guide function alone. By uniformly sampling the branching site (timestep) from the range $[0, T_{\\text{pred}}]$ where $T_{\\text{pred}}$ is the planning horizon, TDP can effectively discover temporally distant optimal goals. In contrast, MCSS lacks this mechanism and may fail to reach such goals—particularly those requiring progress beyond a certain timestep threshold—due to its concurrent sampling scheme [23].\n\nFurthermore, we empirically evaluate the scalability of MCSS under increased computational budgets on our tested benchmarks. Stochastic Sampling (SS) [b] incrementally enhances trajectory quality by using diffusion steps that are 4 times longer than the original. **MCSS+SS** denotes selecting the best trajectory from a batch generated via this stochastic sampling process. As shown in the results below, TDP consistently outperforms MCSS+SS across all test tasks, reaffirming the limited test-time scalability and generalization capacity of MCSS in comparison to TDP.\n\n- Maze2D gold-picking\n|**Environment**|**MCSS+SS**|**TDP**|\n|:-|-|-|\n|Maze2D-*Medium*|17.4±3.2|**39.8** ± 4.2|\n|Maze2D-*Large*|21.2±3.5|**47.6** ± 4.1|\n|Multi2D-*Medium*|29.2±3.6|**74.7** ± 3.0|\n|Multi2D-*Large*|58.0±3.8|**70.0** ± 3.5|\n- Pick-and-Place (PnP) and Pick-and-Where-to-Place (PnWP)\n|**Task**|**MCSS+SS**|**TDP**|\n|:-|-|-|\n|PnP (*stack*)|56.8 ± 0.16|**61.17** ± 0.24|\n|PnP (*place*)|35.5±0.19|**36.94** ± 0.13|\n|PnWP|36.24 ± 0.09|**66.81**± 0.17|\n- AntMaze Multi-goal Exploration\n|**Metric**|**MCSS+SS**|**TDP**|\n|:-|-|-|\n|# found goals &uparrow;|62.8±1.9|**66.1** ± 1.9|\n|sequence match &uparrow;|31.2±1.3|**33.8** ± 1.4|\n|# timesteps per goal &downarrow;|604.4|**558.4**|\n\n> ### Response to Q1 (TDP vs hierarchical methods)\n\nOur evaluated benchmarks are not solvable using standard hierarchical approaches. Below, we clarify the fundamental differences between TDP and hierarchical diffusion planners from both the modeling and benchmark perspectives.  \n**1. Model-aspect Distinction**  \nWhile hierarchical diffusion planners boost **supervised** planning capability with improved compositional behaviors through the use of subgoals and intermediate rewards [a, 31, 4], TDP boosts **zero-shot** planning capability on top of pre-trained diffusion planners. Hierarchical methods excel in generalizing over **labeled** data distributions seen during training. In contrast, TDP is designed to solve **unseen** planning tasks where a guide function is provided only at test time. As a bi-level sampling framework, TDP explores the search space in a compositional manner, enabling efficient discovery of high-reward trajectories even in previously unseen task configurations.\n\n**2. Benchmark-aspect Distinction**  \nThe benchmark suites we evaluate consist entirely of **test-time labeled** tasks that were not seen during training, posing a challenge for hierarchical planners that rely on training-time supervision. For example, while prior work [4] demonstrates that hierarchical diffusion planners can effectively generate *shortest-path* trajectories when both initial and goal states are specified, our gold-picking benchmark presents a fundamentally different challenge: the goal is *hidden* and often not aligned with the shortest path.\n\nFurthermore, the gold-picking task does not provide explicit information about intermediate goals (e.g., the location of the gold), nor is it guaranteed that such a goal even exists. Similarly, the PnWP task offers no precise positional information for either of the two optimal targets (i.e. local and global). As a result, the planner must infer high-reward trajectories purely based on the guide function. While some example trajectories in the supplementary material may appear compositional, these emerge from TDP’s bi-level search rather than explicit supervision. Hierarchical approaches are unlikely to succeed on these benchmarks without incorporating task-specific heuristics or annotations.  \n\n> ### Response to Q2 (PG implementation details)\n\nThe particle guidance term is calculated **only** with the ***control* states**. From line 173 to line 189 in the paper, we describe how particle guidance is computed. Specifically, this term is equal to the gradient of a radial basis function (RBF), denoted as $\\nabla\\Phi$, which can be directly computed with all pairwise distances among *control* trajectories within a batch. For clarity, this process is also illustrated in line 6 of Algorithm 2.\n\n> ### Response to Q3 (Observation and control decomposition)\n\nThe observation and control states are **autonomously decomposed** at test time through binary gradient checking with respect to the inferred task guide function. This process requires no manual intervention or prior domain knowledge. We have previously addressed this in our initial response.\n\n[a] Jinning Li et al. Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning. 2022.  \n[b] Yanwei Wang et al. Inference-Time Policy Steering through Human Interactions. 2025."}, {"review_id": "m6HH9nRMAr", "reviewer": "Reviewer_gnZK", "summary": "The Tree-guided Diffusion Planner (TDP) introduces a training-free, bi-level trajectory-level tree search that first diversifies “parent” trajectories with particle guidance to explore broadly, and then refines each branch locally with fast gradient-guided denoising, balancing exploration and exploitation. Across Maze2D gold-picking, KUKA robot arm manipulation, and AntMaze multi-goal exploration tasks, TDP consistently surpasses diffusion-based planners.", "questions": "1. Definition of trajectory quality. Could you specify whether “quality” refers solely to guide reward J, dynamic feasibility, or another composite metric?\n\n2. Learned vs fixed PG. The paper justifies fixed-potential PG for efficiency. Would a learned potential (as in [6]), paired with Sub-Tree Expansion, further improve convergence?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- **Theoretical analysis.** The paper proves (Proposition 1) that, under a two-Gaussian guide function and appropriate initialization, the bi-level sampling in TDP converges to the global optimum, whereas naïve gradient guidance only reaches a local one. Although the assumptions are stylised, giving any non-trivial optimality proof for diffusion-based planners is novel and valuable.\n\n- **Training-free, test-time computation method.** TDP strengthens a pre-trained diffuser purely at inference time by (i) particle-guided exploration and (ii) gradient-guided local refinement, then picks the highest-scoring leaf trajectory. This compute-at-test-time paradigm is timely, mirroring the machine learning community’s shift toward fast task adaptation without retraining.\n\n- **Empirical gains on diverse guidance shapes.** Across non-convex (Maze2D gold-picking), mixed dense-sparse reward setting (KUKA PNWP), and multi-goal (AntMaze) tasks, TDP surpasses both single-level samplers (Diffuser, MCSS) and tree-only baselines (TAT) by 10–25 %.\n\n### Weaknesses\n- **Narrow experimental coverage.** The evaluation is confined to three custom tasks. Because TDP is presented as a planner for long-horizon problems, it should also be benchmarked on the standard D4RL Maze2D/AntMaze suites and on OGBench [1], which offers a wide variety of goal-conditioned RL environments for competition with more varied and relatively new diffusion planner baselines.\n\n- **Heuristic state decomposition.** The observation-control state split criteria should be handcrafted for each domain.  This manual partitioning requires substantial domain knowledge and may not scale to high-dimensional sensor spaces. \n\n- **Ad-hoc dual guidance scheme.** Applying particle guidance to control states and gradient guidance to observation states departs from the principled formulation of conditional generation of classifier(-free) guidance, which is intended to model a single joint conditional distribution. This dichotomy feels heuristic, weakening the theoretical grounding of the method.\n\n- **Insufficient methodological clarity.** Although Algorithm 1 appears in Appendix A, the main text does not provide a step-by-step description of how Parent Branching, Sub-Tree Expansion, leaf evaluation, and action execution interact. Readers must infer whether the planner operates in an open-loop fashion or replans after each action is executed.\n\n- **Task-specific reward engineering.** TDP introduces a dense, hand-crafted reward 𝐽 for each environment. Because performance may be tightly coupled to how 𝐽 is shaped, TDP depends on domain-specific reward design and cannot be considered fully task-agnostic. Tasks in which such a distance is ill-defined or uninformative—e.g., OGBench’s AntSoccer or other high-dimensional, non-Euclidean goal spaces—would require substantial redesign, limiting the method’s generality and task-agnostic appeal.\n\n[1] Park, Seohong, et al. \"Ogbench: Benchmarking offline goal-conditioned rl.\" arXiv preprint arXiv:2410.20092 (2024).", "was_revised": true, "final_justification": "Through the authors' thorough rebuttal and discussion, all of my concerns have been fully addressed. I hope that the strengthened experimental results and clearer explanations will be reflected in the camera-ready version, and I am raising my score to a 4.", "rebuttal": "We sincerely appreciate Reviewer  **gnZK** for the detailed and constructive feedback. Below, we address each concern you raised and clarify the perceived weaknesses.\n\n> **Narrow experimental coverage** ...\n\nWe provide more experiment results compared with the recent diffusion planner baselines: Diffusion-Forcing (2024) [a], MCTD (2025) [b], AdaptDiffuser (2023) [32], Stochastic Sampling (2025) [c]. We would like to briefly summarize three takeaways from our additional experiments first.\n- TDP surpasses sequential approaches (MCTD, Diffusion-Forcing) on the standard maze offline benchmarks.\n- TDP surpasses AdaptDiffuser on both the standard benchmark (PnP) and the custom task (PnWP). It supports TDP’s test-time scalability and generalization capability.\n- TDP surpasses MCSS+SS on all test tasks. It shows that increasing the test-time computation budget does not always guarantee better performance.\n\n**1. TDP performance on the standard maze benchmark**  \n   - pointmaze-{*medium*,*large*,*giant*}-navigate-v0: 100±0\n   - antmaze-{*medium*,*large*}-navigate-v0: 100±0\n   - antmaze-*giant*-navigate-v0: 98±6\n\nTDP consistently outperforms both MCTD and Diffusion-Forcing across all standard maze benchmarks (see Table 1 in [b]).\n\n**2. Additional baselines on our benchmarks**\n\nWe added two baselines: AdaptDiffuser for the Kuka tasks and MCSS+SS for all test tasks. AdaptDiffuser is a fine-tuned diffusion planner on synthetic expert demonstrations [32], and MCSS+SS refers to selecting the best trajectory from a batch generated via stochastic sampling [c]. MCSS+SS requires 4 times more computation compared to MCSS.\n- Gold-picking\n|**Environment**|**MCSS+SS**|**TDP**|\n|:-|-|-|\n|Maze2D-*Medium*|17.4±3.2|**39.8** ± 4.2|\n|Maze2D-*Large*|21.2±3.5|**47.6** ± 4.1|\n|Multi2D-*Medium*|29.2±3.6|**74.7** ± 3.0|\n|Multi2D-*Large*|58.0±3.8|**70.0** ± 3.5|\n- Pick-and-Place (PnP) and Pick-and-Where-to-Place (PnWP)\n|**Task**|**AdaptDiffuser**|**MCSS+SS**|**TDP**|\n|:-|-|-|-|\n|PnP (*stack*)|60.54 ± 0.18|56.8 ± 0.16|**61.17** ± 0.24|\n|PnP (*place*)|36.17 ± 0.11|35.5±0.19|**36.94** ± 0.13|\n|PnWP|39.72 ± 0.08|36.24 ± 0.09|**66.81**± 0.17|\n- AntMaze multi-goal exploration\n|**Metric**|**MCSS+SS**|**TDP**|\n|:-|-|-|\n|# found goals &uparrow;|62.8±1.9|**66.1** ± 1.9|\n|sequence match &uparrow;|31.2±1.3|**33.8** ± 1.4|\n|# timesteps per goal &downarrow;|604.4|**558.4**|\n\nOur results consistently show TDP's superior performance across all compared benchmarks.\n\n**3. Justification for the custom benchmarks**\n\nRecent diffusion planner approaches focus on **supervised** planning, including sequential methods [a, b], hierarchical methods [31, 4], and fine-tuning approaches [32] for standard offline RL benchmarks.\n\nTDP enhances **zero-shot** planning capability on unseen tasks *shifted* from the trained distribution. Given a test-time defined task, TDP operates as a task-aware planning framework on top of the pretrained diffuser. For example, in the AntMaze environment, the pretrained planners themselves hardly generalize to test-time multi-goal reward tasks since they are trained on the standard offline benchmarks which are single-goal tasks. The gold-picking task is also not solvable with a pretrained diffuser alone, as training trajectories lack intermediate goal conditioning. These tasks can be solved by directly applying TDP in a plug-and-play manner.\n\nOur benchmarks evaluate the zero-shot planning capability where planner models must infer high-reward trajectories purely from the guide function without expert demonstrations. Existing supervised planners challenge to solve our benchmarks. For instance, sequential approaches (e.g., MCTD) cannot outperform TDP on the PnWP task since MCTD explores *single-step* with guide/no-guide action, while TDP explores *multi-step* ahead with diverse bi-level trajectory sampling. Single-step exploration suits standard offline benchmarks with a single optimal goal, but often leads to local optima in multi-goal settings.\n\n> **Heuristic state decomposition** ...\n\nWe would like to clarify that the observation-control state decomposition is neither heuristic nor manually specified. It is a **fully automated**, task-aware procedure performed at planning time. Given a guide function for the test-time task, states are decomposed based on gradient signals: observation states are those receiving non-zero gradients, while control states receive none. This gradient-based criterion enables scalable and domain-agnostic decomposition. We apologize for not making this process sufficiently clear in the manuscript. We will revise the corresponding method section to address this oversight.\n\n> **Ad-hoc dual guidance scheme** ...\n\nWe would like to clarify that TDP does not implement a dual guidance scheme. Instead, TDP formulates a single joint conditional distribution with an integrated guidance term. As shown in Equation (2) of the main paper, the overall guidance distribution is defined as $h=h\\_{gg}\\cdot h\\_{pg}$ where $h\\_{gg}(\\cdot)$ denotes the gradient guidance component, and $h\\_{pg}(\\cdot)$ denotes the particle guidance component. Since both components jointly condition the same pretrained reverse denoising process, the perturbed reverse denoising process is approximated as $\\tilde{p}(\\boldsymbol{\\tau}\\_{i-1}|\\boldsymbol{\\tau}\\_i)\\approx \\mathcal{N}(\\boldsymbol{\\tau}\\_{i-1}; \\mu+\\Sigma g\\_{tdp}, \\Sigma)$ where $g\\_{tdp}=\\nabla_\\tau \\log\\left(h_{gg}(\\boldsymbol{\\tau}_i)\\cdot h\\_{pg}(\\boldsymbol{\\tau}\\_i)\\right)=g\\_{gg}+g\\_{pg}$. Intuitively, Equation (3) can be integrated to $\\boldsymbol{\\mu}^{i-1} \\leftarrow \\boldsymbol{\\mu}^{i-1} + \\Sigma^i g\\_{tdp}$ where $g\\_{tdp}=g\\_{gg}+g\\_{pg}= \\alpha\\_p\\nabla \\Phi(\\boldsymbol{\\mu}^{i-1}\\_{\\text{control}})+\\alpha\\_g\\nabla \\mathcal{J}(\\boldsymbol{\\mu}^{i-1}\\_{\\text{obs}})$. We have a single integrated guidance term, which is the sum of gradient guidance part ($g\\_{gg}$) and particle guidance part ($g\\_{pg}$).\n\n> **Insufficient methodological clarity** ...\n\nWe appreciate this feedback and recognize that the overall pipeline could be clearer. Below, we provide a step-by-step description of how TDP operates. While individual components are described in lines 162-200, readers may benefit from seeing how these components work together.\n\n**Step 1. Parent Branching**: In the first phase of TDP, $N$ parent trajectories are generated via Equation (3). $N$ denotes the number of trajectories sampled in a batch.    \n**Step 2. Sub-Tree Expansion**: In the second phase of TDP, $N$ child trajectories are generated via Equation (4). Each child trajectory is generated from a parent trajectory, and the branching site is randomly chosen among intermediate states in the parent trajectory.  \n**Step 3. Leaf Evaluation**: Now we construct a trajectory tree consisting of parent and child trajectories. The root node of the tree corresponds to the initial state, and the tree has $2N$ leaf nodes. Each leaf node represents a complete trajectory and is associated with a guide score of the path from the root. The path to the leaf with the highest score is selected as the final solution trajectory.  \n**Step 4. Action Execution**: TDP supports both open-loop and closed-loop planning. In open-loop planning, the agent executes the entire solution trajectory as planned. In closed-loop planning, the agent executes only the first action of the planned trajectory, then replans by repeating Steps 1-3 at each timestep.  \n\n> **Task-specific reward engineering** ...\n\nWe clarify that the notion of **“tasks”** mentioned by the reviewer differs from our problem setting. The tasks referenced by the reviewer correspond to the training tasks that a pretrained diffusion planner is expected to handle. While pretrained diffusion planners model underlying system dynamics, TDP samples high-reward solution trajectories (i.e., those with high guide scores for the test task) conditioned on the learned dynamics in a zero-shot manner. TDP equips the pretrained diffuser with the ability to reason over higher-level objectives, such as our benchmarks.\n\nAlthough the reviewer is correct that reward shape affects the performance of TDP, this is inherent to most zero-shot planning methods that rely on reward-guided adaptation at test time. Similar to prior works on zero-shot task transfer [d, e], TDP assumes access to a pretrained model (i.e., Diffuser) and dense reward signals to effectively adapt to unseen tasks.\n\nWhile TDP requires task-specific guide functions, it provides a unified framework that works across diverse domains without requiring task-specific model retraining or expert demonstrations.\n\n> ### Response to Q1 (Definition of trajectory quality)\n\nIn our framework, trajectory quality is defined as a composite metric: the product of the guide reward J and a binary dynamic feasibility indicator. That is, a trajectory must not only align well with the test-time guide function but also be dynamically feasible (i.e., executable without violating environmental constraints).\n\n> ### Response to Q2 (Learned vs fixed PG)\n\nLearned potentials provide marginal improvements in both sample diversity and performance. The results below show TDP with learned vs fixed potentials evaluated on the Kuka benchmarks.\n- **Learned PG vs. Fixed PG**\n  - pairwise trajectory distance: Learned PG (17.54±0.04) vs. Fixed PG (17.38±0.05)\n  - PnP (*stack*) / PnP (*place*) / PnWP: Learned PG (61.28±0.22 / 37.04±0.12 / 67.23±0.15) vs. Fixed PG (61.17±0.24 / 36.94±0.13 / 66.81±0.17)\n\n[a] Boyuan Chen et al. Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion. 2024.  \n[b] Jaesik Yoon et al. Monte Carlo Tree Diffusion for System 2 Planning. 2025.  \n[c] Yanwei Wang et al. Inference-Time Policy Steering through Human Interactions. 2025.  \n[d] Wenlong Huang et al. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. 2022.  \n[e] Junhyuk Oh et al. Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning. 2017."}, {"review_id": "PlHuhvEHXL", "reviewer": "Reviewer_GPcL", "summary": "The authors propose TDP, a test-time planning framework of diffusion planner for offline RL tasks. TDP consists of two stages: Parent Branching and Sub-tree Expansion. This bi-level framework effectively balance exploration-exploitation trade-off.", "questions": "Here are some questions I want to ask:\n\n- It seems that the computational cost of TDP is similar to a naive Diffuser, even though it has a test-time adaptation stage. Could authors explain how is it possible? Furthermore, I would like to ask if we have more computational budget for test-time, would the performance improve?\n- Decomposing states into observation and control states is possible only when the state is given as a compact vector. What if the state is given as a visual input?", "limitations": "Here are some comments I want to suggest:\n\n- As written in the weakness part, there are several works that consider test-time planning of diffusers for improving performance. I strongly recommend authors to conduct more comprehensive literature review and add a discussion part for these works.\n- It might be better to add some analysis on hyperparameters. It seems that we use different guidance scale for different guidance mechanisms and environments. It would be also nice to add some studies related on quality of planning and time complexity in terms of $N_f$", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The authors consider an important topic, test-time scaling of diffusion planners.\n- Extensive experiments results across diverse tasks\n- Code is available in the supplementary material\n\n**Weakness**\n\n- Test-time scaling of diffusion planners is an important topic in offline RL tasks and there are several recent works, which also utilize tree-like structure [1-3]. I strongly recommend authors to compare their method with these works or at least add a discussion.\n- While I acknowledge that authors conduct experiments on diverse benchmarks such as Kuka robot arm manipulation, it would be nice to conduct experiments on much larger tasks such as pointmaze and antmaze giant, proposed by OGBench. Experiment results on those tasks is beneficial for the authors’ claim: bi-level approach for test-time planning is crucial.\n\n[1] Yoon, Jaesik, et al. \"Monte Carlo Tree Diffusion for System 2 Planning.\" *arXiv preprint arXiv:2502.07202* (2025).\n\n[2] Zhou, Guangyao, et al. \"Diffusion model predictive control.\" *arXiv preprint arXiv:2410.05364* (2024).\n\n[3] Chen, Boyuan, et al. \"Diffusion forcing: Next-token prediction meets full-sequence diffusion.\" *Advances in Neural Information Processing Systems* 37 (2024): 24081-24125.", "was_revised": true, "final_justification": "Update the score based on rebuttal response: Discussion on relevant works / Competitive performance on additional benchmarks.", "rebuttal": "We sincerely thank Reviewer **GPcL** for the thoughtful suggestions. Below, we address each of the raised points.\n\n> ... I strongly recommend authors to compare their method with these works or at least add a discussion.\n\nWe agree it is important to clearly position TDP and articulate how TDP differs from the recent approaches.\n\n**1. TDP vs. Existing Approaches**  \nWe would like to clarify why the existing tree-based methods show limited performance on our tested benchmarks. We provide three case studies to elaborate on the differences between TDP and existing methods.\n\n**1-1. Sequential approaches** struggle with multi-goal benchmarks such as PnWP and AntMaze Multi-goal Exploration. MCTD [b] performs **single-step** exploration by selecting between guided and unguided actions at each time step. This approach works well on standard offline RL benchmarks, where the tasks present convex structures with a single optimal goal (e.g., offline maze navigation). However, per-action exploration causes MCTD to often fail to discover distant high-reward goals, converging to local optima. The PnWP benchmark captures this limitation, and the AntMaze Multi-goal Exploration requires identifying better goals to maximize the sequence match score. In contrast, TDP performs **multi-step** exploration through diverse bi-level trajectory sampling, handling the challenging multi-goal scenarios better.\n\n**1-2. Hierarchical diffusion planners** improve compositional behaviors through the use of subgoals and intermediate rewards [31, 4]. Hierarchical methods excel in generalizing over **labeled** data distributions seen during training. Our tested benchmark suites consist entirely of **unlabeled** tasks that were not seen during training, posing a challenge for hierarchical planners relying on training-time supervision. While prior work [4] demonstrates that hierarchical diffusion planners can generate *shortest-path* trajectories when both initial and goal states are specified, our gold-picking benchmark presents a fundamentally different challenge: the goal is *hidden* and often not aligned with the shortest path.\n\n**1-3. Diffusion model predictive control** (D-MPC) [d] learns multi-step proposals using dynamics models $p(s|a)$ to enable adaptation to test-time changes in system dynamics. However, it struggles to generalize to **unseen long-horizon** tasks, as modeling such behavior solely through $p(s|a)$ can be limiting. For example, hovering or complex horizontal navigation remains difficult to capture with standard dynamics models. In contrast, TDP is a Diffuser-based planning framework that models joint distribution $p(s,a)$, effectively solving long-horizon and multi-goal tasks. Furthermore, D-MPC is a **few-shot** learner, requiring fine-tuning with a small set of expert demonstrations. TDP, on the other hand, is a fully zero-shot planner, operating without any additional demonstrations at test time.\n\n**2. Supervised Planning vs. Zero-shot Planning**  \nMost of the recent diffusion planner works focus on enhancing **supervised** planning capabilities [a, b, 32, 31, 4, d]. They are categorized into sequential approaches [a, b], hierarchical approaches [31, 4], and fine-tuning approaches [32, d]. These works improve the modeling of underlying system dynamics from the static offline RL benchmarks. Recent works successfully solve challenging offline benchmarks via reformulating the training scheme, learning value estimator, and test-time scaling.  \nAnother line of recent diffusion planner research focuses on enhancing **zero-shot** planning capabilities [c]. Test-time tasks are shifted from the training distribution, and planners are given access to a pretrained model (i.e., Diffuser) along with dense reward signals to effectively adapt to these unseen tasks. **TDP** is a task-aware planning framework, exhibiting robust zero-shot planning performance in a variety of tasks.\n\nWe will add this discussion to the main paper to better position TDP relative to existing approaches.\n\n**3. Additional Experiments**  \nWe provide the additional experiment results to compare TDP with the recent works. We briefly summarize three takeaways from our additional experiments below.\n- TDP surpasses sequential approaches (MCTD [b], Diffusion-Forcing [a]) on the standard maze offline benchmarks.\n- TDP surpasses the fine-tuning approach (AdaptDiffuser [32]) on both the standard pick-and-place benchmark (PnP) and the custom task (PnWP). It supports TDP’s test-time scalability and generalization capability.\n- TDP surpasses the recent zero-shot planning approach (MCSS+SS [c]) on all test tasks.\n\n**3-1. Result on the standard maze navigation benchmark**\n\nThe results of Diffusion-Forcing and MCTD are referenced from [b].\n- PointMaze (Maze2D)\n|**Dataset**|**Diffusion-Forcing**|**MCTD**|**TDP**|\n|:-|-|-|-|\n|pointmaze-*medium*|65 ± 16|100 ± 0|**100** ± 0|\n|pointmaze-*large*|74 ± 9|98 ± 6|**100** ± 0|\n|pointmaze-*giant*|50 ± 10|100 ± 0|**100** ± 0|\n\n- AntMaze\n|**Dataset**|**Diffusion-Forcing**|**MCTD**|**TDP**|\n|:-|-|-|-|\n|antmaze-*medium*|90 ± 10|100 ± 0|**100** ± 0|\n|antmaze-*large*|57 ± 6|98 ± 6|**100** ± 0|\n|antmaze-*giant*|24 ± 12|94 ± 9|**98** ± 6|\n\n**3-2. Additional baselines on our tested benchmarks**  \nWe added two baselines: **AdaptDiffuser** for robot-arm manipulation and **MCSS+SS** for all test tasks. AdaptDiffuser is a fine-tuned diffusion planner on synthetic expert pick-and-place demonstrations, and MCSS+SS denotes Stochastic Sampling [c] with best-of-batch selection based on the guide score.\n- Maze2D gold-picking\n|**Environment**|**MCSS+SS**|**TDP**|\n|:-|-|-|\n|Maze2D-*Medium*|17.4±3.2|**39.8** ± 4.2|\n|Maze2D-*Large*|21.2±3.5|**47.6** ± 4.1|\n|Multi2D-*Medium*|29.2±3.6|**74.7** ± 3.0|\n|Multi2D-*Large*|58.0±3.8|**70.0** ± 3.5|\n- Pick-and-Place (PnP) and Pick-and-Where-to-Place (PnWP)\n|**Task**|**AdaptDiffuser**|**MCSS+SS**|**TDP**|\n|:-|-|-|-|\n|PnP (*stack*)|60.54 ± 0.18|56.8 ± 0.16|**61.17** ± 0.24|\n|PnP (*place*)|36.17 ± 0.11|35.5±0.19|**36.94** ± 0.13|\n|PnWP|39.72 ± 0.08|36.24 ± 0.09|**66.81**± 0.17|\n- AntMaze Multi-goal Exploration\n|**Metric**|**MCSS+SS**|**TDP**|\n|:-|-|-|\n|# found goals &uparrow;|62.8±1.9|**66.1** ± 1.9|\n|sequence match &uparrow;|31.2±1.3|**33.8** ± 1.4|\n|# timesteps per goal &downarrow;|604.4|**558.4**|\n\n> ... it would be nice to conduct experiments on much larger tasks such as pointmaze and antmaze giant, proposed by OGBench.\n\n In our initial response, we provide the results on the pointmaze-*giant* and antmaze-*giant* standard navigation tasks, evaluated against MCTD and Diffusion-forcing. Our tested benchmarks are similarly long-horizon as OGBench’s standard maze tasks with a giant map. In the original datasets, the trajectory horizons are 1000 steps for medium and large maps, and 2000 steps for giant maps. Our multi-goal exploration task on a large map is designed with comparable long-context, requiring up to 2000 steps to complete.\n\n> It might be better to add some analysis on hyperparameters.\n\nTDP relies on three key hyperparameters as noted by the reviewer: gradient guidance scale $\\alpha\\_g$, particle guidance scale $\\alpha\\_p$, and fast denoising steps $N\\_f$. We provide an empirical analysis of these hyperparameters on the Kuka tasks (PnP (*place*) and PnWP), demonstrating the **robustness** of TDP across a broad range of values for each hyperparameter. Each result used 18 samples per planning. Bolded values indicate the defaults used in the main experiments, and the rest of the hyperparameters are fixed to their default values.\n- $N\\_f$: 50 / **100** / 200 / 400\n    - PnP (*place*): 37.1 ± 0.08 / **37.8 ± 0.09** / 38.1 ± 0.08 / 38.0 ± 0.07\n   - PnWP: 65.7 ± 0.14 / **67.1 ± 0.10** / 68.3 ± 0.09 / 68.8 ± 0.10\n- $\\alpha\\_p$: 0.1 / 0.25 / **0.5** / 1.0\n    - PnP (*place*): 37.4 ± 0.08 / 36.6 ± 0.08 / **37.8 ± 0.09** / 35.5 ± 0.11\n   - PnWP:  66.5± 0.14 / 66.1± 0.11 / **67.1± 0.10** / 64.8 ± 0.13\n - $\\alpha\\_g$: 25 / 50 / **100** / 200\n   - PnP (*place*): 38.1 ± 0.08 / 38.6 ± 0.07 / **37.8 ± 0.09** / 36.0 ± 0.12\n   - PnWP: 67.8 ± 0.09 / 68.2 ± 0.11 / **67.1± 0.10** / 64.0 ± 0.14\n\nWe provide practical guidelines for selecting each value as below.\n- $N\\_f$: It can be selected based on the environment before planning time. A common heuristic is to set $N_f$ between 10-20% of the original diffusion steps used by the pretrained planner.\n- $\\alpha\\_p$: In practice, it typically lies within the range [0.1, 0.5]. This range is compatible across standard environments, where state spaces are normalized (e.g., to [-1, 1]). Increasing $\\alpha_p$ encourages greater diversity in parent trajectory sampling.\n- $\\alpha\\_g$: It should be empirically tuned based on the characteristics of the test-time task and guide function.\n\n> ### Response to Q1 (Computational cost and scaling)\n\nTDP introduces only marginal computational overhead compared to the standard Diffuser, due to its use of fast-denoising in the Sub-Tree Expansion phase. For instance, in the Kuka environment, while the original diffusion process uses 1000 denoising steps, TDP generates child trajectories with just 100 steps, only 10% of the original cost. Also, we provide extensive experimental results in Appendix G, where we vary the number of samples per planning. This study shows that increasing test-time computational budget does not always guarantee better performance.\n\n> ### Response to Q2 (State decomposition for visual inputs)\n\nVisual inputs can be converted into *proxy* states with compact vector representations—by training inverse dynamics and position estimator models, following the approach used for visual maze in prior work [b].\n\n[a] Boyuan Chen et al. Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion. 2024.  \n[b] Jaesik Yoon et al. Monte Carlo Tree Diffusion for System 2 Planning. 2025.  \n[c] Yanwei Wang et al. Inference-Time Policy Steering through Human Interactions. 2025.  \n[d] Guangyao Zhou et al. Diffusion Model Predictive Control. 2025."}, {"review_id": "m6HtsOZ5TB", "reviewer": "Reviewer_2mBS", "summary": "The authors tackle the limitations of standard gradient-based guidance in diffusion models. The authors identify that existing guidance methods often fail in scenarios with non-convex or non-differentiable objectives, which are common in realistic planning tasks, so the authors introduce the the Tree-guided Diffusion Planner (TDP), a novel training-free, test-time planning framework designed to overcome this limitation. TDP addresses this by structuring trajectory generation as a tree search that explicitly balances exploration and exploitation. It uses a bi-level sampling process: (1) diverse parent trajectories are generated using Particle Guidance to ensure broad exploration of the solution space, and (2) local sub-trajectories are refined from these parents using fast, gradient-guided denoising to exploit promising regions.", "questions": "1. See the weaknesses part. Is there anything I misunderstood or overlooked about the weakness I mentioned?\n2. How much tuning effort is required to find effective hyperparameters values for a new task?\n3. For tasks like Maze2D Gold-picking, a straightforward alternative would be to plan sequentially: first plan a trajectory to the intermediate goal (the gold), and then re-plan from that state to the final destination. This approach would also handle the non-differentiable constraint. Could you elaborate on the specific advantages of TDP's holistic, diffusion-based planning over such a simpler, sequential approach?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-motivated. It clearly presents a key weakness of current diffusion planners: the reliance on simple, gradient-based guidance that is ill-suited for the complex, non-convex, and non-differentiable objective functions often encountered in the real world. \n2. The paper is clearly written, with useful illustrations.\n3. The paper provides theoretical justification of bi-level sampling process.\n\nWeaknesses:\n1. Although the method is training-free, it introduces a new set of important test-time hyperparameters. While the paper commendably provides a detailed ablation study on the number of samples, it lacks a similar sensitivity analysis for other critical test-time hyperparameters introduced by the method. Specifically, the guidance strengths for particle guidance, and gradient guidance, as well as the number of fast denoising steps. Without an analysis of how robust the method is to changes in these parameters, it is unclear how much tuning effort is required for a new task. This potentially replaces a \"training\" burden with a significant \"test-time tuning\" burden for these specific parameters.\"\n2. A major weakness of this paper is the lack of comparison and discussion of highly relevant works. In particular, [1] introduces stochastic sampling, an MCMC-based method, alternative sampling-based solution to the limitations of standard gradient guidance that TDP also aims to solve. Similarly, [2] introduces Monte Carlo Tree Diffusion (MCTD) which also employs a tree-based search to guide the diffusion process at test time. It uses meta-actions (e.g., guide / no guide) to navigate the exploration-exploitation trade-ff at the level of sub-plans. Given the absence of a compartive discussion makes it difficult to assess the unique contributions of TDP.\n\n[1] Inference-Time Policy Steering through Human Interactions\n\n[2] Monte Carlo Tree Diffusion for System 2 Planning", "was_revised": true, "final_justification": "While the authors have addressed some of my concerns, particularly regarding the model's robustness to hyperparameters, several critical issues remain unresolved for me.\n\n(1) I am still not fully convinced by the explanation for why TDP performs better than methods like MCTD, especially in single-task scenarios. The rebuttal did not provide a clear and compelling justification for the source of TDP's advantage in these single task settings (see in the discussion with Reviewer GPcL).\n\n(2) I also question the paper's reproducibility, as the code for TDP has not been made available and many implementation details are missing from the paper. Therefore, I am not confident that I could fully reproduce the results on my own.\n\n--- update ---\nWith the authors’ further clarifications, most of my concerns have been well addressed, particularly regarding the contribution compared to MCTD and the reproducibility issue. Therefore, I have raised my score to 4.", "rebuttal": "We appreciate Reviewer **2mBS** for the detailed and thoughtful suggestions.\n> Is there anything I misunderstood or overlooked about the weakness I mentioned?\n\nWhile most of your observations are valid, we believe there are some aspects regarding test-time tuning burden (Weakness 1) and sequential planning approach (Question 3) that may benefit from additional clarification. We will address all the concerns you raised below.\n\n> A major weakness of this paper is the lack of comparison and discussion of highly relevant works.\n\nWe agree it is important to clearly position TDP and articulate how it differs from the relevant approaches.\n\n**1. TDP vs. Existing Approaches**\n\nWe would like to clarify why existing methods are not well-suited to solve our tested benchmarks. We provide three case studies to elaborate the difference between TDP and the existing methods.\n\n**1-1. Sequential approaches** struggle with multi-goal benchmarks such as PnWP and AntMaze Multi-goal Exploration. For example, MCTD [b] performs **single-step** exploration by selecting between guided and unguided actions at each time step. This works well on standard offline RL benchmarks with convex structures and a single optimal goal (e.g., offline maze navigation). However, per-action exploration causes MCTD to often fail discovering distant high-reward goals and instead converges to locally optimal solutions. The PnWP benchmark captures this limitation, and the AntMaze Multi-goal task requires identifying better goals to maximize sequence match scores. In contrast, TDP performs **multi-step** exploration through diverse bi-level trajectory sampling, better handling challenging multi-goal scenarios.\n\n**1-2. Hierarchical diffusion planners** improve compositional behaviors using subgoals and intermediate rewards [31, 4]. Hierarchical methods excel in generalizing over **labeled** data distributions seen during training. Our benchmark suites consist entirely of **unlabeled** tasks unseen during training, posing a challenge for hierarchical planners that rely on training-time supervision. For example, while prior work [4] demonstrates that hierarchical diffusion planners can generate *shortest-path* trajectories when both initial and goal states are specified, our gold-picking benchmark presents a fundamentally different challenge: the goal is *hidden* and often not aligned with the shortest path.\n\n**1-3. Diffusion model predictive control** (D-MPC) [d] learns multi-step proposals using dynamics models $p(s|a)$ to enable adaptation to test-time changes in system dynamics. However, it struggles to generalize to **unseen long-horizon** tasks, as modeling behavior solely through $p(s|a)$ can be limiting. Standard dynamics models struggle to capture long-context reward structures effectively. In contrast, TDP models the joint distribution $p(s,a)$, effectively solving long-horizon and multi-goal tasks. Furthermore, D-MPC is a **few-shot** learner, requiring fine-tuning with a small set of expert demonstrations. TDP is a fully zero-shot planner, operating without test-time demonstrations.\n\n**2. Supervised Planning vs. Zero-shot Planning**\n\nMost of the recent diffusion planner works focus on enhancing **supervised** planning capabilities [a, b, 32, 31, 4, d]. They are categorized into sequential approaches [a, b], hierarchical approaches [31, 4], and fine-tuning approaches [32, d]. These works improve the modeling of underlying system dynamics from the static offline RL benchmark. The recent works successfully solve challenging offline benchmarks via reformulating the training scheme, learning value estimator, and test-time scaling.  \nAnother line of recent diffusion planner research focuses on enhancing **zero-shot** planning capabilities [c]. The test-time tasks are shifted from the trained distribution, and planners are given access to a pretrained model (i.e., Diffuser) along with dense reward signals to effectively adapt to these unseen tasks. **TDP** is a task-aware planning framework, exhibiting robust zero-shot planning performance in a variety of tasks.\n\n**3. Additional baseline**\n\nWe added a baseline the reviewer has mentioned, **MCSS+SS**, for all test tasks. MCSS+SS denotes Stochastic Sampling [c] with best-of-batch selection based on the guide score.\n- **MCSS+SS vs. TDP**\n  - Maze2D-*Medium*/*Large*: MCSS+SS (17.4±3.2 / 21.2±3.5) vs. TDP (**39.8±4.2 / 47.6±4.1**)\n  - Multi2D-*Medium*/*Large*: MCSS+SS (29.2±3.6 / 58.0±3.8) vs. TDP (**74.7±3.0 / 70.0±3.5**)\n  - PnP (stack) / PnP (place) / PnWP: MCSS+SS (56.8±0.1 / 35.5±0.19 / 36.24±0.09) vs. TDP (**61.17±0.24 / 36.94±0.13 / 66.81±0.17**)\n  - AntMaze Multi-goal: # goals (MCSS+SS: 62.8 vs. **TDP: 66.1**), seq. match (31.2 vs. **33.8**), timesteps (604.4 vs. **558.4**)\n\n> Could you elaborate on the specific advantages of TDP's holistic, diffusion-based planning over such a simpler, sequential approach?\n\nWe clarify that the gold-picking task is a **black-box** problem that requires inferring the intermediate goal (i.e., gold) location using only a distance-based guide function, without access to the gold’s exact position. Contrary to the reviewer’s suggestion, it is not feasible to plan two separate trajectories sequentially, as the intermediate goal position is not explicitly known. To address this, TDP performs bi-level trajectory sampling to broadly explore the environment and collect guide signals, enabling it to discover the *hidden* gold location within the map.\n\n> Without an analysis of how robust the method is to changes in these parameters, it is unclear how much tuning effort is required for a new task.\n\nWe agree that the robustness of TDP with respect to the variety of test-time hyperparameters should be elaborated since TDP aims to solve unseen tasks in an efficient manner.  \nTDP relies on three test-time hyperparameters, as noted by the reviewer: gradient guidance scale $\\alpha_g$, particle guidance scale $\\alpha_p$, and fast denoising steps $N_f$. We provide an empirical analysis of these hyperparameters on the Kuka tasks (PnP (*place*) and PnWP), demonstrating the **robustness** of TDP across a broad range of values for each hyperparameter. Each result used 18 samples per planning. Bolded values indicate the default hyperparameter values and their corresponding results in the main experiments, and the rest of hyperparameters are fixed to their default values.\n- **$N_f$**: 50 / **100** / 200 / 400  \n   - PnP (*place*): 37.1 ± 0.08 / **37.8 ± 0.09** / 38.1 ± 0.08 / 38.0 ± 0.07  \n   - PnWP: 65.7 ± 0.14 / **67.1 ± 0.10** / 68.3 ± 0.09 / 68.8 ± 0.10  \n \n- **$\\alpha_p$**: 0.1 / 0.25 / **0.5** / 1.0  \n   - PnP (*place*): 37.4 ± 0.08 / 36.6 ± 0.08 / **37.8 ± 0.09** / 35.5 ± 0.11  \n   - PnWP: 66.5 ± 0.14 / 66.1 ± 0.11 / **67.1 ± 0.10** / 64.8 ± 0.13\n\n- **$\\alpha_g$**: 25 / 50 / **100** / 200\n   - PnP (*place*): 38.1 ± 0.08 / 38.6 ± 0.07 / **37.8 ± 0.09** / 36.0 ± 0.12\n   - PnWP: 67.8 ± 0.09 / 68.2 ± 0.11 / **67.1± 0.10** / 64.0 ± 0.14\n \nIn addition, we provide practical guidelines for selecting each value as follows.\n- **$N_f$**: It can be selected based on the environment before planning time. A common heuristic is to set $N_f$ between 10% and 20% of the original diffusion steps used by the pretrained diffusion planner.\n- **$\\alpha_p$**: In practice, $\\alpha_p$ typically lies within the range [0.1, 0.5]. This range is compatible across standard environments, where state spaces are normalized (e.g. to [-1, 1]). Increasing $\\alpha_p$ encourages greater diversity in parent trajectory sampling.\n- **$\\alpha_g$**: It should be empirically tuned based on the characteristics of the test-time task and guide function.\n\n> This potentially replaces a \"training\" burden with a significant \"test-time tuning\" burden for these specific parameters.\n\nWe would like to clarify why TDP can be an alternative way to solve test-time task rather than training-per-task approaches. TDP’s test-time scalability and generalization capability enable **zero-shot** planning that surpasses fine-tuning approaches in both **performance** and **time cost**. As a standard baseline, **AdaptDiffuser** [32] refers to the fine-tuned diffuser model on synthetic expert demonstrations. TDP outperforms AdaptDiffuser on both standard pick-and-place benchmarks (PnP) and the custom task (PnWP).\n- **Performance**\n|**Task**|**AdaptDiffuser**|**TDP**|\n|:-|-|-|\n|PnP (*stack*)|60.54 ± 0.18|**61.17** ± 0.24|\n|PnP (*place*)|36.17 ± 0.11|**36.94** ± 0.13|\n|PnWP|39.72 ± 0.08|**66.81**± 0.17|\n\n- **Time cost**  (on NVIDIA RTX 3090)\n   - **AdaptDiffuser**: 48 hours (collecting synthetic expert demonstrations) + 2 hours (fine-tuning)\n   - **TDP**: <1 hour (test-time hyperparameter tuning based on the practical guideline above)\n\nFurthermore, collecting expert demonstrations for each test-time task is often impractical, especially in tasks with complex dynamics or limited simulation control. It can be challenging to extract a sufficient number of high-quality trajectories on demand at test time. In contrast, TDP offers a plug-and-play solution that operates directly on top of pretrained diffusion planners—requiring no additional supervision or adaptation—making it a scalable and efficient alternative for diverse unseen tasks.\n\n> How much tuning effort is required to find effective hyperparameters values for a new task?\n\nThe values of $\\alpha_p$ and $N_f$ typically generalize well across tasks within the same environment. In contrast, $\\alpha_g$ is more task-specific and requires heuristic tuning. However, this tuning remains substantially more efficient—both in time and computation—than fine-tuning approaches.\n\n[a] Boyuan Chen et al. Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion. 2024.  \n[b] Jaesik Yoon et al. Monte Carlo Tree Diffusion for System 2 Planning. 2025.  \n[c] Yanwei Wang et al. Inference-Time Policy Steering through Human Interactions. 2025.  \n[d] Guangyao Zhou et al. Diffusion Model Predictive Control. 2025."}]

Author rebuttal:

Meta-review: The paper is judged good from the theoretical and experimental perspectives by all reviewers. The paper has (or at least had before the rebuttal) some weaknesses and concerns that have been addressed during the rebuttal phase:
- The clarity of the methodology was discussed (Reviewer gnZK and to some degree Reviewer k36W),
- The relatively limited novelty was mentioned either explicitly or through comments about some other related work.
- There were mixed opinions on the experimental scope. Reviewer gnZK considered them as narrow while Reviewer GPcL considered them as extensive even though not on large tasks.

Overall, the rebuttal from the authors convinced all reviewers to either stay or move to a score of 4 (weak accept). Therefore the recommendation is to accept the paper.
Decision: Accept (poster)