Title: Prior-Guided Diffusion Planning for Offline Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion Models

Offline Reinforcement Learning

Planning
Abstract: Diffusion models have recently gained prominence in offline reinforcement learning due to their ability to effectively learn high-performing, generalizable policies from static datasets. Diffusion-based planners facilitate long-horizon decision-making by generating high-quality trajectories through iterative denoising, guided by return-maximizing objectives. However, existing guided sampling strategies such as Classifier Guidance, Classifier-Free Guidance, and Monte Carlo Sample Selection either produce suboptimal multi-modal actions, struggle with distributional drift, or incur prohibitive inference-time costs. To address these challenges, we propose \textbf{\textit{Prior Guidance}} (PG), a novel guided sampling framework that replaces the standard Gaussian prior of a behavior-cloned diffusion model with a learnable distribution, optimized via a behavior-regularized objective. PG directly generates high-value trajectories without costly reward optimization of the diffusion model itself, and eliminates the need to sample multiple candidates at inference for sample selection. We present an efficient training strategy that applies behavior regularization in latent space, and empirically demonstrate that PG outperforms state-of-the-art diffusion policies and planners across diverse long-horizon offline RL benchmarks. Our code is available at https://github.com/ku-dmlab/PG.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Prior Guidance (PG) an offline RL approach to guide the diffusion-based planner. The authors argue that current diffusion-based guidance methods including Classifier Guidance and Classifier-Free Guidance and Monte Carlo Sample Selection (MCSS) suffer from limitations such as computational inefficiency, overfitting to out-of-distribution (OOD) actions etc.\n\nPG proposes to replace the standard Gaussian prior in the difussion model with a learned, state-conditioned Gaussian prior, optimizing using a behavior-regularized objective. To avoid backpropagation through the diffusion process, the authors introduce a latent value function that operates directly in the prior space, enabling efficient training and inference.\n\nPG is evaluated empirically on long-horizon planning tasks such as D4RL, Mujoco control suit and Maze2D-U.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- Reformulating the guidance objective in the latent space and optimizing the prior directly is theoretically sound and overall well-motivated.\n- PG avoids the high inference-time cost of MCSS and eliminates backpropagation through denoising network.\n- PG enables tractable, analytically stable regularization by performing directly in the prior space \n- Good performance across some of the baselines including Kitchen add Maze2D-U, which require multi-step planning and skill composition.\n- Overall paper is well-written and easy to follow.\n- The paper provides informative comparisons and ablations between PG and without denoising backpropagation as wel as existing methods like MCSS and Diffusion Q-learning.\n\n## Weaknesses\n- I have some concerns with regards to the empirical results. In Mujoco control environments like Walker2d, HalfCheetah and Hopper PG is consistently outperformed by existing baselines. This raises questions about its generality beyond long-horizon planning.\n- It was also not clear to me what constitutes the Kitchen benchmark, the paper keeps refering to \"Kitchen\" without explaining that this is the FrankaKitchen environment from D4RL? For readers unfamiliar with it, the task complexity and relevance remain unclear (and whether the single reported results is the average across multiple kitchen tasks). It might be more helpful to see performance of individual tasks since this method has shortcomings in some of these mujoco environments.\n- Theoretical justification hinges on DDIM being bijective which in parctice it may not hold (because it's an approximate ODE solver)\n- PG uses a relatively simple Gaussian prior which may not be sufficient in stochastic behavior setting\n\nOverall, I like the ideas in this paper. I would be open to raising my score if the authors can convincingly explain the reasons behind the weaker performance on standard control tasks and provide assurance that these results do not reflect a fundamental flaw in the proposed method.\n\nQUESTIONS:\n- Can you clarify which specific Kitchen tasks were used? Is the reported score averaged over several tasks (e.g., kitchen-mixed-v0, kitchen-partial-v0, etc.) or based on a single one?\n- Why does PG underperform on standard MuJoCo tasks? These tasks are supposed to be easy and saturated within the RL domain. Do you attribute this to horizon length, simplicity of dynamics, or limitations of your prior formulation?\n- How is the inverse dynamics model trained and evaluated?\n\nLIMITATIONS:\n- Domain sensitivity: PG performs well on temporally extended tasks (kitchen, Maze2D-U) but fails to match baseline in standard control environments, suggesting its benefits are limited to certain settings.\n- Because the denoising model is frozen, PG may not adapt well to high-return regions discovered during training - If I understood correctly, PG freezes the diffusion model after behavior cloning. So only the prior is trained to bias generation towards higher-value trajectories. But now suppose the value function discovers a new region in latent space where higher returns can be achieved. Now if we want to shift the prior to sample more from this region but the decoder is unchanged, it may not accurately reconstruct those high-return trajectories.\n- Restricting the prior to a Gaussian form limits expressiveness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Prior Guidance (PG), a novel method for improving diffusion-based planners in offline reinforcement learning (RL).\nthe key idea is to learn a prior conditional on the state instead the gaussian prior in order to steer the diffusion samples to high return regions. At training, the author propose to learn a latent value function (value function that depends on the noise sampled from the prior) to avoid backpropagate gradient through the denoising steps. at inference time, we need just sample from the learned prior which save runtime comparing to other guidance method (classifer guidance or Monte Carlo sample selection)\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Computational Efficiency: Reduces inference time significantly by avoiding sampling multiple candidate trajectories (unlike Monte Carlo methods).\n- Avoids backpropagation through the denoising process by using a latent-space critic.\n- Closed-form Regularization: Behavior regularization is done in the latent space where densities are tractable (Gaussian), allowing for closed-form solutions and improved stability.\n- The method builds on existing diffusion planners with minimal changes: just replace the prior and train an auxiliary value function.\n\nWeakness: \n\n- Additional Complexity: comparing to MCSS, the proposed method requires training and tuning of a latent-space value function and a learnable prior, adding architectural and training complexity.\n- in order to get a tractable regularization term, the author assumes bijectivity of the denoising scheme, which is not really satisfied.\n- Empirically speaking, while the proposed seem to work well, the performance boost is not too important. I think the performance is quite saturated in D4RL Benchmark. it would be important in include more challenging domains (e.g long-horizon tasks in ogbench, humanoid-large-maze ?)\n\nQUESTIONS:\nsee weakness above\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel method, Prior Guidance (PG), into the standard diffusion planning framework. Instead of sampling from a fixed Gaussian prior, the method samples from a learnable prior, which allows the policy to be optimized via a behavior-regularized objective. Experimental results demonstrate that PG performs well across multiple tasks, particularly on long-horizon problems. It also outperforms standard Monte Carlo Sample Selection (MCSS) methods in both performance and computational efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The idea of introducing a learnable prior into the sampling process of diffusion planning is quite novel.\n\n2. Theoretical analysis is provided to justify the training objective of prior guidance and its validity.\n\n3. The paper presents comprehensive experimental results, offering a convincing evaluation of the proposed method.\n\n4. Concrete and detailed implementation details are provided in the appendix.\n\nWeaknesses:\n\n1. Although the proposed PG method performs well on some long-horizon tasks, it does not show better performance on MuJoCo simulation tasks compared to standard diffusion-based policies such as Diffusion-QL.\n\n2. It is unclear how much of the performance gain comes from the incorporation of Diffusion Veteran (DV). The authors are encouraged to include an ablation study using the standard Diffuser as the base model to better isolate the contribution of PG.\n\nQUESTIONS:\n1. Why does the proposed PG method not perform well on some datasets? For example, in Table 1, PG does not outperform some diffusion-based methods such as DQL in MuJoCo tasks. Does the method only work well on long-horizon tasks?\n\n2. How does the method achieve good performance on long-horizon tasks, consistently outperforming MCSS? It seems that prior guidance alone does not provide this capability. Does the improvement come from Diffusion Veteran (DV)?\n\n3. It is unclear whether the method modifies the training of the base Diffusion Veteran. Is the proposed prior guidance applied only at the sampling stage?\n\nLIMITATIONS:\nThe limitations of this work are discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Prior Guidance (PG) for diffusion planning for offline reinforcement learning. The idea is to steer a pretrained diffusion planner by learning a state-conditioned prior noise distribution that correspond to high-value trajectories. This avoids the need of multiple samples in Monte-Carlo search methods. Specifically, they parametrize the prior as a state-conditioned gaussian distribution, which is trained to maximize the (behavior) value function while staying close to the data distribution. By exploiting the bijective nature of the DDIM sampler and aliasing the sample-space value function to a noise-space value function, both objectives can be optimized in the prior space without backpropagating through the diffusion process. The paper evaluates PG on a suite of offline RL domains from the D4RL benchmark, where PG achieves strong performance compared to baselines, with a low computation cost.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The paper presents a simple idea that can be applied to any pretrained diffusion planner -- just steer the diffusion planner by learning the prior distribution.\n2. The mathematical derivations for the noise-space objectives (Equations 4 and 5) are sound.\n3. The empirical results demonstrate state-of-the-art performance on the D4RL benchmark.\n4. The ablation studies and analysis experiments are quite informative. For example, Figure 5 shows a clear tradeoff between optimality and value model exploitation as the number of samples in Monte-Carlo search grows. Figure 7 shows the correlation between the real value function and the aliased noise-space value function, revealing a mismatch in the bijective assumption (though empirically the policy performance is not affected).\n5. The writing is clear and easy to follow.\n\n**Weaknesses**\n1. The D4RL results are not a significant improvement compared to the strongest baselines (DV*, DQL).\n\nQUESTIONS:\n1. Why is the DDIM denoising process bijective in the limit of small discretization steps? Is there a prior work that discussed this?\n2. Equation 2 does not explain what the variables are, e.g. $V$ and $f$. It would help to explain them immediately after the equation.\n3. There is a recent related work on steering flow policies by learning the prior distribution [1]. They also train a noise-space value function to alias the sample-space value function. Can you discuss PG in relation to this work? \n\n[1] Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, Sergey Levine. Steering Your Diffusion Policy with Latent Space Reinforcement Learning\n\nLIMITATIONS:\nThe authors address the limitations of the paper, stating (1) additional training cost, (2) breaking the bijective mapping assumption at practical discretization steps, and (3) no image-based experiments as the main limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}