Title: Single-Step Consistent Diffusion Samplers
Venue: NeurIPS 2025
Accepted: no
Keywords: Efficient Sampling

Unnormalized Distributions

Consistency

Controlled SDEs
Abstract: Sampling from unnormalized target distributions is a fundamental yet challenging task in machine learning and statistics. Existing sampling algorithms typically require many iterative steps to produce high-quality samples, leading to high computational costs that limit their practicality in time-sensitive or resource-constrained settings. In this work, we introduce *consistent diffusion samplers*, a new class of samplers designed to generate high-fidelity samples in a single step. We first propose Consistency-Distilled Diffusion Samplers (CDDS), which demonstrates that consistency distillation can be accomplished within sampling contexts in the absence of pre-collected training datasets. To eliminate the need for a pre-trained sampler, we further propose Self-Consistent Diffusion Samplers (SCDS), which performs self-distillation during training. SCDS learns to perform diffusion sampling and to skip intermediate steps via a self-consistency loss. Through extensive experiments on a variety of synthetic and real-world unnormalized distributions, we show that our approaches yield high-fidelity samples using less than 1% of the network evaluations required by traditional diffusion samplers.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends the consistency model to the generation domain of unnormalized target distributions, and correspondingly proposes two approaches for efficient sampling from unnormalized target distributions: Consistency-Distilled Diffusion Samplers (CDDS) and Self-Consistent Diffusion Samplers (SCDS). CDDS distills a pre-trained diffusion sampler into a single-step sampler by leveraging intermediate states during the diffusion process, thereby reducing the reliance on large datasets. SCDS further eliminates the requirement for a pre-trained sampler by jointly learning diffusion transitions and large shortcut steps through a self-consistency loss. The methods are validated through extensive experiments on both synthetic and real-world benchmarks, demonstrating that they can achieve competitive sample quality with significantly reduced sampling steps.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n1. The paper presents a new class of samplers that can generate high-fidelity samples in a single step, which is a significant advancement in the field of sampling from unnormalized distributions.\n2. The paper provides theoretical results which shows that consistency functions can be distilled from diffusion samplers without requiring access to data from the target distribution.\n3. The authors conduct extensive experiments on a variety of synthetic and real-world distributions, demonstrating the effectiveness of their methods in terms of sample quality and computational efficiency.\n\n### Weaknesses:\n1. The proposed CDDS appears to be a rather straightforward adaptation of consistency models to the generation of unnormalized distributions, which may seem somewhat incremental in terms of innovation.\n2. The experimental section could benefit from comparisons to a wider range of baseline methods to further establish the superiority of the proposed approaches.\n\nQUESTIONS:\n1. Could you explain the differences between the CDDS proposed in the paper and the Consistency Distillation (CD) in the original consistency models?\n2. Could you elaborate on whether other diffusion acceleration techniques, such as shortcuts or consistency trajectory models, are applicable to the generation of unnormalized distributions? If so, should these state-of-the-art diffusion acceleration methods be included in the comparison?\n3. The self-consistency loss in SCDS is designed to ensure that a large step yields the same result as multiple smaller steps. How sensitive is this self-consistency loss to the choice of step sizes, and are there optimal step sizes that lead to faster convergence or better sample quality?\n4. Can the proposed SCDS be applied to existing image generation tasks, and what advantages does it have over existing methods?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces two diffusion sampling methods for unnormalized target distributions. Consistency-Distilled Diffusion Samplers (CDDS) leverage a pretrained diffusion sampler to perform single-step sampling via consistency distillation. Self-Consistent Diffusion Samplers (SCDS) learn to sample in a single step from scratch, combining score estimation and consistency learning within a unified training objective, and requiring no pretrained model. The paper evaluates their efficiency and effectiveness across a range of synthetic and real-world sampling benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n - The paper addresses the important problem of enhancing the sampling efficiency of diffusion samplers. Improvements in this area have the potential to substantially benefit a wide range of real-world applications that rely on efficient sampling methods.\n - The manuscript is well-organized and clearly written. The exposition is concise yet thorough, facilitating a clear understanding of the core contributions and methodologies.\n\n## Weaknesses:\n - CDDS requires both a pretrained sampler and additional training specific to CDDS, which introduces computational overhead. This cost must be amortized by generating a large number of inference samples to be justifiable.\n - The conceptual novelty of CDDS appears limited. It builds upon the same consistency principle as used in Consistency Models (Song, 2023), and Theorem 1 is borrowed from that work. While the application to sampling is different, the methodological innovation is relatively incremental.\n - The empirical results, particularly for SCDS on real-world benchmarks, are not consistently compelling. To more convincingly demonstrate the advantages of the proposed method, a broader experimental evaluation is needed. This should include:\n   - Real-world benchmarks such as Bayesian inference tasks.\n   - Synthetic tasks with increased multimodality and higher dimensionality (where longer diffusion chains may better explore the space).\n   - Detailed analysis of how reduced NFEs translate into actual computational (time) savings (both at training and inference time).\n   - Investigation into how much the performance can be improved by increasing the number of sampling steps until the computational budget is matched with compared baselines.\n   - Provide error bars for experimental results.\n\nQUESTIONS:\n- Could the authors clarify which specific discrepancy measure is used in Equation (7)?\n - How do the proposed methods compare to the baselines in terms of computational capacity, such as the number of parameters? This is important for fair comparisons.\n - How strong/realistic is the assumption of a Lipschitz constraint in Theorem 1, especially in the SDE setting where involved quantities may not be differentiable? Additionally, how does the method account for or bound the error when the corresponding loss function exceeds zero?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a consistency model approach for sampling from unnormalized distributions using only one (or a few) steps. The authors propose two variants: (1) CCDS, which distills an existing trained diffusion-based sampler into a consistency model; and (2) SCDS, which trains a consistency model from scratch using a combination of a sampling loss and a self-consistency loss.\n\nCompared to existing diffusion-based samplers, the proposed methods reduce the number of steps required to generate target samples. Additionally, as the proposed approach is grounded in the optimal control framework, it is able to estimate the normalization constant of the target distribution. \n\nBoth CCDS and SCDS are benchmarked against established diffusion-based samplers on a range of synthetic and real-world Bayesian posterior inference tasks. While CCDS appears to have reasonable compute-accuracy trade-offs, the results for SCDS are a bit mixed.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper is well-motivated with the goal to address the challenge of reducing the number of steps required in diffusion-based samplers. \n\n- The proposed consistency model framework is both intuitive and appealing, offering a promising direction for efficient sampling.\n\n- Empirically, CCDS demonstrates reasonable compute-accuracy trade-offs and even occasionally outperforms diffusion baselines.\n\nWeakness:\n\n- The writing could be significantly improved. In particular, the background on consistency models should be made more technical, especially given how frequently the concept is referenced. This would also help clarify the novelty of applying consistency models in the sampling setting. Additionally, the probability flow ODE, which central to the proposed method, is referenced multiple times but never written out explicitly. There are also typos and unclear sentences throughout (see \"Questions\" section for examples).\n\n- The empirical advantages of CCDS and SCDS remain unclear. Results lack standard errors, and statistically significant differences are not clearly highlighted. SCDS appears to underperform in several cases, especially on the ELBO metric in Table 2.\n\n- Limited novelty for the CCDS approach as it seems to  a straightforward application of consistency distillation to a pre-trained diffusion sampler. \n\n- Several related works on single-step samplers are missing from the discussion and empirical comparison, e.g. [1,2]/ \n\n[1] Efficient and unbiased sampling of boltzmann distributions via consistency models\nFengzhe Zhang, Jiajun He, Laurence I Midgley, Javier Antorán, José Miguel Hernández-Lobato\n\n[2] Training Neural Samplers with Reverse Diffusive KL Divergence\nJiajun He, Wenlin Chen, Mingtian Zhang, David Barber, José Miguel Hernández-Lobato\n\nQUESTIONS:\n- Eq 2,3: what are $\\pi$ and $\\tau$? Are they supposed to be the diffusion prior $p_\\text{prior}$ and target $p_\\text{target}$ mentioned in the text above? \n\n- Eq 2, 3: which one is the reverse  (generative) process and which one is the forward process? From the following definition of $\\mu_\\theta$ as a learned control, it appears that Eq2 corresponds to the reverse process. Could the authors confirm?\n\n- Eq 6: $u$ should be $u_\\theta$?  And $p_\\text{prior}, p_\\text{target}$ should be $\\pi$ and $\\tau$? \n\n- The definition of consistency function in line 117 of page 3 seems to be fairly arbitrary. Is the consistency function $f$ in Theorem 1 uniquely defined? And what does it mean for \"consistency function of a probability flow ODE\"? \n\n- Instead of using the large datasets, CCDS integrates partial ode. Can this trick be applied to standard consistency model training?\n\n- Eq 7, should be a minus sign in between functions?\n\n- The comment after Theorem 1 says \"consistency functions can be distilled from diffusion samplers when only an unnormalized density oracle is available\" -- I wonder where the unnormalized density is used for distillation?\n\n- In practice, how is the sampling loss in eq 5 computed?\n\n- Does estimating the log normalization constant, or ESS,  require discretizing the diffusion chain over multiple timesteps? In that case, is NFE of CCDS or SCDS still 1?\n\n- The discretized RND should be explained in more details. \n\n- In appendix line 813 of page 21, \"v=0\" is assumed -- why this is the case?\n\n- On page 7 line 248-249: is N the number of sampling steps? What is the \"running-cost term\"?\n\n- Table 2 results: the authors may also consider computing the posterior predictive log likelihood on a held-out test dataset.\n\n- Could the authors provide a clearer summary of the empirical performance of CCDS and SCDS, especially considering statistical significance?\n\nLIMITATIONS:\nThe paper does not appear to include a discussion of its limitations. While SCDS does not seem to yield consistently strong empirical results, CCDS also has an important limitation: it requires a pretrained diffusion sampler, which may itself be inaccurate or costly to train. A potential avenue for improvement could be to incorporate a sampling loss into CCDS, which may help correct deficiencies in the pretrained model during distillation.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces **single-step diffusion samplers** that generate samples from unnormalized distributions without requiring iterative SDE simulation. Two methods are proposed:\n\n1. **Consistency-Distilled Diffusion Samplers (CDDS)**: distills a pre-trained sampler into a single-step function using consistency training, even without full trajectories.\n2. **Self-Consistent Diffusion Samplers (SCDS)**: learns a single-step sampler **from scratch**, combining traditional sampling loss with a novel **self-consistency loss** across time scales.\n\nThese methods amortize inference-time costs into training, yielding significant speedups while maintaining high-quality sampling. Extensive experiments are conducted on both synthetic and real-world unnormalized distributions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Dramatic reduction in inference cost is highly relevant\n- Extends consistency training to **unnormalized sampling** tasks\n- encouraging theoretical results for CDDS training\n- encouraging experimental results on small benchmarks\n\nWeaknesses:\n- Most results focus on 2D synthetic or small-scale benchmarks; scalability to image generation is not demonstrated\n- Lack of theoretical grounding and/or motivation on the key innovation for SCDS samplers\n\nQUESTIONS:\n- Why should the self-consistency objective work? Why does this trick on 'synthetic' samples from the control function work? If there would be theoretical/intuitive/toy problem justification for this, the paper would be a lot stronger, as SCDS is the main contribution of the paper.\n- Why would the SCDS method not degrade the sample quality? Perhaps there are some theoretical results you could draw on regarding samplers not perfectly recovering the reverse trajectory (prior to target) but still having good performance. \n-  Can CDDS or SCDS be used for image generation tasks? If not, clarify limitations. Showing results on CIFAR-10 or MNIST would broaden impact.\n\nLIMITATIONS:\nNo, negative societal impacts were not addressed. How could these samplers be negatively affect society?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper propose to extend on-step consistency diffusion samplers that \"convert\"/distill computationally intensive diffusion samplers (computationally expensive at inference because they require a lot of ODE/SDE steps) into computationally efficient one-step samplers.\nThese consistency models already existed, and the paper extends this idea to obtain get efficient unnormalized distribution samplers.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- The paper is very clear, Figure 1 is the clearest explanation of diffusion models I have ever seen.\n- Experiments show one-step sampling with only mild performance decrease \n\nWeaknesses:\n- I do not understand the novelty of the paper: once one has a sampler one can distill it with a consistency loss, whether the sampler is built to sample from unnormalized data distribution, or sample from an empirical data distribution\n\nQUESTIONS:\n- Can authors explain in which way the proposed method is new? and not \"just\" distill a sampler, as in the original consistency sampler?\n- Maybe it would help to clarify the following sentence: \"Notably, unlike standard consistency generative models, the states $\\hat{X}_{t_n + 1}$ and $\\hat{X}_{t_n }$ are obtained from partial integrations of the probability flow ODE rather than from real data samples.\". Could you provide a clear step-by-step comparison of consistency models for generative models and for unnormalized distribution sampling?\n- Can you also comment on the following? \" While our distillation approach builds upon the core principles of consistency generative models, it differs in setting and requirements. **Instead of relying on having access to a dataset from ptarget , our method extends consistency distillation to sampling from unnormalized distributions, making it applicable beyond generative modeling tasks.**\" The way I understand this last sentence is that the proposed method only requires a sampler (whether trained on data or unnormalized density), but once you have the sampler, the distillation is the same.\nIf this is not the same, could you please provide a detailed explanation of why this is not the case?\n\nI am not familiar with the standard experiments in the sampling literature, and I am not able to asses how challenging/significative the experimental results are. It would help if authors could comment on it.\n\nI am very willing to change my score if authors provide a clear answer to my main concern\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-10"}