Title: VarFlow: Proper Scoring-Rule Diffusion Distillation via Energy Matching
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion

Scoring Rule

Distillation

VSD
Abstract: **Diffusion models** achieve remarkable generative performance but are hampered by slow, iterative inference. Model distillation seeks to train a fast student generator. **Variational Score Distillation (VSD)** offers a principled KL-divergence minimization framework for this task. This method cleverly avoids computing the teacher model's Jacobian, but its student gradient relies on the score of the student's own noisy marginal distribution, $\nabla\_{\mathbf{x}\_t} \log p\_{\phi,t}(\mathbf{x}\_t)$. VSD thus requires approximations, such as training an auxiliary network to estimate this score. These approximations can introduce biases, cause training instability, or lead to an incomplete match of the target distribution, potentially focusing on conditional means rather than broader distributional features.
We introduce **VarFlow**, a method based on a **Score-Rule Variational Distillation (SRVD)** framework. VarFlow trains a one-step generator $g_{\phi}(\mathbf{z})$ by directly minimizing an energy distance (derived from the strictly proper energy score) between the student's induced noisy data distribution $p_{\phi,t}(\mathbf{x}_t)$ and the teacher's target noisy distribution $q_t(\mathbf{x}_t)$. This objective is estimated entirely using samples from these two distributions. Crucially, VarFlow bypasses the need to compute or approximate the intractable student score. By directly matching the full noisy marginal distributions, VarFlow aims for a more comprehensive and robust alignment between student and teacher, offering an efficient and theoretically grounded path to high-fidelity one-step generation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- Focuses on distilling diffusion teacher models to efficient single-step student models\n- Identifies a limitation of previous variational distillation methods regarding the computation of the student time conditioned score, which is often intractable. Argues that this results in biases, which could lead to sub-par performance. \n- Propose a new loss based upon the energy distance between teacher and student distributions, which only depends on samples\n- Demonstrate that proposed distillation loss enables superior student models due to better learning dynamics and provides ablation over key hyperparameters\n\nSTRENGTHS AND WEAKNESSES:\n*Quality*\n\nThe quality of the experiments and theoretical motivation for this paper are fairly high. The loss proposed seems very theoretically motivated and avoids the need for estimating intractable terms that previous losses suffer from. The experiments are fairly thorough with sufficient metrics presented for a range of different NFE settings. The only flaw would be the lack of discussion or inclusion of consistency models, which I discuss further in the questions.  \n\n*Clarity*\n\nI found the paper to be very clear in terms of the core problem identified and the proposed solution. The paper was well organized and well written. \n\n*Significance*\n\nThe results seem to indicate that this method does provide significant advantages over baselines in terms of superior quality at low NFE. \n\n*Originality*\n\nThe use of energy distances as the basis for the distillation loss seems highly original.\n\nQUESTIONS:\n*Algorithmic*\n\nIt seems that the estimator used for the loss requires some form of pairwise distance calculations between the samples from the student and the samples from the teacher based on equations 9 and 10. I would think that this increases the cost of the training method relative to prior methods. I would appreciate a discussion on the training cost of the proposed algorithm against prior distillation techniques. \n\n*Empirical*\n\nWhy are consistency models not included in the baselines? While different in methodology, this class of generative models also aims for single step generation of high quality images. I feel that this is important to discuss in the context of single step generation. \n\n\nOverall, I am leaning towards weak acceptance. It is weak due to the lack of comparison against consistency models, which are also targeted towards single step generation. If the authors are able to justify the omission or point me to any discussion of consistency models in their submission, I am happy to raise my score.\n\nLIMITATIONS:\nI think the omission of consistency distillation [1] is a fairly large limitation, as these models have been shown to be very competitive on conditional and unconditional generation at low NFE.\n\nOne limitation is that this framework is only applicable to distillation pipelines, whereas methods such as [1] can easily be applied to distillation or from scratch training. \n\n[1]. Simplifying, Stabilizing & Scaling Continuous Time Consistency Models. Cheng Lu & Yang Song. ICLR 2025.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces VarFlow, a novel framework for distilling large, multi-step diffusion models into efficient, single-step generators. The authors identify a key challenge in prior work like VSD: the need to compute or approximate the student model's intractable marginal score function, which can introduce bias and instability. VarFlow's core contribution is to reframe the distillation objective. Instead of minimizing KL-divergence, it directly minimizes the energy distance between the noisy distributions induced by the student and the teacher. This objective can be estimated purely from samples, thus circumventing the need for student score estimation. The authors provide theoretical justification for their method and demonstrate sota performance on several benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The method is theoretically grounded, and eliminates the time to calculate the Jacobian, which is typically used in VSD.\n2. Extensive experiments have been conducted regarding pixel space, latent space diffusion models, ablations are complete regarding key design components.\n\n**Weaknesses**\n\n1. The training cost is not explicitly evaluated in the paper. This raise questions on the fairness of comparison.\n2. The use of a sample-based Integral Probability Metric (IPM) like the energy distance to match two distributions is a foundational concept in generative modeling, most notably in the development of Generative Adversarial Networks (GANs) and their variants (e.g., MMD-GANs). The authors should discuss the relationship between them.\n3. The presentation/figure illustration could be strengthened.\n\nQUESTIONS:\n1. As mentioned in weaknesses, could the authors comment on the training-time cost of VarFlow? How does it compare to a key baseline like VSD, which requires co-training an auxiliary score network?\n2. In SD3 for example, how many steps are required to generate synthetic data functioning as real data points, how does this part contribute to the total training cost?\n3. Could the authors compare the recent SOTA methods like RAYFLOW[1] for distillation?\n\n\n[1] RayFlow: Instance-Aware Diffusion Acceleration via Adaptive Flow Trajectories\n\nLIMITATIONS:\nUsing Varflows to distill modern large scale text-to-image diffusion/flow models (without any dataset) may require a large amount of synthesized data generated from the base model, which is time consuming, compared to other dataset-free distillation method. Rest of the limitations have been presented in the supplementaries by authors.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method for training generative models by minimizing the Energy Distance between different time steps to train a one-step generative model. It bypasses the need to estimate a student score network and demonstrates strong performance without initializing the one-step generator with the teacher’s weights.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**  \nThe idea is neat, and the results are surprisingly good, especially for training from scratch. I’m really looking forward to the open-sourced version of this work. The experiments are sound, and I believe this paper will make a valuable contribution to the generative modeling community.  \n\nThere’s also the recent paper [*Towards Training One-Step Diffusion Models Without Distillation*](https://arxiv.org/pdf/2502.08005), which doesn’t require teacher score or student score estimation, but still relies on a classifier for class-ratio estimation and requires initializing the student weights with those of the teacher. The method proposed in your paper further addresses these two limitations by using an MMD-style discrepancy. I’m surprised it works so well and am eager to see the code released.  \n\n**Weaknesses:**  \nMany relevant citations are missing, which I believe are important to discuss in the paper:  \n\n1. MMD: [A Kernel Two-Sample Test](https://jmlr.csail.mit.edu/papers/v13/gretton12a.html)  \n2. MMD diffusion: [Distributional Diffusion Models with Scoring Rules](https://arxiv.org/abs/2502.02483), [Inductive Moment Matching](https://arxiv.org/pdf/2503.07565v2)  \n3. Distillation without a student score model: [Towards Training One-Step Diffusion Models Without Distillation](https://arxiv.org/pdf/2502.08005)  \n4. MMD GAN: [MMD GAN: Towards Deeper Understanding of Moment Matching Network](https://arxiv.org/abs/1705.08584) and [Generative Moment Matching Networks](https://arxiv.org/abs/1502.02761). These are especially important since the proposed method essentially performs MMD without kernel learning but in diffusion space. Also see [Diffusion GAN](https://arxiv.org/abs/2206.02262).  \n\n**Suggestion:**  \nIt might be more impactful to present this work as a new method for directly training a one-step model from scratch. Framing it purely as a distillation method can be limiting, as some might argue that in certain cases, there’s no training data available and only the teacher’s score is accessible—for which only methods like [DiffInstruct](https://arxiv.org/abs/2305.18455) would work.  \n\nFor instance, one could define a spread energy discrepancy or diffusion energy discrepancy, similar to [1, 2, 3], as a divergence between the clean distributions. For example, you could define a  diffusive energy discrepancy (DED):\n\n$$DED(p_\\theta(x_0) || q_d(x_0)) \\equiv  \\frac{1}{T} \\sum_{t=1}^{T} ED(p_\\theta(x_t)|| q_d(x_t)),$$\n\nwhich is a valid divergence between the clean model distribution $DED(p_\\theta(x_0)||q_d(x_0))=0 \\iff p_\\theta(x_0)=q_d(x_0)$, see Appendix C of [1] for a proof. This divergence, with sample-based estimation, could be used to train a one-step model entirely from scratch.  This would be amazing.\n\n- [1] [Spread divergence](https://arxiv.org/pdf/1811.08968)  \n- [2] [Integrated KL divergence](https://arxiv.org/abs/2305.18455)  \n- [3] [Diffusive KL Divergence](https://arxiv.org/abs/2410.12456)\n\nQUESTIONS:\nNo\n\nLIMITATIONS:\nThe MMD training quality depends on the batch size, which needs to be large to obtain good results. This requires larger training computation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new training objective called VarFlow, which can train a one-step generative model, either from scratch or building upon a pretrained diffusion model. The idea is to minimize the energy distance, which is a strictly proper scoring rule, between the noisy data distribution and noisy model distribution, over multiple noisy levels. Unlike the dominant approaches, the paper's method is claimed to be able to directly train a high-quality one-step generative model without learning an auxiliary model, like an additional score model for the student distribution. The experimental results are impressive, outperforming almost all existing benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\nTraining high-quality one-step (or few-steps) neural sampler is one of the most important problems in the field. \nSince the advent of diffusion models, most of the recent proposals focus on distilling a one-step sampler out of a high-quality pretrained diffusion model. Consistency training is one of few exceptions that try a fundamentally different approach, allowing training from scratch.\nAlthough this paper frames and emphasizes the proposed framework as a \"distillation\" framework, but as demonstrated in Experiments, this technique can be used to train a one-step sampler from scratch.\nThe authors demonstrates impressive empirical results with the proposed technique across small and large scale datasets. \nThis can be a game-changer and breakthrough, if the training is easy (i.e., stable).\n\n\n## Weaknesses \nThat said, I have several concerns on the clarity of the paper, especially on the originality of the idea and experimental results.\n\n### 1. On novelty\nWhile the paper mentions Distributional Diffusion Models in the main text as its main inspiration, but they do not have any citation for it. I happen to have prior exposure to related literature, specifically [A] and [B] below. Both papers apply energy distance minimization to train generative models. In particular, Eq. (9) of the current manuscript seems to be almost equivalent to Eq. (14) in [B]. I am quite confused about whether the current manuscript proposes the same method and simply demonstrates its effectiveness on complex datasets. Note that both [A] and [B] only report preliminary results on relatively toy/small-scale datasets.\n\n- [A] Shen et al., [Reverse Markov Learning: Multi-Step Generative Models for Complex Distributions](https://arxiv.org/pdf/2502.13747), arXiv 2025.\n- [B] De Bortoli et al., [Distributional Diffusion Models with Scoring Rules](https://arxiv.org/pdf/2502.02483), arXiv 2025.\n\n### 2. On experimental results\nBesides its methodological novelty, another big concern I have is about the source of the numbers reported in Table 1. I have some familiarity in the literature, and most of the recent papers often do not report benchmark on large datasets such as ImageNet 256x256 to my knowledge. For example, I just checked that CT, iCT from (Song et al., 2023), ECT (Geng et al., 2024), SMT (Jayashankar et al., 2025) do not report results for ImageNet 256x256 and MS COCO datasets. Since the numbers are not publicly available, I believe the authors implemented and ran all these baselines on the large datasets themselves, which I highly appreciate. However, I think it would be helpful to include more details about each baseline, since, for instance, it is known that consistency training (CT) methods often require careful tuning of the noise schedule when training from scratch, and the implementation of each baseline for such large datasets would have not been very straightforward. The same comment applies to the distillation results.\n\nQUESTIONS:\n## Questions\n- Why do the authors frame the proposed method as a \"variational distillation\" framework? I think there is no \"variational\" thing in the energy distance minimization framework, and it's not \"distilling\" a teacher model per se. Given this, I think comparing it with variational score distillation is quite confusing.\n- Can you clarify the methodological difference from [A] and [B] above, including the concerns I raised in Weaknesses?\n- Regarding experimental results in Table 2, can you clarify which numbers are directly from the paper and which are reproduced by the authors?\n- In the distillation setup, when the authors \"initialize\" a U-net with a pretrained diffusion model and train/use it as a one-step sampler, what value do you set for the time conditioning?\n- Is the training stable? Can you show loss trajectories from training?\n- A demonstration of this idea on any toy dataset would be very insightful.\n\n## Typos and suggestions\n- The notation $p\\_\\theta(\\cdot|\\mathbf{x}\\_t,t)(\\cdot|\\mathbf{x}\\_t,t)$ seems to be a typo (line 127, 129).\n- Figure 1 has too small fonts and texts are not readable.\n\nI will be happy to increase the scores if I have any misunderstanding and the authors can address my concerns.\n\nLIMITATIONS:\nLimitations are not clearly mentioned. I think the authors should particularly discuss its comparison to [A] and [B].", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}