Simple Distillation for One-Step Diffusion Models
Authors: Huaisheng Zhu
Teng Xiao
Shijie Zhou
Zhimeng Guo
Hangfan Zhang
Siyuan Xu
Vasant G Honavar
Venue: NeurIPS 2025 poster ()
Keywords: Efficient Image Generation
Primary area: applications

Abstract: Diffusion models have established themselves as leading techniques for image generation. However, their reliance on an iterative denoising process results in slow sampling speeds, which limits their applicability to interactive and creative applications. An approach to overcoming this limitation involves distilling multistep diffusion models into efficient one-step generators. However, existing distillation methods typically suffer performance degradation or require complex iterative training procedures which increase their complexity and computational cost. In this paper, we propose Contrastive Energy Distillation (CED), a simple yet effective approach to distill multistep diffusion models into effective one-step generators. Our key innovation is the introduction of an unnormalized joint energy-based model (EBM) that represents the generator and an auxiliary score model. CED optimizes a Noise Contrastive Estimation (NCE) objective to efficiently transfers knowledge from a multistep teacher diffusion model without additional modules or iterative training complexity. We further show that CED implicitly optimizes the KL divergence between the distributions modeled by the multistep diffusion model and the one-step generator. We present results of experiments which demonstrate that CED achieves competitive performance with the representative baselines for distilling multistep diffusion models while maintaining excellent memory efficiency.

Simple Distillation for One-Step Diffusion Models
Huaisheng Zhu', Teng Xiao 3.4, Shijie Zhou?,
Zhimeng Guo', Hangfan Zhang', Siyuan Xu', Vasant Honavar' Pennsylvania State University, University at Buffalo
3 Allen Institute for AI (A12) University of Washington hvz53120psu. edu
Abstract
Diffusion models have established themselves as leading techniques for image generation. However, their reliance on an iterative denoising process results in slow sampling speeds, which limits their applicability to interactive and creative appli- cations. An approach to overcoming this limitation involves distilling multistep diffusion models into efficient one-step generators. However, existing distillation methods typically suffer performance degradation or require complex iterative training procedures which increase their complexity and computational cost. In this paper, we propose Contrastive Energy Distillation (CED), a simple yet effective approach distill multistep diffusion models into effective one-step generators. Our key innovation is the introduction of an unnormalized joint energy-based model (EBM) that represents the generator and an auxiliary score model. CED optimizes a Noise Contrastive Estimation (NCE) objective to efficiently transfers knowledge from multistep teacher diffusion model without additional modules or iterative training complexity. We further show that CED implicitly optimizes the KL divergence between the distributions modeled by the multistep diffusion model and the one-step generator. We present results of experiments which show that CED achieves performance comparable to that of representative baselines for distilling multi-step diffusion models while maintaining excellent memory efficiency.
Introduction
Diffusion models have become the prominent method for image generation, capable of producing
highly realistic and diverse outputs through stable training process [18
45
49 44 Unlike
Generative Adversarial Networks (GANs) [14] and Variational Autoencoders (VAEs) 26), diffusion
models rely on iterative denoising procedure that progressively refines an initial Gaussian noise into detailed images [18] 50 This method, while effective, is computationally intensive, often requiring dozens
hundreds of neural network evaluations. Consequently, the slow sampling speed significantly reduces the practicality of diffusion models in interactive, creative applications.
To address this issue, several efforts have aimed at reducing the sampling steps required in reverse diffusion processes 34
68 52 These approaches still require multiple steps to generate images. To significantly improve the efficiency of diffusion models, one-step diffusion methods have been proposed. In particular, distillation techniques have become increasingly popular for one-step generation, achieving state-of- ofte-of-the-ar performance [70]. One line of work is score-based distillation, which includes iteratively training an auxiliary score model with the one-step generator [35] 61 65]
using additional discriminator create GAN 669 64 both obtaining great results.
However, these methods heavily rely on auxiliary models and the iterative training of multiple components, leading to substantial GPU consumption, prolonged training times, and increased complexity due to the additional hyperparameters introduced by these extra components, which require careful tuning. In contrast, trajectory-bast -based distillation offers more lightweight alternative
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

by progressively increasing the sampling intervals of the student model 31 46 51 29].. This strategy avoids the need for extra models and iterative training, significantly simplifying the training pipeline. Nevertheless, these techniques typically experience great performance degradation compared to the original diffusion models when generating images in just one step. Therefore, in this paper, it naturally raises a question: How can we design simple, efficient and effective distillation algorithm for one-step diffusion models without iterative training and additional components?
In this paper, we approach the problem by in- troducing an unnormalized joint Energy-based Model (EBM) that models the auxiliary score model (commonly referred to as the "fake score model" in prior work) and the one-step generator. Our key innovation lies utilizing this EBM distill knowledge from teacher diffusion model. The energy function designed to directly quan- tify the discrepancy between images generated by the one-step generator and outputs from the fake score model. A central challenge of our frame- work is to enable efficient training without relying on additional modules or complex architectures. To address this issue, we propose Contrastive En- ergy Distillation (CED), method that trains our joint maseg model inspired by the Noise Contrastive Estimation (NCE) framework [16]. Specifically, we sample synthetic images from a pre-trained teacher diffusion model prior to training, and from the generator's 's previous training iteration to serve as negative samples. These are then used to optimize the NCE objective. This approach removes the need for extra components and iterative updates, enabling more resource-efficient training compared to previous methods, as shown in Figure Furthermore, we show that, with a specific design of the energy function, our method implicitly optimizes the Kullback-Leiblee (KL) divergence an objective widely used in distribution matching distillation of one-step diffusion models. This provides a foundation for the effectiveness of our approach in transferring knowledge from multi-step diffusion models to single-step generator.
3x Memory
2 Memory
1x Memory
Fake Score Model
Fake Score
Fake Score Model
Model
Beertivv Update
One-step
One-stop Generator
One-step
Generator
Generator
Distillation
Distillation
Distillation
Teacher
Teacher
Teacher
Model
Model
Model
DMD/SID
(b) CD/Flow
(c) CED (our)
Figure 1: Training pipelines for different kinds
of distillation methods. Our method (CED) uses
only 1/3 or 1/2 the memory compared with them by loading fewer models during training.
In summary, the main contributions of this paper are: (i) We propose Contrastive Energy Distillation (CED), a simple and efficient distillation algorithm leveraging contrastive loss on our joint energy- based model to transform multi-step diffusion models into one-step generator. Inspired by noise contrastive estimation, CED eliminates the need for auxiliary modules, offering a novel and stream- lined approach for designing one-step diffusion model distillation strategies. (ii) We demonstrate that CED implicitly optimizes the KL divergence for distribution matching distillation, providing a principled foundation for its distillation effectiveness. (iii) CED achieves competitive performance that also rely solely on one-step generators during training without auxiliary components.
2 Related Works
Diffusion models have become powerful framework for generative modeling 18
53 44
50
22
achieving remarkable success across wide range of domains, including image generation
42
45
audio synthesis (27], video generation [0] 48, and molecular design [9 62]. These models work by gradually transforming random noise into coherent data through reverse diffusion process. Despite their state-of-the-art performance, the iterative nature of diffusion models leads to computational overhead, making them less suitable for real-world applications. To accelerate the sampling process of diffusion models, numerous techniques have been proposed to speed up generation through distillation. These approaches have achieved impressive results and can be broadly categorized into score-based
distillation 35| 47 65
69 34 and trajectory-basee distillation [52 46 32
51
36
63
Score-based distillation methods were initially introduced in the context of text-to-3D synthe- sis [40] (66 (55), where pre-trained text-to-image diffusion model is used to define a distribution matching loss. Inspired from these approaches, score distillation has been extended to the training of diffusion models themselves [65
II
35 37 These approaches either involve iteratively training an
auxiliary score network alongside the one-step generator K5
47 61 65.. or incorporate an additional
discriminator to form a GAN-like framework [69 64 to further improve their performance. While effective, these methods depend heavily auxiliary models and the iterative training of multiple
2

---

components, resulting in high GPU usage, longer training durations, and added complexity from extra hyperparameters associated with the additional components that require careful tuning.
Trajectory-based Distillation methods eliminate the need for additional modules and iterative training, which greatly simplify the overall pipeline. They offer more streamlined approach by
progressively increasing the sampling intervals of the student model [I 15 31 32
46
51 (9]
Luhman al. [34] precompute dataset of image-noise pairs using an ODE sampler and train
student model to learn the mapping in single step. Recified Flow simplifies the ODE trajectories
to make them easier for one-step models to learn [31 32 Progressive Distillation eliminates the
need for precomputed data by iteratively training student models, each reducing the number of sampling steps [46]. Consistency Distillation (CD) 52] and TRACT [1] further improve training by enforcing self-consistency of the student's 's outputs along the ODE trajectory, aligning them with
the teacher model. Moreover, Flow matching (FM) 30 based methods distill themselves to shorten sampling steps. However, when used to generate images in a single step, these methods often suffer from significant performance degradation compared to the original diffusion models. Therefore, in this paper, we propose a simple, efficient, and effective approach to distilling multistep diffusion model into a one-step generator using Contrastive Energy Distillation (CED). CED requires neither auxiliary models nor iterative training, and outperforms trajectory-basee distillation methods, which rely on loading the teacher model during training. CED relies solely on the one-step generator during training, which make the process both simple and efficient.
3 Notations and Preliminaries
Diffusion Models. Given a dataset fx(*), ,..., x(N} sampled from a real data distribution Preal (x) diffusion models learn to generate these samples by progressively denoising corrupted versions [8
[50]. In the forward diffusion process, noise is incrementally added to a sample x ~ Preat across T timesteps until it becomes pure Gaussian noise. Formally, at each timestep t, the diffused samples
follow Preat (xx) = S Preat (x)q (x; x) dx with q(x x) ~ NV (axx, II where (tt ot > are scalars determined by the noise schedule. During training, the model learns to reverse this corruption process by the distribution g(x,-1 xx) ~ A/ (x--1:4 / (x,, f) oT which predict a denoised estimate, u(xe, t), conditioned on the timestep and the noisy sample xt. By iterative refinement, the model recovers images resembling those drawn from Preat. Once trained, the denoised estimate aligns with the gradient of the likelihood function (i.e., the score function) of the diffused distribution:
Syeal (xe, t) Vx. log Preat , (xx) X2 Ots/ (xt,t)
(1)
of
Distribution Matching Distillation (DMD). To distill a pretrained multi-step diffusion model into single-step generator Go. a distribution matching approach can be used 64 65). This method seeks to minimize the expected approximate Hellback-Leibler (KL) divergence, LDMD =
E, (IDLL (Ptake . Pleaa 2)). between the diffused target distribution, e and the diffused genera- tor distribution, Ptake,t, across different timesteps with the following objective function:
VOLDMD E. (ʃ (Sreal (F (Go(z), ()), Stake (F (F(Go()),t), f))
dGo(z) dz
(2)
d0
where z ~ 1), is the guassian noise, F' is the diffusion forward process with noise level corresponding time step and Sreal and Stake are score function defined in Equation ( DMD employs a frozen pre-trained diffusion model, Preat as the teacher, while iteratively updating Make during the training of G.. This involves denoising score- matching loss applied samples produced by the one-step generator Gø where z ~ (0, I) is a guassian noise.
Reinforcement Learning. When fine-tuning models with reinforcement learning, the learned reward function serves as a feedback to guide the model's training. Specifically, given a noise vector (O I), we generate image based on this input. Following widely adopted optimization objective from the fine-tuning of large language models (LLMs) (20 (2]) the training objective is:
max Ni. (x, 2)] 3DKL [Go (x 2)]lGa (x x)].
where we consider modeling the reinforcement learning process for the one-step generator Go and O' serves as a reference model from previous optimization step or supervised fine-tuned model, similar to those commonly used in large language model literature. The reward function r-)() is trained prior to the reinforeement learning stage and provides feedback as the reward model.
3

---

4 Simple Distillation Approach for One-Step Diffusion Models
In this section, we introduce simple Contrastive Energy Distillation (CED) based approach to training one-step generator that mimics the generative process of a multistep diffusion model. We introduce a joint energy-based model that represents the auxiliary (fake) score model Stake and the one-step generator Ge. To simplify the training pipeline and avoid iterative updates involving auxiliary models (fake score models), we employ CED to directly train the one-step generator using an approach inspired by Noise Contrastive Estimation (NCE). We leverage connection between CED and distribution matching distillation methods to design a simple and efficient method to distill multi-step step diffusion model into one-step generator. Our approach requires loading the student model only during training, significantly reducing computational complexity and memory use while achieving competitive performance relative to Kate-of-the--e art methods.
4.1 Joint Energy-based Models
We can interpret the two-step learning process of one-step diffusion models as leveraging a "fake score model," Stake designed approximate the distribution of the one-step generator. However, during iterative training, optimization errors may prevent Stake from accurately reflecting the true distribution of the generator. Conceptually, this discrepancy can be viewed as an energy function that captures the difference between samples generated by the fake score model and those from the one-step generator. Based this viewpoint, we introduce an unnormalized density simultaneously represent the fake score model's 's distribution and its deviation from the neestee generator.
exp (-E E (x, Go(z)))
Pa (x z) = p (x
ZZ)
(4)
Zo (z)
where Zg is the partition function normalizing factor. The term Pa.s a (x z) denotes the distribu- tion that generates samples x from an input z drawn from pure Gaussian noise z ~ (O, P (x] z) denotes the distribution of samples produced by iteratively denoising 1). Similarly, (N by the reverse process of diffusion model based on the fake score model Stake,9 and x. E is used to measure the discrepancy between generated samples from fake score models and one-step genera- tors. Moreover, E(x, Gz)z can be viewed as a distance function between samples produced by the fake score models and those generated by the one-step generator-using, for instance, the 1.2, Secudo-Huber, or I.PIS--Hube distance ( In this formulation, a higher energy value indicates a larger discrepancy between the two sets of samples, while lower energy implies a lower discrepancy. Therefore, this joint EBM provides calibrated distribution for Plake samples that are well well-aligned with the one-step generator are assigned higher probabilities, while poorly aligned samples receive lower probabilities. In other words, pa,s offers a more accurate estimation of the image distribution of Gg, and also capturing the interplay between the fake score model and the step generator.
4.2 Training with Contrastive Energy Distillation
The goal of distillation methods is to align the distribution of the fake score model (Ptake) with that of the teacher diffusion model (Preat). In this paper, we use the calibrated distribution (joint EBM) po. introduced in earlier sections, to approximate the teacher model Preat Accordingly, we treat the joint EBM defined in Equation (4) as modeling the teacher distribution Preat- Based on this formulation, we optimize the parameters of the joint energy function using a conditional variant of Noise Contrastive Estimation (NCE) [66] First, NCE samples contrastive examples from both the data distribution and noise distribution that should closely approximate the data distribution. Second, it computes the likelihoods of these samples under both the model and the noise distributions, then optimizing a binary classification objective. Under our joint energy formulation from Equation ([), the log-odds reduce to log Pe. - log Pa = (xx, Go (z)), simplifying the objective to a standard binary classification form with z ~ (o, I) and referred to Contrastive Energy Distillation (CED):
CCED = -Ex-po. llog 1 + exp EE (x, Go(z))) - Ex'rp, [log
(5)
+ exp (-E (x', G
In this formulation, we replace pa,s with the teacher model Preal (x 2) as our target distilled distribution for sampling positive examples in the first term. Meanwhile, one-step diffusion distillation typically involves fitting the fake score model Ps to outputs from the generator Gø (z) of the previous

---

Algorithm Simple Distillation of One-step Diffusion Models
Require:
Teacher diffusion model Pinee
1: Initialize the one-step generator with the teacher's score network Gana (+) Strue (-)
2: Sample x from Puue (x z) with z ~ (0, I)
3: Warm-up train with C Exx, z) to obtain ones-step generator Gg.
4: Get negative samples x' from Ggr (z) and construct a triplet (x,x', z) for each z ~ IN(
5: for each training iteration do
6: Get a batch of samples (x,''22
7: Update one-step generator's parameters with LCED in Equation (4) with (x,x',
8: end for
optimization step, where 0' denotes the carlier parameter set of 0. To streamline the iterative process and avoid training an additional fake-score model, we replace py with Gy for the negative samples in the second term of Equation ( Since ps is designed to approximate the distribution of images from the previous optimization step following earlier iterative learning approaches for one-step diffusion models- substitution remains valid. Thus, propose the following loss to distill the teacher model without extra models or iterative training, thereby simplifying the distillation process:
CCED Exxxp_ (I + exp (E (x, Ge(z)))) Ex [log (1 + exp (-E E (x', (z()))) (6)
where the energy function E serves
distance metric, where smaller values correspond to reduced discrepancy between the two input samples. For instance, when using the L2 distance, we have E (x. Go (z)) || Gz(z) 12 To further streamline the training process, we begin with brief warm-up phase for the one-step generator Gg. We then optimize the objective specified in Equation (6). The full details of this procedure can be found in Algorithm 1
4.3
Relation to Distribution Matching Distillation
In this section, we establish the connection between our proposed objective and Distribution Matching Distillation (DMD) methods. We begin by formulating the KL divergence objective of distribution matching distillation from a reinforcement learning perspective:
max- -DKL (x Pxeapte 2)) = Eyg, [log Preat 2) - loog Plake (x 2)]
log (x z)
log
Plake [x| z)
Byg r(x,2) - IIKL (phace( (x
(x 2))
(7)
Prake (x z)
Ptake (x| z))
where rxx, 2) = log(Preal (x Pine (x 22) can viewed as an auxiliary reward function and Og the initial parameter of the fake score model, initialized from the teacher diffusion model. Hence, DMD can be interpreted in a manner analogous to Reinforcement Learning From Human Feedback (RLHF) 139 AT 60, 58 59) Specifically, the procedure has two stages: (1) learning reward model and (2) optimizing the primary objective. To relate DMD to our single-step training approach, we reformulate it that these two steps combined into single unified procedure. Specifically, begin with the standard RLHF training pipeline, where the first step is to learn a reward model. For the DMD objective in Equation 7 we use Density Ratio Estimation methods to learn the reward model log(Prea x 2)/p (x 2)). Since this process relies estimating the log ratio, a straightforward solution is to train a classifier (i.e., a discriminator) with logistic regression to approximate it:
LDEE - Ex-ppxallll [log (o(((x, 2))) E xppp [log (1 - a (1(x, 2))]
where (-) denotes the sigmoid function, and each data sample is treated as though it drawn from distribution with binary labels- one class for samples from Preal and one class for samples from Prake Then, the log density ratio can be linked to the optimal classifier probabilities via Bayes' rule 2
log Preal {x z)
P(c = 1) = x, log
(h' (x, z)) (x'
Take
x
P(c 0)
(hr (x,
where h²" (-) is the optimal solution for Equation 8 and is the constant ratio P(c = 1)/P(c 0) between the priors of two classes that can be estimated with sample size. Although can optimize
5

---

Equation 7 with the learned reward (i.e.. density ratio) from Equation (8) this still involves two- step process and additional computational overhead. To simplify matters and align with our proposed objective of Equation (6). we propose a direct optimization method for the DMD objective that does not require an RI. training loop or separate discriminator. The key insight lies in using a specific parameterization for the discriminator, which allows us to extract the optimal solution in closed form circummeennninn the iterative RL loop in Equation (7). Specifically, under this parameterization, the optimization problem defined in Equation 7 admits straightforward analytical solution:
Plake [x z) = Z(z)
Piake z) exp((xxxx z)),
(10)
where 7(z) fx Piake (x) 2)) exp(r(xx, (x) = fx Preat 2) = 1 by the definition of r(.) in Equation
7 and 0" is the optimal solution of paramter 0. A detailed derivation of this optimal format (Equation
(10 is provided in the Appendix [A.]] To unify the process into a single optimization step, we can
combine Equations
(TO) and ( with simple algebra, yielding the following relationship:
X 2)
(h" (x, [ (h'((,2)
log Pfake [x
2)
log
Take
log
h" (x, z)
(11)
8o Plake
x 2)
Piake x %)
Then, by combining the two-step optimization in Equation (8) with Equations and (5). a direct optimization algorithm, resulting in the following objective:
we derive
DiRE -Ex-Praul log (a (log
Prake 90 Pake
z))
})]
E x'
do log (1 (log
Plake x 90 luke
2
))]
(12)
P
(x
For p (xx z), we can use a single-step generator parameter, modeling the generator as a Gaussian distribution Plake ~ NGG I²)). Then, we can convert this objective with the trainable parameters by replacing 8" with 0 and we can see the relation with our proposed objective in Equation (6):
CDrE = Exx-pa [log (1 + exp E (x, Go(z))))] + Ex [log (1 + exp (-E (x', Go Go(z))))),
(13)
where E (x, Go(z)) - (log Plate (x 1z) log Prace (x zi) - -[|x- (z(z) |x Ga (z) 112). Therefore, although the objective in Equation (8) requires a two-step training process from the reinforcement learning perspective, we unify these steps by noting that they have the relation between two steps' optimal parameter (see Equations (II) and (12). By optimizing Equation (3) ) to its optimal solution, we implicitly fulfill the DMD objective, effectively merging the original two-step learning procedure into a single step. This streamlined strategy aligns closely with methods commonly employed in Direct Alignment Algorithms for RLHF 41), which further discuss in Appendix A.2 Furthermore, implicitly optimizing DMD objectives can be viewed as a special case of our proposed CED framework, realized through a specific definition of the underlying energy functions.
On
4.4 Generalized Extensions and Practical Implementations
We have presented a simple distillation algorithm for one-step diffusion models that aligns with distribution distribution-matchingg objectives in Equation (7). and we will next explore extension of this approach using different loss formulations. Since our CED in Equation (13) and ( can be seen as a
binary classification problem, it naturally extends more general form [ 13
LCED Exx-raa f (- Exx, Gz(z))) + Ex (-E((
(14)
We summarize the different classification loss functions ft and f (c.g.. Logistic, Hinge [7], Brier [13), Exponential (12) in Table [] includ- ing the Logistic loss from Equation TT). For practical implementation, the complete training procedure is detailed in Algorithm [ To sim- plify the training process and reduce computa- tional costs, we first generate synthetic samples from the teacher diffusion model and perform warm-up phase by optimizing the one-step gen-
Table Summary of various functions of general- ized extensions of CED for Equation (1.
Loss
f+ E)
f (-E)
Logistic Hinge
log(1 + e)
log(1 + BB
max(0, 1 + E)
max(0, 1 E)
Brier Exponential
14.E e
.
14.E
B
erator to reconstruct images from pure Gaussian noise. Specifically, we minimize the discrepancy

---

9.0
8.0
-8 HD
9.3
FID
9.2
7.0
9.1
9.0
8.8
4.0
8.7
Precision 1%6 Recall 176)
3.0
101214161829
6
4 Training Steps (x10*) Training Steps (x10*)
3
8
Training Steps (x10*)
Training Steps (x10*)
(a) Cifar-10
(b) ImageNet
Figure 2: Results on Cifar- 10 and ImageNet of different training steps, where CD [52 requires
800,000 steps reach FID 3.55 on CIFAR-10 and 600,000 steps to get FID 6.20 on ImageNet.
with the auxiliary energy function Ezz). obtaining an intermediate model 0', which subsequently provides negative samples for the final training objective (Line 3). To further enhance efficiency, we construct a dataset consisting of triplets (x, x', where each noise vector z corresponds to positive samples from the teacher diffusion model and negative samples x' from the warm-up generator Ga before training. This dataset with (x, x', z) is then used to optimize G2 according to Equation (14
5
Experiment
5.1 Experimental Setup
Datasets and Models. To thoroughly evaluate CED, we two representative benchmark datasets from prior works: CIFAR-10 (32x32) for unconditional generation (28) and ImageNet (64x64) for conditional generation 8 To distill the teacher diffusion models, employ EDM models (22). which are widely used in distillation-based methods.
Evaluation Protocol. We measure image generation quality using the Frechet Inception Distance (FID) and the Inception Score (IS) [38]. For FID, we generate 50k samples and compare them against the training set used by the EDM teacher model as the reference. We also consider Precision and Recall when evaluating conditional generation on ImageNet 64x64, where we follow previous works with predefined reference batch compute both metrics 9 52
Baselines. We assess the effectiveness of one-step distillation methods and other efficient diffusion models without directly utilizing the original image dataset, instead focusing comparing the quality of images generated by CED. Based on the models employed during student model training, we categorize the baseline methods into the following groups: (1) Diffusion Models, which include methods aimed at accelerating sampling speed as well as original diffusion models; (2) (Generative Adversarial Networks) GANs with extra discriminator models; (3) Distillation with Teacher & Fake Score Models, which utilize both Teacher and Auxiliary (Fake) Score Models; (4) Distillation with Teacher Models; and (5) Distillation with Student Models Only. Note that these categorizations based specifically on the models involved during the training of student generators.
Implementation Details. We implement CED on top of the EDM codebase [22). initializing the one-step generator with Oune from the teacher diffusion EDM model. To reduce computational overhead in Equation (14) (see Algorithm[] D we randomly sample negative samples for only 25% of the batch compared to the positive samples- to optimize the objective effectively. Moreover, compute the energy function using the LPIPS- distance [9], which balances perceptual similarity and robustness to outliers. As outlined in Algorithm | we first sample from the teacher
model generate training data. Specifically, use 35 sampling steps for the Cifar-10 10 and 79 steps for ImageNet with Heun's ' 2nd-order method as shown Line of Algorithm
5.2 Image Generation
We present the image generation results comparing various baseline methods on the CIFAR-10 10 and ImageNet datasets in Tables 2 and respectively. Specifically, we include two variants of our approach: CED-DRE and CED. The key difference between these variants is whether we load the previously obtained warm-up student model parameters 6g during training. From the density ratio estimation perspective, CED-DRE calculates the energy as (x, Ga(z)) -[Ix- (z()))) Go (2) where we replace the traditional 1.2 distance with the LPIPS-Huber distance. Firstly, we observe that preserving the intermediate model parameters l' does not significantly impact the final performance. Therefore, to further save memory, can safely omit storing these parameters. Instead,
7

---

Table 2: Results of unconditional image gen-
Table Results of conditional image generation on
eration on CIFAR-10 with FID and IS.
ImageNet 64 x 64 with FID, Precision and Recall.
METHOD
NFF ()) FID (.) IS (t)
METHOD
NFE (1) FID ())
Prec. (1) Rec. (f)
Diffusion models
Diffusion models
Score SDE [3
2000
2.38
9.83
DDIM [50]
13.7
0.65
0.56
DDPM
18
1000
3.17
9.46
10
18.3
0.60
0.49
LSGM 54
147
2.10
DPM solver
33
10
7.93
FDM
22
35
1.97
3.42
GANs
DEIS 66
10
6,65
BigGAN
14.7
9.22
20
3.10
StyleGAN2 24
8.32
9.21
DDPM
11.0
0.67
StyleGAN2-ADA
23
2.92
9.83
iDDPM 38
250
2.92
0.74
0.62
Distillation with Teacher & Fake Score Models
Diff-Instruct [35
4.53
9.89
ADM 9
250
2.07
0.74
0.63
DMD
3.77
EDM
2.30
SID 70
1.92
9.98
GANs
Distillation with Teacher Models
BigGAN- deep
4.06
0.79
0.48
Knowledge Distillation [3
9.36
Distillation with Teacher & Fake Score Models
DENO (LPIPS) [68]
3.78
33
5.57
TRACT
3.78
DMD 665
2.62
PD |6
9.12
SID 70
1.52
0.74
0.63
CD (LPIPS)
52
3.55
9.48
Distillation with Teacher Models
CTM w/o GAN
25
5.19
TRACT
7,43
1-rectified flow (+distill)
31
6.18
9.08
BOOT 15
16.3
0.68
0.36
2-rectified flow
31
12.21
8.08
PD 46
15.39
0.59
0.62
+distill RD
4.85
9.01
3-rectified flow
31
8.15
8.47
CD (LPIPS)
52
6.20
0.68
0.6
+Distill
8.79
CTM w/o GAN
25
Distillation with Student Models Only
Distillation with Student Models Only
CED-DRE
2.95
9.36
CED-DRE
3.88
0.72
0.60
CED
2.96
9.42
CED
4.06
0.74
0.58
we pre-sample negative examples and directly load the current student model itself during training, as outlined in Algorithm [ Furthermore, results consistently demonstrate that CED outperforms baseline methods using only teacher models like Consistency Models (CM) and Flow Matching (FM), while significantly reducing memory consumption of loading models by approximately 50% during training. Also, our method CED can outperform DMD which includes iterative training with extra models CIFAR-10 datasets. Although method does surpass approaches involving both teacher and auxiliary (fake) score models with iterative training like DMD) and SiD on ImageNet, simplifies the training pipeline by eliminating the necessity of loading and maintaining three models simultancously, thus achieving a 66% reduction in memory usage of loading models. Also, unlike these methods-which require iterative updates and thus demand careful hyperparameter tuning for multiple models-CED avoids extra hyperparameter search, streamlining the overall training process.
Moreover, evaluate CED different training steps CIFAR-10 and ImageNet, as shown in Figure 2 Remarkably, CED achieves strong performance with just 100,000 training steps on CIFAR- 10 (FID 3.53) and 40,000 steps ImageNet (FID 5.31). In comparison, CD requires 800,000 steps reach a similar FID of 3.55 on CIFAR-10 and 600,000 steps to achieve an FID of 6.20 on ImageNet, while also requiring the loading of additional models during training (based on their official code implementation). These results further demonstrate the efficiency of our proposed algorithm, which achieves good performance with relatively few training steps. Notably, CED does not incur additional training overhead. This efficiency likely stems from the fact that CED avoids the need for extra timestep sampling like previous distillation methods (e.g., CD) during training, thereby simplifying the process and potentially enabling faster convergence.
5.3 Analysis and Ablation Study
Table 4: Results Cifar- 10 with various distance functions for the energy E.).
In this section, we conduct experiments to analyze the impact of various distance metrics used in the energy func- tion and different loss formulations together with analysis of generated imge quality, as summarized in Table I
Method
FID (L)
IS (t)
L2
4.07
9.23
L1
3.78
9.23
Ablation Study with Different Distance Functions. We investigate the impact of different distance functions
Pseudo-Huber
3.04
9.34
LPIPS Hubbr
2.96
9.42

---

(c) CED (our) Figure 4: Case Study of generated images from different models trained on ImageNet.
(a) EDM
(b) SiD
model training. Specifically, we modify the energy function Exx, Ga(z)) in Equation 14) by sub- stituting various loss functions, including L2 distance, L1 distance, Pseudo-Hube [6] 51, and LPIPS-Huber [29]. We adopt the logistic loss formulation as described Table ! and the corre- sponding results on CIFAR-10 are reported in Table The Pseudo-Huber distance function, known for being less sensitive to outliers than the squared L2 distance, can potentially reduce gradient variance during training. Our results show that it consistently outperforms the standard L2 distance on Cifar-10, indicating its effectiveness. We also explore the LPIPS- Huber distance function, which encourages the model to minimize perceptual differences between generated samples and the ground truth, as demonstrated in prior works on generative modeling [29]. Among the tested metrics, the Seeudo-Huber distance function significantly outperforms L2 and other alternatives, summarized in Table 4 Therefore, we select the LPIPS-Huber distance as the primary choice for our model.
3.3
9.5
Ablation Study with Different Loss Functions.
Hinge
Exp
We investigate the impact of different loss function formulations the final objective in Equation [4] The specific formats of these loss functions are presented in Table [ Our results show that both the Logistic and Hinge loss formulations achieve the best performance compared to other alterna- tives, making them strong candidates for practical for training one-step diffusion model. Ad- ditionally, in our implementation, we observe that Hinge loss leads to faster convergence (i.e., fewer training steps) compared to Logistic loss, making it a particularly attractive choice when aiming for competitive results with reduced training time.
3.2
Brier
Log
9.4
3 3.1
IS (
3.0
9.3
2.9
2.8
9.2
FID
IS
Figure 3: Results on CIFAR- 10 with different loss functions, as shown in Table "Exp" de- notes Exponential and "Log" denotes Logistic.
Analysis of Generated Image Quality. We further generate images using different models, including EDM (our teacher model), SiD (a state-of-ther tfrte-ar one-step generator) and our proposed method (CED). We selected representative images from each model for qualitative comparison. Although model (CED) does not outperform SiD in terms of FID scores, it achieves comparable image quality. as illustrated in Figure 4 Moreover, the quality of images generated by CED are also comparable those produced by the teacher model, further demonstrating the effectiveness of our approach to generate high quality images. Notebly, CED requires only one third of the memory load models during training, making it particularly attractive for scaling to larger models with more parameters. More examples CED-generated images are included in the Appendix C
Conclusion
In conclusion, we introduce CED, a simple and efficient distillation algorithm leveraging noise contrastive estimation distill multi-step diffusion models into a single-step generator. Our method implicitly optimizes the KL divergence commonly used for distribution distribution-matching distillation, pro- viding theoretical justification for its effectiveness. Empirically, CED outperforms existing one-step training methods that avoid iterative procedures but still require loading the teacher model during training, incurring additional memory overhead. contrast, CED significantly simplifies the pipeline for one-step image generation while achieving superior performance. We believe this work paves the way for a new class of one-step distillation techniques and holds strong potential for extension to larger-scale tasks, such as eext-to-image generation with high-capacity models.

---

Acknowledge
This work was supported in part by grants from the National Science Foundation (2226025) and the National Center for Advancing Translational Sciences and the National Institutes of Health (UL1 TR002014) and by the Center for Artificial Intelligence Foundations and Scientific Applications and the Institute for Computational and Data Sciences at Pennsylvania State University.
References
[1] David Berthelot, Arnaud Autef. Jierui Lin, Dian Ang Yap. Shuangfei Zhai, Siyuan Hu, Daniel Zheng, Walter Talbott, and Eric Gu. Tract: Denoising diffusion models with transitive closure dimeddistillation arXiv preprint 23X1v:23033042465 2023.
[2] Steffen Bickel, Michael Brückner, and Tobias Scheffer. Discriminative learning under covariate shift. Journal of Machine Learning Research, 10(9), 2009.
[3] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-355 1952.
[4] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 2018.
[5] Andreas Buja, Werner Stuetzle, and Yi Shen. Loss functions for binary class probability estimation and classification: Structure and applications. Working draft, November, 3:13, 2005.
[6] Pierre Charbonnier, Laure Blanc-Féraud, Gilles Aubert, and Michel Barlaud. Deterministic edge-preserving regularization in computed imaging. IEEE Transactions on image processing, 6(2):298-311, 1997.
[7] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20:273-297, 1995.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[9] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[10] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guider video synthesis with diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7346-7356, 2023.
[11] Yves Franceschi, Mike Gartrell, Ludovic Dos Santos, Thibaut Issenhuth, Emmanuel de Bézenac, Mickaël Chen, and Alain Rakotomamonjy. Unifying gans and score-based diffusion generative particle models. Advances in Neural Information Processing Systems, 36:59729- 59760, 2023.
[12] Yoav Freund and Robert Schapire. A theeoetii generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 55(1):1 19-139, 1997.
[13] Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):35--3777 2007.
[14] Ian J Goodfellow, Jean Pouget-Abes Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[15] Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, and Joshua M Susskind. Boot: Data-free distillation of denoising diffusion models with bootstrapping. In ICMI 2023 Workshop on Structured Probabilistic Inference & Generative Modeling, 2023.
10

---

[16] Michael Gutmann and Aapo Hyvärinen. coise-contrastive estimation: A new estimation principle for unnormalized statistical models98. In Proceedings of the thirteenth interna- lional conference on artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings, 2010.
[17] Jonathan Heek, Emiel Hoogeboom, and Tim Salimans. Multistep consistency models. arXiv preprint arXiv:2403.068800 2024.
[18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[19] Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation 3d. In International conference on machine learning, pages 8867-8887. PMLR, 2022.
[20] Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato. Richard E Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In International Conference Machine Learning, pages 1645-1654. PMLR, 2017.
[21] Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Shane Gu, and Rosalind Picard. Human-centric dialog training via offline reinforcement learning. arXiv preprint arXiv:2010.0588 2020.
[22] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 01126565-265777 2022.
[23] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in neural information processing systems, 33:12104-12111 121144 2020.
[24] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 8110-8119, 2020.
[25] Dongjun Kim, Chieh Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279, 2023.
[26] Diederik Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013.
[27] Zhifeng Kong, Wei Ping. Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09661, 2020.
(28) Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[29] Sangyun Lee, Zinan Lin, and Giulia Fanti. Improving the training of rectified flows. Advances in Neural Information Processing Systems, 37:63082- 6009, 2024.
[30] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow 2022.
matching for generative modeling. arXiv preprint arXiv:2210.00
[31] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300 2022.
[32] Xingchao Liu. Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. In The Twelfth International Conference on Learning Representations, 2023.
[33] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.

---

[34] Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02.88, 2021.
[35] Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff- instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36:76525-76546, 2023.
[36] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14297-14306, 2023.
[37] Thuan Hoang Nguyen and Anh Tran. Swiftbrush: One-step text-to-imaae diffusion model with variational score distillation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7807-7816, 2024.
[38] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162-8171. PMLR, 2021.
[39] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[40] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using
2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
[41] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728-533 -57441, 2023.
[42] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 265/5 1(2):3, 2022.
[43] Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, and Xuefeng Xiao. Hyper-sd: Trajectory segmented consistency model for efficient image synthesis, arXiv preprint arXiv:2404. 13686, 2024.
[44] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[45] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36494, 2022.
[46] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00511 2022.
[47] Tim Salimans, Thomas Mensink, Jonathan Heek, and Emiel Hoogeboom. Multistep distillation of diffusion models via moment matching. Advances in Neural Information Processing Systems, 37:36046-36070. 2024.
[48] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
[49] Jaseha Cock-Dic,stein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-22662222 2265. pmlr, 2015.
[50] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint 2011:22010 2020.
12

---

[51] Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint arXiv:2310.14188 2023.
[52] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 2023.
[53] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 13456, 2020.
[54] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. 2021. URL https://arxiv. org/abs/2106.0520 2021.
[55] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12619-12629)
2023.
[56] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36:8406-8441, 2023.
[57] Teng Xiao, Mingxiao Li, Yige Yuan, Huaisheng Zhu, Chao Cui, and Vasant G Honavar. How to leverage demonstration data in alignment for large language model? a self-imitation learning perspective. arXiv preprint arXiv:2410.
2024.
[58] Teng Xiao, Yige Yuan, Zhengyu Chen, Mingxiao Li, Shangsong Liang, Zhaochun Ren, and Vasant G Honavar. Simper: A minimalist approach to preference alignment without hyperpa- rameters. arXiv preprint arXiv:2502.00883 2025.
[59] Teng Xiao, Yige Yuan, Mingxiao Li, Zhengyu Chen, and Vasant G Honavar. On connection between imitation learning and rlhf. arXiv preprint arXiv:2503.05077 2025.
[60] Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, and Vasant Honavar. Cal-dpo: Calibrated direct preference optimization for language model alignment. Advances in Neural Information Processing Systems, 37:1 4429--144320 2024.
[61] Sirui Xic, Zhisheng Xiao, Diederik P Kingma, Tingbo Hou, Ying Nian Wu, Kevin Patrick Murphy, Tim Salimans, Ben Poole, and Ruiqi Gao. Em distillation for one-step diffusion models, 2024. URL https://arxiv. org/abs/2405.1em 16852.
[62] Minkai Xu, Alexander S Powers, Ron O Dror, Stefano Ermon, and Jure Leskovec. Geometric latent diffusion models for 3d molecule generation. In International Conference on Machine Learning, pages 13592-38610. PMLR, 2023.
[63] Hanshu Yan, Xingchao Liu, Jiachun Pan, Jun Hao Liew, Qiang Liu, and Jiashi Feng. Perflow: Piecewise rectified flow as universal plug-and-play accelerator. arXiv preprint arXi::2405.075 10, 2024.
[64] Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and Bill Freeman. Improved distribution matching distillation for fast image synthesis. Advances in neural information processing systems, 17:47455-47487 2024.
[65] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2024.
[66] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204. 13902, 2022.
[67] Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: A unified predictor- corrector framework for fast sampling of diffusion models. Advances Neural Information Processing Systems, 36:49842-449990 49869, 2023.
13

---

[68] Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. In International conference on machine learning, pages 42320-42400. PMLR, 2023.
[69] Mingyuan Zhou, Huangjie Zheng, Yi Gu, Zhendong Wang, and Hai Huang. Adversarial score identity distillation: Rapidly surpassing the teacher in one step. arXiv preprint arXiv:2410.14919, 2024.
[70] Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In Forty-first International Conference on Machine Learning, 2024.
14

---

A Omitted Derivation Details and Background
A.1 Derivation of Equation 10
We derive Equation ② following 41 57 We first consider the Distribution Matching Distillation
objective in Equation from the reinforcement learning perspective:
max 9
1 r(x, 2) _L
|Pake x
2) (x 2)
(15)
where we introduce a hyperparameter 39 to control the weight of KL divergence. The reward function
r(x, z) = log(Preal 2) Piake (x 2)) and 0o the parameter the reference model. Then, we
have:
max E 200.)) [x(x,)) 3DKL
(Piste (x
Zz) Pake (x z))
Plake (x
max N10.1
r(x, 2)
-llog
6o Pfako
(x
z)
Plak (x (2)
1
(16)
min E. (0.)( Eya
log
rxx, 2)
Place
Ou Lake
(x (2)
B
Pfako (x zz)
min (o.
log
log Z(z)
@n 7(z) Plake
(x 2) exp (3r(x,2)
where Z(2) = Ex Txof(x z) exp
(xx x)).
Then, we can obtain:
Plake (x z)
min En 0
log
log Z(z)
90 Z(z) [(z) Pake (x 2.) exp (x)(x)
Plake ( 2)
(17)
min (2 Bp
log pia (x z))
log ((2)
min Ez [DK..
Phate (x 2) 1z)llffate (x 2)) log [(z)
where plate (x
2) Z (x 2) exp (x((x We find that the previous equation to optimize the KL divergence and obtain the optimal parameter as follows:
(x z) exp (1)(
Plake (x
z
Piake (x z)
Ao Pfake
(18)
Z(z)
A.2 Direct Alignment Algorithm
We follow the represenative direct alignment algorithm Typically, fine-tuming Large Language Models (LLMMs) via reinforcement learning first requires training reward model using human-preferred data pairs, denoted (x.,, xi), representing human- preferede and human- dimaa--dippeeeered images in setting. The reward model can be trained on image pairs with Bradley-Terry (BT) models 3
exp (r (xw; z))
p (x
x
z)
(19)
exp (r (Xw, z)) + exp (r (xi, z))
We can obtain the reward model based on the optimal policy as follows:
(x. 2)
r(x, x) 3(og
log (z)))
(20)
6o Prake (x, z)
Then, get the BT model with the format of optimal policy:
p" (x.: Xi 2)
(21)
Pla (x:1z) Pun (xi12)
20 (dh.
1 + exp
B log
log P0
(De. 22)
15

---

Figure 5: Images generated by one-step diffusion models trained with CED on ImageNet.
To avoid the complexity of two-step training, 41
introduces direct optimization approach, replacing the optimal parameter H with trainable parameters 6. This strategy integrates the traditional two-stage process- first training reward models and subsequently training reinforcement learning models using these reward models into unified, single-step training pipeline:
min log (Xw > Xi z))
min log
(22)
1 + exp
Blog d0
Pla (xi12)
B log (x (p))
Pogi (Xew d0
Piate (x]
Therefore, our approach, viewed from the perspective of Density Ratio Estimation, shares concep- tual similarities with direct alignment algorithms, as both leverage related techniques to unify the conventional two-step training process into a single step.
B Experiment Details
B.1 Implementation Details
We present implementation and setup details of CED in this section. For experiments, Adam optimizer with an effective batch size of 512 for CIFAR-10 and 1024 for ImageNet. Training conducted on 2 NVIDIA A100 GPUs. We train fixed square resolutions and use learning rate 3e-5. Moreover, we perform I 10000 steps warm-up training as demonstrated in Line of Algorithm Our code of CED is based the implementation EDM 22
the
C Additional Experiments
In this section, we present more results generated from CED on Cifar-10 and ImageNet as shown in which further shows the effectiveness of our proposed method.
Figure 5 and Figure 6
16

---

Figure 6: Images generated by ste-step diffusion models trained with CED on Cifar-10.
D Limitations
This work introduces an algorithm for efficiently distilling knowledge from multi-step diffusion models into one-step diffusion models for image generation tasks. However, the current focus is limited to image generation, while other domains-Suun as scientific discovery and 3D reconstruction, which could also benefit from fast diffusion models to enhance the efficiency of their generative pipelines.
E Broader Impact
This work can be used to accelerate both image generation and text-to-image generation, enhancing the user experience in text-to-image systems. Furthermore, it can be extended to other domains for efficient high-quality sample generation, such as 3D reconstruction, audio synthesis, and scientific discovery.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide the main claims in the abstract and introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work are put into Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
Our paper does not include concretely theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the experimental results Section 5 and code in supplementary materials to reproduce our results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
We provide code and data in supplementary materials with sufficient instruc-
tions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips //nips.cc/pil cc/public/guu
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the experimental settings in section and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We have included statistical signifance in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have included computing resources in Appendix B.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our code conforms with the NeulPS code Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have included broader impacts in Appendix E
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have discussed this in Section 5
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
23

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
24


Reviews:
[{"review_id": "ZNv5DpXryL", "reviewer": "Reviewer_deet", "summary": "The paper introduces Contrastive Energy Distillation (CED), which uses a joint energy-based model and Noise Contrastive Estimation to distill multi-step diffusion models into efficient one-step generators. CED eliminates auxiliary models and iterative training, reducing memory usage by 1/2 while achieving competitive image quality and requiring fewer training steps than existing approaches.", "questions": "1. How does CED perform against recent non-distillation approaches? The authors are suggested to include direct comparisons with at least 2-3 non-distillation one-step methods on the same datasets. Clearly articulate when distillation is preferable.\n2. How does CED scale to high-resolution images (e.g., 512×512, 1024×1024) and complex datasets beyond CIFAR-10/ImageNet 64×64? Are the memory advantages maintained at scale? The authors should provide experiments on at least one high-resolution dataset (ImageNet 256×256 or 512×512) with memory usage analysis. \n3. The connection to distribution matching distillation (Section 4.3) is valuable, but is the theoretical contribution sufficiently novel beyond applying existing NCE frameworks to this setting? The authors should clarify the novel theoretical challenges specific to diffusion distillation that necessitate the proposed energy-based formulation.", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths：\n1. Clear presentation: Well-structured with clear motivation, problem statement, and generally clear mathematical exposition.\n2. Theoretical foundation: Provides a principled connection between CED and distribution matching distillation, offering theoretical justification for the approach.\n3. Broad applicability: The energy-based formulation could potentially extend to other generative modeling domains beyond images.\n\nWeaknesses：\n1. Incremental contribution: Primarily combines existing techniques (EBMs, NCE), limiting theoretical novelty.\n2. While the paper provides thorough comparisons within distillation methods, the evaluation scope is unnecessarily narrow. Recent work has demonstrated that non-distillation approaches can achieve competitive one-step generation quality, such as shortcut models [Frans et al., 2024]. The paper lacks a clear articulation of distillation's specific scenario constraints (no access to original training data, goal of accelerating a specific pre-trained teacher), and fails to provide broader context by comparing against these alternative paradigms in evaluation. \n3. Dataset scope: Evaluations are limited to relatively simple datasets (CIFAR-10, ImageNet 64x64); unclear how it scales to high-resolution or complex domains.\n4. Modest performance gains: While memory-efficient, quality improvements are incremental compared to the computational savings achieved.\n\n[Frans et al. 2024] \"One Step Diffusion via Shortcut Models\" ICLR 2025.", "was_revised": true, "final_justification": "I recognize the contributions of this paper, but I still feel that the restricted evaluation somewhat limits the paper's quality, particularly in its lack of comprehensive evaluation on larger image resolutions and a comprehensive comparison with non-distillation approaches. Given these considerations, I raised my score, but still with some reservations.", "rebuttal": "We gratefully appreciate your time in reviewing our paper. We would like to clarify some misunderstandings regarding our approach.\n\n**Q1. Incremental contribution: Primarily combines existing techniques (EBMs, NCE), limiting theoretical novelty.**\n\n**A1.** Thanks for your comments! We believe that the Reviewer has important misunderstandings of our method. **Our method is not merely a straightforward combination of EBMs and NCE** and the core contribution and novelty lie in simplifying the training pipeline for distilling multi-step diffusion models into a single-step generator. Specifically, previous methods derive their loss from distribution matching [5] or score identity distillation [6]. However, these methods can’t avoid computing extra (or \"fake\") score models and optimizing the distillation process through iterative updates, which needs to load multiple models and significantly increase training complexity and resource demands. Our method simplifies the distillation process by rethinking it through the perspective of a novel joint energy-based model to connect fake score models, teacher models and one-step generators. By leveraging NCE alongside an approximation of the original loss, we derive an objective that avoids the need for additional models or iterative optimization. **We believe that our formulation of a joint energy-based model, combined with the adaptation of NCE to eliminate reliance on fake score models, is a novel contribution for simplifying the distillation of one-step generators.**\n\nMoreover, **we offer a novel and distinct perspective for understanding the loss as discussed in Section 4.3.** Notably, the derivation in Section 4.3 does not rely on any prior assumptions about the joint energy-based model. Instead, we derive a similar loss from the perspective of distribution matching-based distillation. Interestingly, the loss obtained from this distribution matching view shares a similar form with the one derived using our proposed joint EBM and NCE framework. **This unified perspective not only provides novel theoretical insight but also helps explain the effectiveness of our method by establishing a clear connection to existing distribution matching-based distillation approaches.**\n\nIn summary, our method is not a simple combination of EBMs and NCE. **The core novelty lies in our analysis, which reformulates distillation-based methods as a joint energy-based model.** This perspective allows us to simplify the entire training pipeline and establish a clear connection between our proposed algorithm and distribution matching losses—commonly used in distillation—providing a deeper understanding of why our method is effective. As a result, our approach introduces a novel and simple training framework that offers new insights for designing distillation algorithms, enabling efficient one-step generators distilled from multi-step teacher diffusion models, while avoiding the complexity of prior methods that require iterative procedures and multiple components.\n\n**Q2. The evaluation scope is unnecessarily narrow. The paper lacks a clear articulation of distillation's specific scenario constraints.**\n\n**A2.** Thanks for your comments and suggestions! Our method is based on a distillation technique that does not rely on high-quality real data, which is a key focus of this paper. We aim to simplify the distillation pipeline specifically in settings where real data is not available. We will further clarify this point in the revised version, particularly in the experimental setup. We believe our method addresses a different scenario compared to works like [3], which focus on training with real data. To further verify the effectiveness of our approach, we also compare our method with some baselines trained with real data in A5. \n\n\n**Q3. Dataset scope: Evaluations are limited to relatively simple datasets (CIFAR-10, ImageNet 64x64); unclear how it scales to high-resolution or complex domains.**\n\n**A3.** Thanks for your suggestions! Due to the limit of time, it’s hard to realize full dataset training on these large datasets with our limited computational resources. To verify the effectiveness of our approach on complex tasks, **we conduct an experiment on the fine-tuning one step generator SDXL-DMD2 [4] to generate high-resolution images at 1024×1024 resolution** and following the evaluation method in Diffusion-DPO [2]. Specifically, we fine-tune SDXL-DMD2 using our proposed loss function on the Pick-a-Pic v2 dataset and evaluate its performance based on win rates across several reward models, including ImageReward, Aesthetic Score, and Pick Score. The win rate measures how often the fine-tuned model outperforms the original model based on different reward models, with a performance above 50% indicating an improvement. All evaluations are conducted on the test split of the Pick-a-Pic v2 dataset. Here are the following results **corresponding to our current optimization steps (the training is not converged)**:\n\n| Reward Model       | Pick Score | Aesthetic Score | ImageReward |\n|--------------|---------------|---------------------|---------------------|\n| CED         |   58.60  |           56.20          |  55.60  |\n\nThis experiment demonstrates both the effectiveness and efficiency of our approach, as we successfully fine-tune the model using a single A100 GPU. Achieving a win rate above 50% indicates that our method consistently improves image quality in the complex scenario like text-to-image generation. In contrast, the DMD2 [4] framework—which focuses on training text-to-image diffusion models for one-step generation using real data—requires significantly more resources, including 8 compute nodes and multiple model downloads, as shown in their Github Repo. We also attempted to fine-tune SDXL-DMD2 on a single node with their loss and found the training process to be extremely slow. **Given the current optimization steps of DMD2 loss, we observed a PickScore win rate of only 52.80%, with aesthetic and image reward scores falling below 50%.** These results indicate that the DMD2 framework is challenging to train during the fine-tuning stage, particularly due to the difficulty of adjusting hyperparameters across multiple components. These results highlight the efficiency and potential of our method, particularly for complex scenarios such as text-to-image generation.\n\n**Q4. Modest performance gains: While memory-efficient, quality improvements are incremental compared to the computational savings achieved.**\n\n**A4.** Thanks for your comments! Our method outperforms previous distillation approaches that require loading teacher diffusion models during training and converges faster than methods such as consistency model distillation, as discussed in Section 5.2 and illustrated in Figure 2. This demonstrates both the effectiveness and efficiency of our approach.\n\n**Q5. How does CED perform against recent non-distillation approaches? The authors are suggested to include direct comparisons with at least 2-3 non-distillation one-step methods on the same datasets. Clearly articulate when distillation is preferable.**\n\n**A5.** Thanks for your suggestions! To address when distillation is preferable: if high-quality image data is not available, current distillation techniques are particularly suitable, as they are typically performed in settings that do not require real image data. Moreover, we compare our method with the latest approach, IMM, accepted at ICML 2025. It is important to note that this method is trained on real image data, unlike our setting, which does not rely on real data and here are corresponding results (results are from IMM) for one-step generation:\n\n|    Cifar10   | FID |\n|--------------|---------------|\n| ICT  |   2.83      |       \n| ECT         |    3.60  |  \n| sCT         |    2.97  |          \n| IMM         |   3.20  | \n|      CED-DRE    |      2.97         |\n\nAs shown in the table, our model achieves the second-best performance and has results comparable to the best-performing method. **Note that our method doesn’t rely on real data or loading additional teacher models during training.** This further demonstrates the effectiveness and potential of our approach for distilling one-step generators from multi-step diffusion models.\n\n**Q6. The connection to distribution matching distillation (Section 4.3) is valuable, but is the theoretical contribution sufficiently novel beyond applying existing NCE frameworks to this setting?**\n\n**A6.** Thanks for your comments! We think that the authors may have some misunderstanding of our method and analysis. We propose a joint energy-based model to capture the relationships among fake score models, teacher models, and one-step generators, and we train it using NCE to simplify the training pipeline—eliminating the need for iterative optimization and additional models. Furthermore, we provide a novel analysis that bridges our energy-based formulation through the perspective of distribution matching distillation. Specifically, we derive a loss from the distribution matching perspective that shares a similar form with our proposed energy-based method, offering deeper insight into why our approach is effective for distilling one-step generators. **This analysis is particularly important and is specific to our energy-based formulation, as it shows that our method implicitly optimizes a distribution matching objective—an approach that has been widely validated in prior work but typically involves a much more complex training process.**\n\n\n[1] Inductive Moment Matching.\n\n[2] Diffusion Model Alignment Using Direct Preference Optimization.\n\n[3] One Step Diffusion via Shortcut Models\n\n[4] Improved Distribution Matching Distillation for Fast Image Synthesis\n\n[5] One-step Diffusion with Distribution Matching Distillation\n\n[6] Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation"}, {"review_id": "9FeWFmvM6w", "reviewer": "Reviewer_e2GZ", "summary": "This paper proposes Contrastive Energy Distillation (CED), a simple method for distilling one-step diffusion models. CED removes the need for introducing fake and real score models during the iterative training process by introducing a joint energy-based model that connects the one-step generator and fake score function. The method achieves competitive performance with improved memory efficiency compared to existing distillation approaches.", "questions": "1.\tWhat is the distillation efficiency (e.g., convergence speed) compared to methods that use a fake score model? Is there any direct comparison?\n\t2.\tSince the CED training method requires pre-generated data, it would be helpful to see how different methods also requires pre-generated data.\n\t3.\tTo better understand the core idea of the paper: Is it correct to say that previous methods rely on both a fake score model and a real score model to learn an energy metric, whereas CED introduces a joint energy-based framework that explicitly models the energy, allowing the fake and real score models to be removed? Please correct me if I’m misunderstanding.", "limitations": "Discussed in Weakness and Questions.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe paper addresses a practical problem by simplifying the distillation process for diffusion models. The proposed method is simple yet effective, achieving satisfying performance while removing the need for both the fake score model and the real score model during training, which improves memory efficiency. It is also interesting to see that the distillation remains effective despite removing these components.\n\nWeakness:\n1. The paper’s structure, particularly Sections 4.1 to 4.3, is not very reader-friendly and needs significant improvement in writing clarity. For example, the motivation and intuition for introducing the joint energy-based function in Section 4.1 should be explained more clearly. \n2. Additionally, the connection and distinction between DMD and CED remain unclear despite the extensive discussion in Section 4.3.\n3.  It would also be helpful to include more results on distillation efficiency (convergence speed) compared to other methods, as the current paper only provides a CD comparison in Figure 2.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns, and I am inclined to maintain my current score.", "rebuttal": "We sincerely thank the reviewer for the comments and suggestions. Please see our clarifications below:\n\n**Q1. The paper’s structure, particularly Sections 4.1 to 4.3, is not very reader-friendly and needs significant improvement in writing clarity. For example, the motivation and intuition for introducing the joint energy-based function in Section 4.1 should be explained more clearly.**\n\n**A1.** Thanks for your suggestions! Our motivation stems from the training process used in previous distillation-based methods, which typically involve an additional model to first train a fake score model $p_{\\text{fake}, \\phi}$ that approximates the score function of samples generated by the one-step generator $G_\\theta$.  However, the ultimate goal is to match the distribution of samples from the teacher model. To capture the relationships among these three distributions—those of the one-step generator, the fake score model, and the teacher model—we are inspired to formulate the process as a joint energy-based model.\nSpecifically, we calibrate the fake score distribution by modeling the discrepancy between the samples from the one-step generator (which we treat as the energy in our framework) and the samples from the learned fake score model in Equation (4). This is because the learned fake score model cannot accurately capture the target distribution. Therefore, we use the energy function to calibrate the original fake score distribution, enabling it to better align with the distribution of samples generated by the one-step generator $G_\\theta$. Since our final objective is to align with the teacher model, we construct this joint formulation that equals the distribution of teacher models and use NCE to optimize this energy-based model to achieve this goal. In the revised version, we will further clarify our high-level motivation and intuition about modeling the relationships among the three models. And we aim to simplify the training process by eliminating the need for intermediate fake score models, highlighting the core idea of our approach more clearly.\n\n**Q2. Additionally, the connection and distinction between DMD and CED remain unclear despite the extensive discussion in Section 4.3.**\n\n**A2.** Thanks for your suggestions! Before addressing this question, we would like to first clarify that the original DMD approach **requires additional model training and iterative optimization**. In contrast, our proposed CED method **does not involve loading extra models and avoids iterative training altogether**, making it fundamentally different from the original DMD. **Furthermore, we derive a new objective from the persepctive of DMD, as shown in Equation (13), which simplifies the original loss and removes the need for complex training procedures.** Notably, we observe that this objective implicitly optimizes DMD and shares a similar format with CED when a specific form of the energy function is defined. The difference between CED and CED-DRE in Table 2 is based on different formats of energy functions. Specifically, CED is based on $\\boldsymbol{E}\\left(\\mathbf{x}, G_\\theta(\\mathbf{z})\\right)=\\left\\|x-G_\\theta(\\mathbf{z})\\right|^{2}$ and CED-DRE is baed on $\\boldsymbol{E}\\left(\\mathbf{x}, G_\\theta(\\mathbf{z})\\right)=-\\left(\\left\\|\\mathbf{x}-G_\\theta(\\mathbf{z})\\right\\|^{2}-\\left\\|\\mathbf{x}-G_{\\theta^{\\prime}}(\\mathbf{z})\\right\\|^{2}\\right)$ with L2 distance.\n\n**Q3. It would also be helpful to include more results on distillation efficiency (convergence speed) compared to other methods, as the current paper only provides a CD comparison in Figure 2. What is the distillation efficiency (e.g., convergence speed) compared to methods that use a fake score model? Is there any direct comparison?**\n\n**A3.** Thanks for your questions! We conducted an experiment to compare the convergence speed of our method with SiD, a strong baseline that requires training fake score models. **Our results show that CED achieves an FID of 3.85 on CIFAR-10 using only 20 A100 GPU hours, whereas SiD requires 100 A100 GPU hours to reach an FID of 4.0.** This demonstrates that our method converges significantly faster than methods like SiD that rely on training auxiliary score models.\n\n\n**Q4. Since the CED training method requires pre-generated data, it would be helpful to see how different methods also requires pre-generated data.**\n\n**A4.** Thanks for your questions! We use pre-generated data to avoid the need for loading additional models during training and to enable efficient reuse of data without generating new samples at each optimization step. In contrast, consistency model-based distillation techniques require access to the outputs of teacher models, making it necessary to load those models and difficult to rely on pre-generated data. Similarly, methods that involve training auxiliary fake score models also depend on teacher model outputs and require additional model training, which further prevents the use of pre-generated data.\nIn our case, the key advantage lies in the design of our loss function, which does not require the output from teacher models—only samples generated by them. This unique characteristic allows us to effectively use pre-generated data, significantly reducing memory consumption and simplifying the training pipeline.\n\n**Q5. To better understand the core idea of the paper: Is it correct to say that previous methods rely on both a fake score model and a real score model to learn an energy metric, whereas CED introduces a joint energy-based framework that explicitly models the energy, allowing the fake and real score models to be removed? Please correct me if I’m misunderstanding.**\n\n**A5.** Thanks for your question! Previous methods model the fake score model, real score model, and one-step generator as separate components, requiring them to optimize each part individually during DMD training. **In contrast, our approach captures the relationships among these components within a single joint energy-based model and eliminates the need for explicitly modeling or training fake score models.**\nOur connection to prior methods, such as DMD, is established through the analysis presented in Section 4.3. We demonstrate that optimizing our CED objective is equivalent to implicitly optimizing DMD under a specific formulation of the energy function. This unified perspective not only simplifies the training process but also provides a deeper theoretical understanding of how our method relates to and improves upon previous approaches."}, {"review_id": "JAzpp7GNmp", "reviewer": "Reviewer_SDf9", "summary": "This paper introduces Contrastive Energy Distillation (CED), a simple and efficient method to convert slow, multi-step diffusion models into fast, one-step image generators. The key idea is to frame the distillation as a contrastive learning task, where a student model learns to distinguish \"positive\" samples from a teacher model against \"negative\" samples from a previous version of itself. This approach eliminates the need for complex auxiliary models and iterative training, significantly reducing memory usage and training complexity. The authors provide theoretical justification for their method and show through experiments that CED achieves competitive image quality with far greater efficiency than comparable techniques.", "questions": "See Weaknesses.", "limitations": "See Weaknesses.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The core idea of using a contrastive Energy objective to traina  one-step generator is interesting.\n- The paper provides a compelling theoretical connection between the proposed CED objective and KL divergence.\n- Experiments show competitive performance.\n\nWeaknesses:\n- This paper seems to overclaim multiple points. In multiple places, the paper claims CED is a distillation algorithm with the property of Student Models Only. However, it still requires the teacher to generate noise-image pairs, which is expensive and not student-only. Besides, CED does not only use 1/3 or 1/2 memory compared to score distillation and consistency distillation. The memory cost of distillation includes model weights, optimizer state, and so on.\n-  The performance gap between SiD and CED is still notably large.\n- Lack an important variant: remove the negative phase in the CED objective, which is closely related to InstaFlow [1].\n- Lack of comparison to some recent SOTA with one trainable model, e.g., IMM [2] and shortcut model [3].\n\nMinor:\n- The title of Section 4.3 would be clearer and better if it were changed to 'Relation to Reverse KL Divergence'. Specifically, the usage of reverse KL in diffusion distillation is first proposed in VSD and Diff-Instruct. It is somewhat inappropriate to use DMD to represent reverse KL divergence.\n\n[1] InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation, ICLR 2023.\n\n[2] Inductive Moment Matching, ICML 2025.\n\n[3] One Step Diffusion via Shortcut Models, ICLR 2025.", "was_revised": true, "final_justification": "The rebuttal by authors addresses some of my concerns, thus I raise the score to 4, assuming that the author completely eliminates the overclaiming/misleading writing in the paper, especially in Figure 1 and Tables 2-3.", "rebuttal": "We sincerely thank the reviewer for the comments and suggestions. Please see our clarifications below:\n\n**Q1. In multiple places, the paper claims CED is a distillation algorithm with the property of Student Models Only. However, it still requires the teacher to generate noise-image pairs, which is expensive and not student-only.**\n\n**A1.** Thanks for your comments and suggestions! We would like to kindly remind the reviewer that we have demonstrated that **“Note that these categorizations are based specifically on the models involved during the training of student generators.” in line 283.** We do not claim that our method avoids using teacher models entirely; rather, we emphasize that the teacher model is not required during the training process. We will further clarify this in the revised paper.\n\nMoreover, the time cost for pre-sampling before training is significantly lower than the overall training cost. **For example, CED achieves an FID of 3.85 on CIFAR-10 using only 20 A100 GPU hours, while SiD requires 100 A100 GPU hours to reach an FID of 4.0. In our case, generating the pre-sampled data for CIFAR-10 takes only about 2–3 A100 GPU hours, which is negligible compared to the large training time gap between CED and SiD.** The key advantage of pre-sampling is that it eliminates the need to generate samples during each training step. Instead, we can reuse the pre-sampled data throughout training, which significantly improves overall efficiency.\n\n**Q2. Besides, CED does not only use 1/3 or 1/2 memory compared to score distillation and consistency distillation. The memory cost of distillation includes model weights, optimizer state, and so on.**\n\n**A2.** Thanks for your comments and suggestions! We think that the memory savings—specifically the reduction of 1/3 to 1/2 in model loading memory—are mentioned in Section 5.2. For example, line 304 states: “significantly reducing memory consumption of loading models by approximately 50% during training,” and line 309 notes: “thus achieving a 66% reduction in memory usage of loading models.” These statements highlight our focus on reducing memory usage related to model loading. We will further clarify this in the revised version. Moreover, we acknowledge that the description in Figure 1 may have caused some confusion. We apologize for this and will clarify the figure and explicitly emphasize the 1/3 to 1/2 memory savings in model loading more clearly in the revised version. Additionally, methods like DMD and SiD require iterative training across multiple models, which can lead to increased memory usage due to the need to store optimizer states. In this paper, we focus on the practical costs associated with model loading and the complexity of training pipelines, which often introduce more hyperparameters and result in slower convergence rates. **Moreover, we believe the reviewer should not overlook our contribution to improving the training efficiency of current baselines. Unlike consistency-based distillation techniques, our method does not require access to the teacher model's outputs.** Additionally, **we avoid the need to optimize auxiliary fake score models**, which would otherwise require extra memory and additional optimization steps. We propose a new perspective for designing a simple and efficient distillation technique for one-step generative models distilled from multi-step diffusion models—without relying on a complex training framework.\n\n\n**Q3. The performance gap between SiD and CED is still notably large.**\n\n**A3.** Thanks for your comments and suggestions! We would like to clarify that our primary contribution is not to propose a state-of-the-art method for distilling multi-step diffusion models into one-step generators without real data. Rather, our goal is to introduce a new perspective that challenges the existing complex training frameworks with extra models and iterative optimization. We believe our approach offers a simpler alternative that significantly reduces computational cost and training time, while providing fresh insights into the design of efficient distillation techniques. Moreover, as mentioned in our previous response, our method demonstrates significantly lower computational cost—for example, **CED achieves an FID of 3.85 on CIFAR-10 using only 20 A100 GPU hours, whereas SiD requires 100 A100 GPU hours to reach an FID of 4.0.** This highlights the efficiency of our approach and reinforces its potential as a promising alternative distillation training pipeline.\n\n\n**Q4. Lack an important variant: remove the negative phase in the CED objective, which is closely related to InstaFlow [1].**\n\n**A4.** Thanks for your suggestions. We have removed the negative phase in the CED objective and get the following results:\n\n|    Cifar10   | FID |\n|--------------|---------------|     \n| w/o Negative         |   3.99  | \n|      CED    |      **2.97**         |\n\nWe observe that our method significantly outperforms the variant without the negative gradient component, further demonstrating the effectiveness of our approach.\n\n**Q5. Lack of comparison to some recent SOTA with one trainable model, e.g., IMM [2] and shortcut model [3].**\n\n**A5.** Thanks for your suggestions! We would first like to clarify that our method is designed for distillation settings that do not rely on real image data, whereas IMM and shortcut models are trained using real image data. Additionally, since the shortcut models do not report results on CIFAR-10 or ImageNet, we include the results of IMM on CIFAR-10 as reported in their paper:\n\n|    Cifar10   | FID |\n|--------------|---------------|     \n| IMM         |   3.20  | \n|      CED-DRE    |      **2.97**         |\n\nWe observe that our method outperforms IMM, even though IMM is trained on real image data.\n\n**Q6. The title of Section 4.3 would be clearer and better if it were changed to 'Relation to Reverse KL Divergence'.**\n\n**A6.** Thanks for your suggestions! We will modify this in the revised version."}, {"review_id": "xOnMnvxSKn", "reviewer": "Reviewer_x1Sm", "summary": "This paper introduces Contrastive Energy Distillation (CED), a simple and efficient method to distill multi-step diffusion models into one-step generators. By leveraging a joint energy-based model and a contrastive loss inspired by Noise Contrastive Estimation, CED avoids auxiliary models and iterative training. It achieves competitive performance on CIFAR-10 and ImageNet.", "questions": "+ Please report on training time and compare it with other method like DMD, DMDv2, CD, SiD\n+ Author should use same initial noise and same class input for sampling for Figure 4. So the reader can visually compare between methods", "limitations": "No limitations besides the weaknesses I mentioned above", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n+ CED eliminates the need for auxiliary score models and iterative updates, reducing memory usage and simplifying implementation.\n\n+ On CIFAR-10 and ImageNet, CED achieves results close to or better than other one-step distillation methods, with faster convergence and lower resource demands.\n\n+ The paper includes thorough ablation studies on energy functions and loss choices, offering insights into design decisions (e.g., LPIPS-Huber and hinge loss work well).\n\n+ A quick glance in the provided code, I see it quite simple to implement so this is a plus.\n\n**Weaknesses**\n\n+ The method is only evaluated on 32×32 CIFAR-10 and 64×64 ImageNet; more difficult settings like text-to-image diffusion tasks are not explored.\n\n+ If I understand correctly then the method relies the generator from previous iteration $G_{\\theta'}$, which, while less burdensome than training the generator alongside with the auxiliary score model, still introduces an computational cost.  Therefore, I'm not convinced it is actually more efficient than DMD or previous approach as there is no training time comparison, also DMDv2 [1] has introduce some tricks to make it much more efficient.\n\n+ Lack comparison with DMDv2, which I believe also report on same setting 32×32 CIFAR-10 and 64×64 ImageNet.\n\nReference:\n\n[1] Yin et al. Improved Distribution Matching Distillation for Fast Image Synthesis. NeurIPS 2024", "was_revised": true, "final_justification": "The author address my concerns well but I already give a high score with a huge faith in this work so I decide to keep inital score of **5**.", "rebuttal": "We appreciate the reviewer's perception of our contributions to simplifying the training pipeline and being easy to implement, and we thank the reviewer for their insightful questions. Please find our detailed responses below:\n\n**Q1. The method is only evaluated on 32×32 CIFAR-10 and 64×64 ImageNet; more difficult settings like text-to-image diffusion tasks are not explored.**\n\n**A1.** Thanks for your suggestions! Due to limited time and computational resources, it is challenging for us to perform model distillation from scratch for text-to-image diffusion models initialized with multistep teacher diffusion models. Instead, we conduct fine-tuning experiments on SDXL-DMD2 on the text-to-image diffusion tasks. Specifically, we fine-tune SDXL-DMD2 using our proposed loss function on the Pick-a-Pic v2 dataset and evaluate its performance based on win rates across several reward models, including ImageReward, Aesthetic Score, and Pick Score. The win rate measures how often the fine-tuned model outperforms the original model based on different reward models, with a performance above 50% indicating an improvement. All evaluations are conducted on the test split of the Pick-a-Pic v2 dataset. Here are the following results corresponding to our current optimization steps:\n\n| Reward Model       | Pick Score | Aesthetic Score | ImageReward |\n|--------------|---------------|---------------------|---------------------|\n| CED         |   58.60  |           56.20          |  55.60  |\n\nThis experiment demonstrates both the effectiveness and efficiency of our approach, as we successfully fine-tune the model using a single A100 GPU. Achieving a win rate above 50% indicates that our method consistently improves image quality in the complex scenario like text-to-image generation. Therefore, it shows the potential of our approach for more difficult settings like text-to-image diffusion tasks.\n\n**Q2. Therefore, I'm not convinced it is actually more efficient than DMD or previous approach as there is no training time comparison, also DMDv2 [1] has introduce some tricks to make it much more efficient.**\n\n**A2.** Thanks for your suggestions and comments! Our method relies on the distillation setting without real image data. However, DMDv2 operates under a different setting from ours, as it incorporates GANs and relies on real data to enhance performance during training. This makes a direct comparison with our method challenging. To fairly evaluate the efficiency of our approach, we instead compare it with another distillation method that, like ours, does not use real data—SiD. **The results are as follows: CED achieves an FID of 3.85 on CIFAR-10 using only 20 A100 GPU hours, whereas SiD requires 100 A100 GPU hours to reach an FID of 4.0.** These findings highlight the training efficiency of our method in achieving comparable performance with significantly lower computational cost.\n\n**Q3. Lack comparison with DMDv2, which I believe also report on same setting 32×32 CIFAR-10 and 64×64 ImageNet.**\n\n**A3.** Thanks for your suggestions and comments! DMDv2 operates under a fundamentally different setting from ours, as it incorporates GANs and leverages real data to boost performance during training. As a result, making a direct comparison with our method is not straightforward. However, we include comparisons with DMD—without the use of GAN techniques—in Tables 2 and 3. Our method demonstrates improved performance on CIFAR-10 and achieves comparable results on ImageNet, all while avoiding the need for additional models and iterative optimization. Moreover, we believe that integrating our method with GANs presents an interesting direction for further improvement. To explore this, we combined CED with a GAN-based approach similar to DMDv2 and evaluated it on CIFAR-10 using real image data. This resulted in an impressive FID of 1.85, demonstrating a significant performance boost over previous methods. However, due to time constraints, we were unable to conduct experiments on ImageNet, and we note that DMDv2 does not report results on CIFAR-10 for direct comparison. We plan to extend our experiments to ImageNet in future work. Nonetheless, these preliminary results highlight the strong potential of our method and its compatibility with state-of-the-art techniques to further enhance performance.\n\n**Q4. Please report on training time and compare it with other method like DMD, DMDv2, CD, SiD.**\n\n**A4.** Thanks for your suggestions! Since DMDv2 extends DMD by incorporating GANs to train one-step diffusion models with real data, it falls outside the scope of our setting, which focuses on distillation-based methods without access to real data. Moreover, the authors of DMD do not provide training code, making direct comparison difficult. Instead, we compare our method with SiD, which adopts a similar iterative training scheme, relies on additional models and claims faster convergence. Until now, we have the following results: CED achieves an FID of 3.85 on CIFAR-10 using only 20 A100 GPU hours, whereas SiD requires 100 A100 GPU hours to reach an FID of 4.0. Moreover, CD requires approximately 1000 A100 GPU hours on CIFAR-10 to achieve an FID of 3.55, whereas our proposed method reaches comparable performance with only about 30 A100 GPU hours. This significant reduction in computational cost further demonstrates the efficiency of our approach.\n\n**Q5. Author should use same initial noise and same class input for sampling for Figure 4. So the reader can visually compare between methods**\n\n**A5.** Thanks for your suggestions! Due to format constraints in the rebuttal, we will include these results in the revised version of the paper."}]

Author rebuttal:

Meta-review: This paper introduces Contrastive Energy Distillation (CED), a simple and efficient method to convert slow, multi-step diffusion models into fast, one-step image generators. The key idea is to frame the distillation as a contrastive learning task, where a student model learns to distinguish "positive" samples from a teacher model against "negative" samples from a previous version of itself. By leveraging a joint energy-based model and a contrastive loss inspired by Noise Contrastive Estimation, CED avoids auxiliary models and iterative training. It achieves competitive performance on CIFAR-10 and ImageNet.


This work has several clear strengths:
- The core idea of using a contrastive Energy objective to traina one-step generator is novel and empirically effective.

- CED eliminates the need for auxiliary score models and iterative updates, reducing memory usage and simplifying implementation.

- This work provides theoretical justification, including a compelling theoretical connection between the proposed CED objective and KL divergence.


However, the scalability concern is one major remaining concern. The scalability to more complex data generation, such as larger resolution and even videos, is unclear. Moreover, it is also difficult to understand CED with model size, given the existing results. I partially agree with Reviewer deet’s final justifcation that the restricted evaluation somewhat limits the paper's quality, particularly in its lack of comprehensive evaluation on larger image resolutions and a comprehensive comparison with non-distillation approaches.

While some concerns remain, I believe this work is novel and interesting enough for being accepted to NeurIPS. It will promising to further explore its scalability to larger models and more complex data.

In summary, I recommend Accept (Poster).
Decision: Accept (poster)