Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
Authors: Bowen Zheng
Tianming Yang
Venue: ICML 2025 poster (2025)
Keywords: Diffusion Models
One-Step Generators
Generative Models
Image Generation
GAN
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: Diffusion distillation is a widely used technique to reduce the sampling cost of diffusion models, yet it often requires extensive training, and the student performance tends to be degraded. Recent studies show that incorporating a GAN objective may alleviate these issues, yet the underlying mechanism remains unclear. In this work, we first identify a key limitation of distillation: mismatched step sizes and parameter numbers between the teacher and the student model lead them to converge to different local minima, rendering direct imitation suboptimal. We further demonstrate that a standalone GAN objective, without relying a distillation loss, overcomes this limitation and is sufficient to convert diffusion models into efficient one-step generators. Based on this finding, we propose that diffusion training may be viewed as a form of generative pre-training, equipping models with capabilities that can be unlocked through lightweight GAN fine-tuning. Supporting this view, we create a one-step generation model by fine-tuning a pre-trained model with 85% of parameters frozen, achieving strong performance with only 0.2M images and near-SOTA results with 5M images. We further present a frequency-domain analysis that may explain the one-step generative capability gained in diffusion training. Overall, our work provides a new perspective for diffusion training, highlighting its role as a powerful generative pre-training process, which can be the basis for building efficient one-step generation models.

Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
Bowen Zheng' Tianming Yang'
Abstract
1. Introduction
Diffusion distillation is a widely used technique to reduce the sampling cost of diffusion models, yet it often requires extensive training, and the student performance tends to be degraded. Re- cent studies show that incorporating a GAN ob- jective may alleviate these issues, yet the under- lying mechanism remains unclear. In this work, we first identify a key limitation of distillation: mismatched step sizes and parameter numbers be- tween the teacher and the student model lead them to converge to different local minima, rendering direct imitation suboptimal. We further demon- strate that a standalone GAN objective, without relying a distillation loss, overcomes this limita- tion and sufficient to convert diffusion models into efficient one-step generators. Based on this finding, we propose that diffusion training may be viewed as form of generative pre-training, equip- ping models with capabilities that can be unlocked through lightweight GAN fine-tuning. Supporting this view, we create a one-step generation model by fine-tuning a pre-trained model with 85% of parameters frozen, achieving strong performance with only 0.2M images and near-SOTA results with 5M images. We further present frequency- domain analysis that may explain the one-step generative capability gained in diffusion training. Overall, our work provides a new perspective for diffusion training, highlighting its role as a power- ful generative pre- pre-training process, which can be the basis for building efficient one-step generation models.
Diffusion Models (DMs) (Song & Ermon, 2020a;b; Ho et al., 2020) have emerged as powerful tools for generative tasks, such as image and video synthesis, surpassing tradi- tional methods such as Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) in terms of sample qual- ity. However, the generation process in DMs slow and requires a series of iterative steps that impose significant computational overhead.
To speed up the generation process, several distillation tech- niques (Salimans & Ho, 2022; Gu et al., 2023; Song et al., 2023; Yin et al., 2023; Zhou et al., 2024a;c; Hsiao et al., 2024) have been proposed to reduce the multi-step diffusion process into a few- or one-step model. While promising, these methods necessitate large computational resources for the distillation and often fall short of matching the perfor- mance of the original multi-step diffusion models. However, recent advances (Sauer et al., 2023; Xu et al., 2023; Kang et al., 2024; Sauer et al., 2024; Li et al., 2024a; Kim et al., 2024a) have shown that incorporating a GAN objective in the distillation process can improve efficiency. demonstrat- ing the potential of GAN-based 'distillation' approaches.
In this work, we first identify a fundamental limitation of ex- isting distillation methods: the local minima of the teacher and student models may differ significantly, hindering ef- fective knowledge transfer. In contrast, the GAN objective naturally circumvents this issue. Building on this insight, we propose D20 (Diffusion to One-Step), a novel approach that relies solely on an exclusive GAN objective, eliminating the need for instance-level distillation losses. D20 achieves competitive performance with drastically reduced data re- quirements, while conventional distillation methods demand vast training images when trained without GAN objective. This significant reduction suggests that the D2O model is likely not learning from scratch, as that typically requires tens or millions of images.
Institute of Neuroscience, Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology. Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai, China. Correspondence to: Tianming Yang <tyang @ ic..ac.cn"..
Based on these results, we propose that the diffusion ob- jective can be viewed as a generative pre-training process, wherein the model learns generalized generative capabilities that can be rapidly adapted to downstream tasks. To verify this idea, we create the D20-F model, with most parameters
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
Generative Pre Training
PD
14
(1e6 K]
GIV
CD
12
Coio
3k4p
BOOT
10
DMD
ADA-EN
8
DMD2 (Longer)
Fime-Tine Tuning
MSD (DM)
6
MSD (ADM)
D20 (Durs]
4
D20 (Ours)
Gencrder (Pantally Troan))
0
102 Training Images (M)
Figure 1. A. Comparison between our methods (D20 and D20-F) and other methods on ImageNet 64x64, Our models show competitive results with much smaller training set than the competing models. D2O-F model. We initialize the generator with a pre-trained diffusion model and freeze most convolutional layers during the fine-tuning, simple GAN objective is adopted. C. A detailed illustration of freezing method used D2O-F blocks. Most convolutional layers (blue) are frozen. Only the normalization layer and the skip connection (red) are fine-tuned.
frozen during the fine-tuning. The D20-F model shows high efficiency similar to D20 and exhibits even better per- formance. The success of the freezing method supports our theory that diffusion models possess universal generative capabilities and can efficiently converted into an efficient one-step generator with GAN objective.
solvers (Lu et al., 2022a;b; Zhou et al., 2024b; Xue et al., 2024). We use the Euler solver as an example here. With this solver, we can sample iteratively to get the final results.
2.2. Diffusion Distillation
A diffusion distillation method typically consists of series of teacher models H and student models Fg. These teacher models often have more steps than the student models. For example, in Progressive Distillation (PD) (Salimans & Ho, 2022), teacher models are defined as:
2. Potential Limitation of Distillation Methods
2.1. Diff'usion Models
There are several ways to interpret diffusion models, such as the score-matching approach (Song & Ermon, 2020a) and the probabilistic perspective (Ho et al., 2020). For the training process, given a clean image zo and random noise the training objective typically involves combining these with varying proportions and asking the model to predict the clean image. This task is equivalent to predicting the noise the differences between image and noise.
H = ( 44 4j, F (F ((rr,) ti (g), 1z. tx),
i-j=j- =j k=
i- = #
where sg is the stop gradient to prevent gradient leakage to the teacher models.
Recently, an increasing number of distillation methods have found that adding an extra GAN loss alongside the distilla- tion loss can be highly effective (Sauer et al., 2023; Xu et al., 2023; Kang et al., 2024; Sauer et al., 2024; Li et al., 2024a; Kim et al., 2024a). This observation motivates us to explore the mechanism underlying the performance gap between the original distillation methods and the gain provided by simple GAN loss.
In the EDM framework (Karras et al., 2022), a clean image To is perturbed by adding Gaussian noise 24; ~> Na E.I, where 4 is the standard deviation determined by a time-step scheduling function T'i, NN. Here, N denotes the total number of discrete time steps, and i refers the current step index. A score model g& ( 4) is then tasked with predicting the clean image Zo using this perturbed image.
2.3. Inconsistent Local Minima
With this pre- pre-traine trained score model. an ODE solver S, can defined as:
In these distillation methods, teacher model H must pass through the neural network multiple times and contain more parameters than the student, while the student model Fş passes through the neural network in a single iteration and has fewer parameters. We reveal the potential problem
So t4, f--1) 4-4
(go( zT,)) + Ztt,
There are many different types of Sg. e.g.. the Heun solver introduced in EDM (Karras et al., 2022), or other high-order
2

---

Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
introduced by such setup.
A.
B.
211
2.1
The difference between the teacher models and the student models leads to significant inductive bias: the teacher model can transform between pixel space and latent space several times while the student model can only perform the transformation once. We speculate that this may produce different optimization landscapes and different local min- ima between the teacher and student model. Forcing the student model to approximate the teacher model's 's results may, therefore, yield sub-optimal results.
2.0
Siudani
1.9
1.8
FID Target
1.7-
Student model
1.8
Dataset
1.6
0.1
0.5
10
25
Teather Steps
Sigma
Figure 2. The FIDs of teacher models. The FIDs are similar when computed against the target dataset across different teacher steps (gray A) or sigmas the intermediate steps (gray in B). When computed against the student model, the FID increases as the steps grow (blue A) and decreases as the sigma decreases (steps=2, bluc in B).
To verify our hypothesis, we use the Fréchet Inception Dis- tance (FID) (Heusel et al., 2018) to compare two image sets. Specifically, we compute the FID between the teacher and student models, as well as the FID against the training-set images. FIDs are calculated using 50,000 images generated by each model with fixed seeds ranging from 0 to 49,999, and features for FID computation are extracted using Inception model.
We define a series of teacher models with 2, 4, 6, 8, and 10 steps in progressive distillation (PD) manner. For instance, two-step teacher model parameterized as:
them (FID=1.78 between a two-step teacher and a one-step student). The student model fails to mimic the teacher model at the instance level, which might be the reason that limits the performance of distillation methods.
H = Fo(FFFFFFFF f2, t1), to).
where 2(,, 2), T(O, 2)}, and similarly for other configurations. Both teacher and student models are trained using only the GAN loss and initialized from well- trained score model in EDM (Karras et al., 2022). All teacher models achieve FID < 2.2 within a few training steps (less than 5 million training images), ensuring com- parable performance across models. Finally, we compute the FID between the student model and each teacher model, following the evaluation protocol in EDM (Karras et al., 2022).
3. Fast Converge With An Exclusive GAN Objective
3.1. Training Without Distillation Objective
We have demonstrated that the key issue with distillation methods arises from the fact that the teacher and student models, despite sharing the same architecture, are optimized differently due to the teacher's 's multi-step process. Directly aligning the student's output with the teacher's at each in- stance, which forces the student into the teacher's specific local minima (e.g., requiring Istudent to match Kteacher), may be suboptimal for the student model. A simple GAN objective can bypass this issue, guiding the student to align with the overall data distribution rather than replicating pre- cise instances, allowing it to refine Istudent even when it differs from Steacher: and thus find an optimal solution in its own parameter space. In addition, using the real im- ages instead of the teacher model's results (Lin al., 2024) in GAN objective pushes the student toward the real data distribution, thus bypassing imperfections in the teacher's distribution and achieving better performance.
Figure 2A. illustrates the FIDs for teacher models with varying numbers of steps. While the FID values computed with the target dataset are similar across teacher models, those with more steps show higher FID values when com- pared with the one-step student model. This suggests greater divergence from the student model as the number of teacher steps increases. Additionally, when we fix the teacher model's steps to two and vary the sigma of the in- termediate time step, the FID computed with the one-step student decreases (Figure 2B.) Teacher models with higher sigma values generate images that are more similar to those produced by the one-step student model. More details of the loss function, time scheduler, and student/teacher model in distillation methods can be found in Appendix B.
3.2. Baseline
Based on this reasoning, we create the D20 (Diffusion to One-Step) model based on diffusion U-Net architecture with only a GAN objective. We first test its performance in a small-scale experiment on CIFAR-10 10 (Krizhevsky, 2009) as the baseline. The non-saturating GAN objective used
These results reveal that the teacher and student models may achieve similar performance, but in different ways. Teachers with fewer steps are more similar to the one-step student, but there is still significant difference between
3

---

Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
Table Discriminators that either strict too weak can lead to model collapse. Multi-scale discriminators (LPIPS and PG) show better performance.
Table 2. Augmentation leads to poor results. RI regularization
improves performance significantly.
Augmentation
Tr1
FID (L)
Discriminator
FID (L)
2.21
StyleGAN2 (Scratch)
collapse
X
1.98
StyleGAN2 (Pre-trained)
collapse
A
1e-3
1.77
VGG16
4.04
le-4
1.66
LPIPS
3.49
1e-5
1.66
PG
2.21
for the discriminator and the generator, respectively:
the original work.
Ex [log (Do)) + E,[1 log(D(G((z)))
As illustrated in Table discriminators with multiple scales are better than the vanilla VGG discriminator. Overall, the PG discriminator with a fusion feature based on VGGI6-BN and EfficientNet-iice we will use the PG discriminator and objective in the fol- lowing experiments.
min F. - (z)( Go
achieves the best results. Therefore,
With a pre-trained diffusion U-Net model, the score model go is used as the generator Gg. and a pre-trained VGG16 (Simonyan & Zisserman, 2015) is used as a discriminator by default. EDM with NCSN+ (Song al., 2021) architecture used the generator. We choose VGGI6 (Simonyan & Zisserman, 2015) baseline discriminator. Adaptive augmentation (ADA) (Karras et al., 2020a) is adopted with rl regularization (Mescheder et al., 2018) where "/ 0.01 as suggested in StyleGAN2-ADA (Karras et al., 2020a). Our baseline setting shows good performance on the CIFAR-10 dataset with only 5M training images and already achieves performance (FID=4.04) close to that of the Consistency Distillation (Song al., 2023) (FID=3.55).
3.4. Augmentation and Regularization
Differentiable augmentation (diffAUG) was introduced in (Zhao et al., 2020) to prevent the overfitting of discrimina- tors, but we find that leads to poor results in our method (Table 2). This may be because EDMs are pre-trained with data augmentation. We disable all augmentation in all of our further experiments.
In the original PG, regularization for the discriminator was not used. However, our experiments show that it is im- portant to apply regularization on the discriminator to prevent overfitting and improves performance significantly (Table 2). The value of %+1 should be kept small since the discriminator of the Projected GAN generates multiple out- put logits, and a large //1 may lead to numerical explosion. With optimal settings, this setup can obtain satisfying results with only 0.2 million training images (FID=3.62), With million training images, the results are close to SOTA results (FID=1.66).
3.3. Discriminator
An appropriate discriminator is crucial since represents the overall target distribution and determines the performance of the generator. The StyleGAN2 (both scratch and pre-trained with GAN objective) (Karras et al., 2020b), VGG, LPIPS (Zhang et al., 2018). and Projected GAN (PG) (Sauer et al., 2022) discriminators are compared. Adaptive augmenta- tion (ADA) is adopted for StyleGAN2, VGG, and LPIPS discriminators with regularization where T-1 0.01.
4. Diffusion Objective as Generative Pre-training
For the PG discriminator, of PG objective:
non-saturating version
4.1. Generative Pre-training
max (Di)
lec
D20 attains strong results using far fewer training images than most previous distillation methods (See Appendix F for more detail). These gains unlikely to come from the brief GAN fine-tuning alone, since 0.2 million images are insufficient for the model to learn complex distribution from scratch. GAN training typically requires tens of mil- lions of images, and training diffusion models often needs hundreds of millions or even billions images.
min Go
Exl- (og((
lEc
We use VGG16 with batch normalization (VGG16-BN) (Simonyan & Zisserman, 2015; loffe & Szegedy, 2015) and EfficientNet-lited (Tan & Le. 2020) as feature networks P. We adopt differentiable augmentation (diffAUG) (Zhao al., 2020) without a gradient penalty by default, following

---

Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
A.
Table D20-F: ablation experiments. v/ indicates correspond- ing layer can be tuned and X indicates frozen layers. The percent- age in the brackets is the proportion the total parameters these
40
Methods
020
14-
D20 CD
30
12-
D20
layers.
D20
10-
2... D204 13
20-
Norm(7.9%% Conv (85.8%) QKV (2.1%) Skip(4.0%) (J)
10-
/
/
1.66
0
x
/
/
1.54
2
3
0
1
4
5
Training images (Million)
Training Images (Million)
X
/
1.60
X
X
2.51
Figure 3. A. Performance of the D20 and CD models with (D20-F and CD-F) and without (D20 and CD) freezing the convolutional
Therefore, we speculate that the diffusion U-Net acquires an intrinsic generative capability during its original diffusion training, which the GAN fine-tuning phase merely unlocks. While directly using the pre-trained diffusion model for in- ference may be inefficient (Ma et al., 2023; Li et al., 2023a; Zou et al., 2024; Li et al., 2024b; Zhao et al., 2024; Chen et al., 2024a; Kim et al., 2024b; Chen et al., 2024b), it can be fine-tuned with more direct and effective generative objective, such as GAN-based objective, to utilize these intrinsic generative capabilities and produce fast and effi- cient generative model.
layers. B. Effects of adding an extra CD loss to the D20 and D2O-F models.
diffusion U-Net. D20-F produces satisfying images with as few as 0,2 million training steps (FID=4.12). The per- formance further reaches near the SOTA level with only million steps (FID=1.54). In comparison, training gen- erative model with similar performance typically requires tens or hundreds of millions of training steps (100M for StyleGAN2-ADA, 200M for EDM, on CIFAR-10). These results strongly support our theory that the initial diffusion training already provides a sufficient generative capability that can be rapidly fine-tuned for one-step generation.
This approach avoids redundancy by eliminating the role of noise as a redundant information destroyer during the iterative inference process. Instead, the noise serves as an index that maps a sample from the perturbed distribution onto its counterpart in the target distribution. similar to Consistency Models (Song al., 2023; Song & Dhariwal, 2023) and GANs.
Notably, with the majority of parameters in both the dis- criminator and the generator frozen, the training process of D20-F is stable with minimal instances of mode collapse. Thereby, the freezing method circumvents the inherent in- stability of using GANs.
4.2. Achieving One-Step Generation via Freezing
To verify the feasibility, efficiency, and generality of this per- spective, particularly in fine-tuning generative pre-training models for one-step generation, we create the D20-F (Diffu- sion to One-Step Generators with Freezing) model in which most of the diffusion model's parameters are frozen during the fine-tuning with a GAN objective. If the model indeed relies on its pre-trained generative capabilities, we should expect to observe performance similar to D)20.
Finally, freezing parameters different from lora-based methods (Luo al., 2023; Lin et al., 2024). Lora-based methods adjust the convolutional parameters, although in low-dimensional manner. By freezing the convolution layer, our method suggests that the diffusion models already possess the capacity for one-step generation.
4.3. Freezing Is Not Suitable for Distillation
To find the optimal setting, we conduct the ablation ex- periments in which different sets of parameters are frozen or tunable during training (Table 3). The results indicate that freezing most the convolutional layers leads to the best performance. Further freezing the QKV projections (Vaswani al.. 2023) causes a slight decrease in the per- formance, and freezing the skip layers the residual con- nections leads to worse results, suggesting that tuning these two types of layers is necessary.
The freezing method does not work well for the distillation methods. To demonstrate this, we compare the original CD and the CD with frozen convolutional layers (CD-F) (Fig. 3A.). CD-F performs poorly, which suggests that forcing the student model to emulate the teacher model re- quires adjustments in the convolutional layers. This further supports our hypothesis that forcing the student model to replicate the teacher model's 's outputs at the instance level will lead to suboptimal results due to the mismatch between their optimal local minima. This explains the inefficiency of distillation methods as a fine-tuning target: they require the model to learn generative capabilities anew.
Based on these results, the D20 model is trained with most of the original parameters locked (85.8% of the parame- ters contained in the convolutional layers) in the pre-trained
5

---

Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
Input
0.08
10
0.06
Output
0.00
0.00
O, O--1
20
Radius
C.
Or,r(Or,O,-a))
-
Cheps
5
10 15
20
25
30
Time Steps
18
42
Radius
Figure 4. A. Spatial and frequency domain visualization. First row: Inputs at different time steps. Second row: Outputs at different time steps. Third Fourth row: the difference between the current outputs and the previous outputs clearer visualization) and the frequency domain. Fifth and Sixth row: the difference between the current outputs and the current inputs in the spatial and frequency domain. B. Radial Averaging of DFFT (O,, O.-1), corresponding to fourth row A. Smaller radii indicate lower frequencies. During inference, the model selectively enhances increasingly higher frequency components at later time steps. C. Radial Averaging of DeFr(Ou (Ot, It), corresponding to the last row A. All frequency bands are suppressed, which is more so in carly time steps. That is because the model removes the redundant noise each time step.
spatial domain (enhanced for
4.4. Extra Distillation Objective
with the diffusion objective remains unclear. We speculate that these generative capabilities might involve frequency- specific processing.
Previous works (Sauer al., 2023; Xu al., 2023; Kang al., 2024; Sauer et al., 2024; Li et al., 2024a; Kim al., 2024a) employ both the distillation objective and the GAN objective. To assess their contribution to performance, we test our model with an additional distillation loss incorpo- rated. We use the Consistency Distillation (CD) objective (Song et al., 2023) here, as easy to implement and per- forms well. Detailed information about the CD objective can be found in Appendix C. The comparisons conducted on the CIFAR-10 dataset.
We use the log-frequency difference as core metric to quantify frequency response:
DFFT #2) = log (IFFT(z)) + +1) 1)- - log (IFFT(x2) + 1);
where Ti and i2 represent two quantities to be compared.
Global Frequency Evolution We first notice that the in- formation across the entire frequency range changes be- tween the input and the output at different time steps, mea- sured by DFFT (O., It) (Figure 4 A. and C.). This is con- sistent with the denoising process carried out by the model, which suppresses noise across the entire spectrum, Further- more, when examining the frequency differences between the outputs of two consecutive time steps FTFT(O, (O,, O,-1), we observe a distinct pattern: low-frequency components are enhanced in the earlier time steps, while the enhanced frequencies shift toward the middle and high ranges in later time steps (Figure and B.). This pattern provides quan- titative support for the commonly observed phenomenon that diffusion inference progressively restores signals from low to high frequencies (Rissanen et al., 2023; Dieleman, 2023). It suggests that diffusion models may process differ- ent frequency components in a time-step-dependers manner during inference.
We find that the extra CD loss significantly slows down the convergence in the early stage. Although it performs slightly better than D2O, this advantage disappears when compared against D20-F (Figure 3B.). In addition, adding CD loss nearly doubles the training resources and time.
Although the extra distillation loss may offer potential bene- fits, such as in image editing or classifier-free guidance (Ho & Salimans, 2022), these benefits may be achieved in our model with other methods such as adjusting the downstream task.
4.5. Frequency-Specific Processing
Though our experiments provide evidence for the general generative capabilities in diffusion models, the exact mech- anism by which these capabilities emerge during training
6

---

Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
Table comprehensive comparison between D2O, D20-F, and previous models on CIFAR-10.
Table 5. comprehensive comparison between D20, D20-F, and previous work on AFHQv2 64x64 and FFHQ 64x64.
Method
NFE (1)
Unconditional Conditional FID (CT
(D)
Method
NFE (4)
AFHQV2 FFHQ
FID (L)
Training From Seratch
Training From Scratch
DDPM (Ho al., 2020)
1000
3.17
DDIM (Song et 2022)
100
4.16
EDM (Karras et al., 2022)
79
1.96
2.39
Score SDE (Song et al., 2021)
2000
2.20
DPM 3 (I.u
2022a)
2.65
Post-training
EDM (Karrus et al., 2022)
35
1.98
1.79
BigGAN
al., 2019)
14.73
BOOT (Gu et al., 2023)
9.0
AyyeGGAN-ADAA IKarras al., 2020s)
2,92
9.82
2.42
SiD (Zhou et al., 2024a), a = 1.0
1.62
1.71
SAN (Takidu et al., 2024)
1.36
SiD (Zhou et al., 2024a),
1.2
1.71
1.55
(Song & Dhariwal, 20233
9.54
D20 (ours)
1.23
1.08
iCT (Song& & Dhariwal, 202.3
2,46
9,80
D20-F (ours)
1.31
0.85
iCT-deep (Song Dhariwal, 2023)
2.51
9.76
iCT-doep (Song& Dhariwal, 2023)
2.24
9.89
Post-training
(Salimans Ho, 2022)
9.12
DRNO (Zheng et al., 2023)
3.78
CD (Song et al., 2023)
3.55
CD (ong al., 2023)
2.93
CTM (Kim et al, 2024a)
1.98
1.73
CTM (Kim et al., 2024a)
1.87
1.63
DMD (Yin
2023)
2.62
SiD (Zhou et al., 2024a), 1.0
2.02
10.01
1.93
(Zhou et al., 20244), a D20 (ours)
1.92
9.98
1.71
1.66
10.11
1.58
D20-F (ours)
1.54
10.10
1.44
Figure 5. Sample comparison between EDM (Top), D20 (Middle) and D2O-F (Bottom) AFHQv2 (left) and FFHQ (right).
Block-wise Frequency Specialization We also examine the frequency response of the U-Net blocks. Our analyses reveal a natural frequency-specific processing pattern, influ- enced by the difftusion U-Net structure (Figure 6). The deep blocks, operating at lower resolutions, are limited in their ability to process high-frequency information. In contrast, the shallow blocks, operating at higher resolutions, are bet- ter suited for capturing a broader range of high-frequency information. In addition, there is evidence of frequency- specific processing across different blocks at various time steps. feeghfrry components often undergo more sig- nificant changes in the shallower layers, particularly during later time steps (See Appendix E for more details).
trained diffusion models are from EDM. We report FID for all datasets, Inception Score (IS) (Salimans et al., 2016) for CIFAR-10, precision and recall metric (Kynkäänniemi et al., 2019) for ImageNet 64x64 following previous works. For the discriminator, we use VGG16 with batch normalization and EfficientNet-liteOO as the feature network for CIFAR- 10, DeiT (Touvron et al., 2021). and EfficientNet-lited for the other datasets. The optimal settings for discriminator, augmentation, and regularization determined in previous experiments are used. More implementation details can be found in Appendix C.
These results hint that frequency-specifie processing may underlie the generative capabilities observed in diffusion models, which could be rapidly adapted and integrated with a simple GAN objective to efficiently produce a one-step generator.
5.2. Results
The results can be found in Table 4, 5 and 6. Both D20 and D20-F achieve competitive results on all datasets tested. Our methods not only are suitable for unconditional genera- tion but also show great performance on class-conditionn generation tasks. Moreover, D20-F outperforms D20 by a significant margin in most datasets. except AFHQv2 64x64. On CIFAR-10, our methods achieve FID of 1.54 and an IS of 10.11. Similarly, superior performance is found on AFHQv2 64x64 (FID=1.23) and FFHQ 64x64 (FID==085)) In ImageNet 64x64, the model also achieves SOTA perfor- mance (FID=1.16, precision=0.77).
5. Comprehensive Performance Test
5.1. Basic Setting
Finally, conduct comprehensive evaluation of both D20 and D20-F on CIFAR-10, AFHQv2 64x64 (Choi et al., 2020), FFHQ 64x64 (Karras et al., 2019), and ImageNet 64x64 (Deng et al., 2009). The results for class-conditiinal generation and the comparison against previous studies are reported on CIFAR-10 10 and ImageNet 64x64. The pre-
We further compare the outputs from the converted one-step
7

---

Revisiting Diffusson Models: From Generative Pre-training to One-Step Generation
noise levels. The training process helps the model learn the necessary generative capabilities. The destruction process allows the model to sample from a known distribution and generate or edit images according to the task.
Table A comprehensive comparison between D20, D20-F, and the competing models on ImageNet 64x64 (class-conditional..
Method
NFE (L) FID (J) Prec. (1) Rec. (t)
Training From Scratch
An architecture based on discrete tokens, such as auto- regressive models, could also be framed within this frame- work with a few simple adjustments. Specifically, instead of merely removing tokens, we could replace them with tokens randomly sampled from discrete uniform distribu- tion (mixing these tokens is also viable option). After the pre-training process, we can fine-tune the pre-trained model by sampling a set of random tokens and adopting a simple GAN objective to achieve one-step generation.
RIN (Jabri et al., 2023)
1000
1.23
DDPM (Ho al., 2020)
250
11.00
0.67
0.58
ADM (Dhariwal & Nichol, 20
250
2.07
0.74
0.63
EDM (Karras et al., 2022)
79
2.64
iCT (Song& & Dhariwal, 2023)
1
4.02
0.70
0.63
iCT (Song & Dhariwal, 2023)
2
3.20
0.73
0.63
iCT-deep (Song & Dhariwal, 2023)
3.25
0.63
iCT-deep (Song & Dhuriwal, 2023)
2.
2.77
0.74
0.62
BigGAN dieepp--eepppp (Brock et al., 2019)
4.06
0.79
0.48
StyleGAN2-XL (Sauer et al., 2022)
1.51
Post-truining
PD (Salimans & Ho, 2022)
15.39
BOOT (Gu et al. 2023)
16.3
0.68
0,36
6.2. Limitation and Future Work
DENO (Zheng et al.. 2023)
7.83
0.61
CD (Song
2023)
6.20
0.68
0.63
CD (Song ct al., 2023)
2
4,70
0.69
0.64
Our model is based on the diffusion U-Net architecture. The effectiveness of our approach on different architectures, such as DiT (Peebles & Xie, 2023), remains to be explored. Addi- tionally, while the proposed generative pre-training method demonstrates good performance on the one-step generation task across several datasets, including the relatively com- plex ImageNett 64x64, it has yet to be tested on datasets with higher resolution and more complex generation tasks, such as the COCO (Lin et al., 2015) dataset. Finally, our efforts to explore the generative capabilities through the di diffusion models' frequency response are only preliminary and need to be expanded. In the future, we will focus on extending our methods to more architectures, datasets, and tasks, as well as exploring the underlying mechanisms of how the diffusion objective provides universal capacities to diffusion models.
CTM (Kim al., 2024a)
1.92
0.70
0.57
CTM (Kim al., 2024a)
1.73
0.64
0.57
DMD (Yin ct al. 2023)
2.62
SiD (Zhou ul., 2024g), 1=1.0
2.02
0.73
0.63
SiD (Zhou al., 20244), 4=1.2
1.52
0.74
0.63
sCD-S (Lu & Song, 2025)
2.97
sCD-S
Song, 2025)
ECM-S (Geng ct al. 2024)
4.05
ECM-S S (Geng et 2024)
2
2.79
DMD2 (Yin
2024)
1.51
DMD2 (Yin et al., 2024), longer training
1.26
MSD (Song et al., 2024), 4 students, DM
2.37
MSD (Song ul., 2024), 4 students, ADM (20( D20-F(ours)
1,20
1.42
0.77
0.59
1.16
0.75
0.60
generator and the original diffusion model EDM (Figure 5, more samples can be found in Appendix G). The produced images are similar, and there is a certain degree of variation, as we do not force the student model to replicate the teacher model at the instance level, allowing it to perform differently to surpass the original model. Based on FID (calculated using EDM's results on CIFAR-1O), D20-F (1.87) performs similarly to the other distillation methods like CD (1.76) and significantly better than StyleGAN-XL. (2.60). Together, these results demonstrate the robustness and effectiveness of our methods, supporting our hypothesis that one-step generation utilizes the generative capabilities gained from pre-training.
7. Conclusion
In this work, we aim to improve the efficiency of training one-step generative models from diffusion models. We first identify a fundamental limitation of distillation methods: the teacher and student models may converge to distinet lo- cal minima and the discrepancy hinders knowledge transfer. Therefore, we create the D2O model with a GAN objective to circumvent this issue. The D20 model produced results beyond our expectations, and we further explored the un- derlying mechanism. We propose that the training of the original model with the diffusion objective can be viewed as generative pre-training where the model learns generaliz- able capabilities that can be rapidly fine-tuned and integrated into one-step generator. To demonstrate the feasibility of the idea, we build the D20-F model in which significant portion of the original model parameters are frozen during fine- tunettnnig. The model achieves competitive performance with far fewer training images, supporting our hypothesis that the one-step generation can be achieved through rapid fine-tuning of the pre-trained generative capabilities in diffu-
6. Discussion
6.1. Generative Pre-training
Here, we propose that diffusion model training can be viewed as generative pre-training process. With this view, the generative pre-training should consist of two key pro- cesses: a destruction process that reduces information to a known noise distribution and a sulf-supe training pro- cess that trains the model to recover samples with varying
8

---

Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
sion models. The frequency analysis provides a preliminary explanation for the freezing technique by revealing the tem- poral and bise frequency specialization in diffusion models. Together, our work provides key intuition for de- veloping efficient and high-performance generative models in the future.
Esser, Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis, 2021.
Geng. Z., Pokle, A., Luo, W., Lin, J., and Kolter, J. Z. Consistency models made easy, 2024. URL https: //arxiv. org/ http://bs//22....// /tt.:/240.........
Goodfellow, I. J.. Pouget-Abadie, J., Mirza, M.. Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversaria networks, 2014.
Impact Statement
The primary advantage of transforming diffusion models into one-step generators lies in increased efficiency, which reduces the computational resources needed for generation. This not only leads to energy savings but also makes high- performance models accessible to users with standard com- puting resources, promoting fairness Al usage.
Gu, J., Zhai, S., Zhang, Y., Liu, L., and Susskind, J. Boot: Data-fire distillation of denoising diffusion models with bootstrapping. 2023.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by two time-scale update rule converge local nash equilibrium, 2018.
However, similar to other generative models, there po- tential risks, such as the misuse of these models to generate harmful content like violence or pornography. These risks are already part of ongoing industry discussions, and re- sponsible deployment, along with clear ethical guidelines, is essential to prevent misuse.
Ho, J. and Salimans, Classifier-free diffusion guid- ance, 2022. 2207. 12598..
URL
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models, 2020.
References
Brock, A., Donahue, J., and Simonyan, K. Large scale gan training for high fidelity natural image synthesis, 2019.
Hsiao, Y-TT., Khodadadeh, S., Duarte, K., Lin, W.-A., Qu, H., Kwon, M., and Kalarot, R. Plug-and-pla diffusion distillation, 2024. URL https: http///axvvvvvr/////ee /bbs/ 2406. 01954.
Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. Maskgit: Masked generative image transformer, 2022.
Huang, L., Fang, R., Zhang, A., Song, G., Liu, S., Liu, Y., and Li, H. Fouriscale: A frequency perspective on training- high-resolution image synthesis, 2024, URL https: setps://arxive hwtp /tw.////////..././///////// 12963. 1299..
Chen, G., Zhao, X., Zhou, Y., Chen, T., and Cheng, Y. Accelerating vision diffusion transformers with skip arxia. org/ abs/ branches, 2024a. URL https: /ttpps::///xxv..../ 2411. 17616.
loffe, and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift, 2015.
Chen, Shen, M., Ye, P., Cao, J., Tu, C., Bouganis, C.-S., Zhao, Y., and Chen, T. d-dit: A training-free acceleration method tailored for diffusion transformers, 2024b. URL https: // /twpiviivvvv./////v//aa///aaaava org/ abs///b///ae 2406. 01125.
Jabri, A., Fleet, D., and Chen, T. Scalable adaptive compu- tation for iterative generation, 2023.
Choi, Y. Uh, Y., Yoo, J., and Ha, J.-W. Stargan v2: Diverse image synthesis for multiple domains, 2020.
Kang. M., Zhang, R., Barnes, C., Paris, S., Kwak, S., Park, J., Shechtman, E., Zhu, J.-Y,, and Park, T. Dis- tilling difffusion models into conditional gans, 2024. URL https://arxiin www.awwwwww...oraaa..aa..aoreere abs wwt.o////g........////// 221.224055..559
Deng, J., Dong, W. Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vi- sion and Pattern Recognition, pp. 248-255, 2009. doi: 10.1109/CVPR22009 5206848.
Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks, 2019.
Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., and Aila, T. Training generative adversarial networks with limited data, 2020a.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis, 2021.
Dieleman, S.
Perspectives on diffusion, 2023.
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. Analyzing and improving the image quality of stylegan, 2020b.
URL https: http:///aanderaaa/e ai/2023 07/20/ perspectiven html.
9

---

Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models, 2022.
Lu, C. and Song. Y. Simplifying, stabilizing and scaling continuous-time consistency models, 2025. URLhttps.
/ //acxi...rrg/bbb2211..111
Kim, D., Lai, C.-H., Liao, W.-H., Murata, N., Takida, Y., Uesaka, T., He, Y., Mitsufuji, Y., and Ermon, S. Consis- tency trajectory models: Learning probability flow ode trajectory of diffusion, 2024a.
Lu, C., Thou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm- solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps, 2022a.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps, 2022b. URL https:
Kim, G., Kim, B., Park, E., and Cho, S. Diffusion model compression for image-to-image translation, 2024b. URL httpss:/aat /arpiv//a/vv////aa//ai/iiiiaiiia org/abs//////oooooo.gg...o//..rr 2/////111.111 17547.
/ //arxiv. org/ abs 22062226.2222 00927.
Krizhevsky, A. Learning multiple layers of features from tiny images. 2009.
Luo, S., Tan, Y., Patil, S., Gu, D., von Platen, P., Passos, A., Huang, L.., Li, J., and Zhao, H. Lem-lora: uni- versal stable-diffusion acceleration module, 2023. URL https: / www.v.oor//a...eee. 2.1..b//5......
Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved precision and recall metric for assess- ing generative models, 2019.
Ma, X., Fang, G., and Wang, X. Deepcache: Acceler
ating diffusion models for free, 2023. URL https:
Kynkäänniemi, T., Karras, T., Aittala, M., Aila, T., and Lehtinen, J. The role of imagenet classes in fréchet in- ception distance, 2023. URL https:://// //tpvv.ogx//g/////// abs/2203. 06026.
Mescheder, I.., Geiger, A., and Nowozin, S. Which training methods for gans do actually converge?, 2018.
Lee, S., Jung. S.-W., and Seo, H. Spectrum translation for refinement of image generation (stig) based on contrastive learning and spectral filter profile, 2024. URL https: //arxiv. org/ abs/ 2403. 05093.
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., MeGrew, B., Sutskever, I., and Chen, M. Glide: To- wards photorealistic image generation and editing with text-guided diffusion models, 2022.
Li, J., Cao, J., Zou, Z., Su, X., Yuan, X., Zhang, Y., Guo, Y., and Yang, X. Distillation- free one-step diffusion for real-world image super-resolution, 2024a. URL https: //arxiv. org/ aww//////12 2410. 04224.
Peebles, W. and Xie, S. Scalable diffusion models with transformers, 2023.
Ramesh, A., Dhariwal, Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents, 2022.
Li, L., Li, H., Zheng, X., Wu, J., Xiao, X., Wang, R., Zheng, M., Pan, X.. Chao, F., and Ji, R. Autodiffusion: Training-free -free optimization of time steps and architectures for automated diffusion model acceleration, 2023a. URL https: / /aaxivivvvvv org/ abs 93.9.30100 10438.
Rissanen, S., Heinonen, M., and Solin, A. Generative modelling with inverse heat dissipation, 2023. URL
https::/arxxiv.rogg/ abs/ 220/ 13397.
Saharia, C., Chan, W., Saxena. S., Li, L., Whang, J., Den- ton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., Salimans, T., Ho, J., Fleet, D. J., and Norouzi, M. Photorealistic text-to-image diffusion models with deep language understanding. 2022.
Li, S., Hu, T., van de Weijer, J., Khan, F. S., Liu, T., Li, L., Yang. S., Wang, Y., Cheng, M.-M., and Yang, J. Faster diffusion: Rethinking the role of the encoder for diffusion model inference, 2024b. URL https::tp/// httpp:s///axxvoo/o/ www.i.orgg abs/2312. 09608..
Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models, 2022.
Li, T., Chang, H., Mishra, S. K., Zhang, H., Katabi, D., and Krishnan, D. Mage: Masked generative encoder to unify representation learning and image synthesis, 2023b.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans, 2016.
Lin, S., Wang, A., and Yang. X. Sdxl-lightmin:: Progressive adversarial diffusion distillation, 2024.
Sauer, A., Schwarz, K., and Geiger, A. Stylegan-xl: Scaling stylegan to large diverse datasets, 2022.
Lin, T.-Y., Maire, M., Belongie, S., Bourdev, I.,, Girshick, R., Hays, J., Perona, P., Ramanan, D., Ziticck, C. L., and Dollár, P. Microsoft coco: Common objects in con- text, 2015. URL https: // arxii. org.org///bo... 0312.
Sauer, A., Lorenz, D., Blattmann, A., and Rombach, R. Adversarial diffuuson distillation, 2023. URL https: / //aaxiivorg/aabaa. 17042.
aaxxiv.orgabs//14.
10

---

Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
Sauer, A., Boesel, F., Dockhorn, T., Blattmann, A., Esser, P., and Rombach, R. Fast high-resolution image synthesis with latent adversarial difffusion distillation, 2024. URL https: arxii. org/ abs/ 2403. 12015.
Xu, Y., Zhao, Y., Xiao, Z., and Hou, T. Ufogen: You forward once large scale text-to-image generation via diffusion gans, 2023. URL abs 2ts/2311.0922
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition, 2015.
Xue, S., Liu, Z., Chen, F., Zhang, S., Hu, T., Xie, E., and Li, 7. Accelerating diffusion sampling with optimized time steps, 2024. URL https: /ttp///axvvvv/vv org wwooorrg/aaa... / 2402.1737
Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics, 2015.
Yin, T. Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., and Park, T. One-step diffusion with distribution matching distillation, 2023.
Song. J., Meng, C., and Ermon, S. Denoising diffusion implicit models, 2022.
Yin, T., Gharbi, M., Park, T., Zhang, R., Shechtman, E., Durand, F., and Freeman, W. T. Improved distribution matching distillation for fast image synthesis, 2024. URL https:/ wwwaaaaivvvv........... org abs 2405 14867.
Song, Y. and Dhariwal, P. Improved techniques for training consistency models. ArXiv, abs/2310.14189, 2023. URL http:///ppp.semancccccccoooooo/ htt//CoopuuID:2242222
Zhang, R., Isola, P., Efros, A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as perceptual metric, 2018.
Song. Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution, 2020a.
Song, Y. and Ermon, S. Improved techniques for training score-based generative models, 2020b.
Zhao, S., Liu, Z., Lin, J., Zhu, J.-Y, and Han, S. Differen- tiable augmentation for data-efficient gan training, 2020.
Song. Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Rcsre-based generative modeling through stochastic differential equations, 2021.
Zhao, W., Han, Y., Tang, J., Wang, K., Song, Y., Huang, G., Wang, F., and You, Y. Dynamic diffusion trans- former, 2024. URL https: http:s///axvvv.rrg////// 2410 03456.
Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. Consis- tency models, 2023.
Zheng. H., Nie, W., Vahdat, A., Arizzadeneshelii K., and Anandkumar, A. Fast sampling of diffusion models via operator learning, 2023.
Song. Y., Lorraine, J., Nie, W., Kreis, K., and Lucas, J. Multi-student diffusion distillation for better one-step generators, 2024. URL https:: //appi://axxvv.../ org/ abs/ 2410.22 23274..
Zhou, M., Zheng, H., Wang. Z., Yin, M., and Huang, H. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation, 2024a.
Takida, Y., Imaizumi, M.. Shibuya, T., Lai, C.-H., Ue- saka, T., Murata, N., and Mitsufuji. Y. San: Inducing metrizability of gan with discriminative normalized lin- ear layer, 2024. URL /tttps:/aaxiv... wwt.://iiioooooooooooooooooo/ee abs/ 2301. 12811.
Zhou, Z., Chen, D., Wang, C., and Chen, C. Fast ode-based sampling for diffusion models in around 5 steps, 2024b. URL https: http://wwwwx./a/aa////a 223 2. 00094.
Zhou, Z., Chen, D., Wang. C., Chen, C., and Lyu. S. Simple and fast distillation of diffusion models, 2024c. URL https: org/ abs /20.
Tan, M. and Le, Q. V. Efficientnet: Rethinking model scaling for convolutional neural networks, 2020.
Touvron, H., Cord, M., Douze, M.. Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image trans- formers & distillation through attention, 2021.
Zou, C., Liu, X., Liu, T., Huang, S., and Zhang, L. Ac- celerating diffusion transformers with token- lokenwisse feature caching, 2024. URL https: //t/xiivvvv//////////////////avv www.riii.g../.ooeooo orgg/bs/ 2410 2410.05317.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need, 2023.
Wang, J., Wu, S., Xu, K., and Yuan, Z. Frequency com- pensated diffusion model for real-scene dehazing, 2023. URL https:ttp/// //axss/////vvvv www.wwww.......eee.. abs/2308 10510.

---

Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
A. Related Work
Image Generation GAN (Goodfellow et al., 2014; Brock et al., 2019; Karras et al., 2020b) models have dominated the image generation field for a long time. Quantization-base generative models (Esser al., 2021: Chang et al., 2022; Li al., 2023b) first encode images into discrete tokens, then use transformer to model the probability distribution between et al., 2015; Song & Ermon, 2020a; Ho et al., 2020; Song & Ermon, 2020b;
the tokens. Diffusion Models Song et al., 2021) or score-based generative models try to learn an accuracy estimation of scores (the gradient of the log probability density) to sample from a perturbed distribution with a Gaussian kernel to the image distribution. Diffusion models have achieved great success in image generation (Dhariwal & Nichol, 2021; Nichol et al., 2022; Ramesh et al., 2022; Saharia et al., 2022) and are widely used in different fields.
Accelerating Diffusion Inference Many recent works tried to accelerate the inference process of diffusion models, often focusing on the redundancy inherent in these models. This typically involves both architectural and temporal redundancy (Ma et al., 2023; Li et al., 2023a; Zou et al., 2024; Li et al., 2024b; Zhao et al., 2024; Chen et al., 2024a; Kim et al., 2024b; Chen et al., 2024b). These methods often involve reusing feature maps or outputs from neighboring time steps to reduce the computational cost of inference.
Diffusion Distillation One of the challenges of diffusion models in practice is the high computational cost incurred during fine multi-step generation. A series of distillation methods (Salimans Ho, 2022: Gu et al., 2023; Song et al., 2023; Yin al., 2023; Zhou et al., 2024a;e; Hsiao al., 2024) have been proposed to distill diffusion models one-step generators. Distillation models with GAN were introduced recently. GAN loss has been used as an auxiliary loss of distillation loss (Sauer et al., 2023; Xu et al., 2023; Kang et al., 2024; Sauer et al., 2024; Li et al., 2024a; Kim et al., 2024a), or a replacement of instance- level loss (L22 or LPIPS) as in (Lin et al., 2024). Both of these approaches achieved good performance, demonstrating the potential of GAN-based distillation.
Frequency Perspective Recently, several studies investigated diffusion models from frequency perspective. For example, in (Rissanen et al., 2023), the authors analyze the changes induced by the diffusion forward process on images from a frequency viewpoint. In (Lee et al., 2024), analysis was conducted based on the frequency differences between inputs and outputs. In (Wang et al., 2023; Huang et al., 2024), a frequency- based improvement method was proposed, enhancing the model's s efficiency and performance.
B. Definition of Typical Distillation Methods
Progressive Distillation (PD) uses progressive distillation strategy. Using Sp with a sufficient number of steps, e.g., Nmaz = 1024, as a teacher, PD first distills a student model Fa with fewer steps, typically N₁ = _Naaz 512. Then with the fixed student model F as teacher where sg means stop gradient, PD further distills the student with N2 = Naax = 256. This progress is repeated until a one-step generator is produced.
Consistency Distillation (CD) and Consistency Trajectory Model (CTM) are similar. Both of these two methods adopt a joint training strategy in which student model F, is used to be part of the teacher model H( 4, u, 8) F (S ( L, u), u, 8) the same time. CTM uses an iterative solver S* & here and it further projects U. and G, to the pixel space by F (Va, 8, miin) and F (ū,, s, min) and then calculates D (Uuosan Do...) ) as loss. The three models are summarized in Table 7, 8 and 9, We modify the definition of PD with a EDM manner here to be consistent. L, u, 8 are the start, intermediate and end time steps of the trajectory, respectively. PD sets the u to the "middle" of and s. CD sets the u as the successor of t and fixes the s to zeros. PD uses a separate training strategy since the teacher model is student model from previous training, and PD supervises fo with the intersection of æ 0 and (lope Ug), where Slo U.) the slope between 24 and Ug.
Table Sampling strategies for time steps of commonly used distillation methods
Method
s To.N)
PD
T(i,N, 2 T((+)),))
CD
T(i,N), Si 1,
T(O, N)
CTM
T(1,), 2i TUNN), T(KN)Okks)
12

---

Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
Table 8. Loss function of commonly used distillation methods
Method
Loss Function (L)
PD CD
D (xe, t), r-s (H Ct. t, u, s) xt) + x+)
(Tt 8), H( L u, s)
CTM
D (Fo (, s),8 8, mmin), F (H ly u, 8), 8, mmin))
Table 9. Definition of teacher and student model of commonly used distillation methods
Method
Student Model (Fa)
Teacher Model (H)
S (Sa( u). F (F U,,
N; Nmaz
PD
(Fo(ar,, f) x1) +
N: 7 Nouz
CD
(La(ar, 1) x2) + +.
(S,(,, u), u,
CTM
t,
(S u), u)
C. Implementation Details
We EDMs the original models for all datasets and methods ensure fair comparison. EMA decay applied the weights of the generator for sampling following previous work. EMA decay rate is calculated by EMA Halflife and EMA warmup is used as in EDM. EMA warmup ratio is set to 0.05 for all datasets, and this leads to a gradually growing EMA decay rate. We use Adam optimizer with 31 = 0, .2 = 0.99 without weight decay for both the generator and the discriminator. No gradient clip is applied. For conssicona generation, we use no classifier guidance but simple class embedding in the discriminator following StyleGAN2-XL. Mixed precision training is adopted for all experiments with BF data type, and results with Float and Float32 are similar. We apply no learning rate scheduler. Images are resized to 224 first before they fed the PG discriminator or CD-LPIPS loss. All models are trained NVIDIA A 100 GPUs. Hyperparameters used for D2O and D2O-F training can be found in Table 10. D20 and D20-F share the same hyperparameters except for the number of training images, since D20-F will not overfit when D20 gets worse FID.
slassifier-uuidanee
cluster of
Table 10. Hyperparameters used for training D20 and D20-F
Hyperparameter
CIFAR-10
AFHQ 64 64
FFHQ 64 64
ImageNet 64 x 64
G Architecture
NSCN++
NSCN+
NSCN++
ADM
G LR
lc-4
2c-5
2c-5
6
D LR
lc-4
4c-5
4c-5
4c-5
Optimizer
Adam
Adam
Adam
Adam
31 of Optimizer
0
0
82 of Optimizer
0.99
0.99
0.99
0.99
Weight decay
No
No
No
No
Batch size
256
256
256
512
Or (Regularization)
le-4
le-4
le-4
4e-4
EMA half-life (Mimg)
0.5
0.5
0.5
50
EMA warmup ratio
0.05
0.05
0.05
0.05
5 (D20)
(D20)
5 (D20)
(D20)
Training Images (Mimg)
5 (D20-F)
10 (D20-F)
10 (D20-F)
5 (D20-F)
Mixed-precision (BF16)
Yes
Yes
Yes
Yes
Dropout probability
0.0
0.0
0.0
0.0
Augmentations
No
No
No
No
Number of GPUs
8
8
13

---

Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
We follow the parameterization in Consistency Model for all of our models:
O
J (1
Cskip (t)
data
Cout (t)
(t +)²
I 1
data
We set / - maz when applying D20/D20-F with Cakip R O and Cout R 1. When training with CD loss, we use Heun solver. The time step scheduler is slightly different from the scheduler defined in EDM. We reverse the index and define as a function of the current step and total steps:
Ti,,N) (Omas (1- n (Tmin Tmax" )))
"
Following Consistency Distillation, we 18, == 7, Omin 0.002 and Omaz = 80.
D. CLIP-FID Results
Potential data leakage in FID when using discriminator pre-trained on ImageNet has been a concern (Kynkäänniemi et al., 2023). We provide CLIP-FID in Table D. Our method consistently shows superior or competitive performance with significantly less training data. The result indicates that the superior performance not due to information leakage but the pretrained ability gained in diffusion training.
Table 11. Performance comparison across datasets: CLIP-FID, FID, and training images
Dataset
Model
CLIP-FID
FID
Training Images
CIFARIO
EDM
0.53
1.98
CD
1.26
4.10
100M
SiD
0.65
1.92
400M
D20-F
0.66
1.56
5M
FFHQ
EDM
1.18
2.39
SiD
0.80
1.55
500M
D20-F
0.81
0.83
9M
AFHQv2
EDM
0.40
1.96
SiD
0.32
1.62
300M
D20-F
0.18
1.24
7M
ImageNet
EDM
0.82
2.64
CD
2.93
6.87
1000M
SiD
0.75
1.52
930M
D20-F
0.51
1.13
-6
E. Details of Boock-Wise Frequency Analysis
We first notice natural correlation between the U-Net architecture and spatial frequency (Figure 6A.). The deeper layers, which operate lower resolutions, are limited in their ability to process high-frequency information. The shallower layers, operating higher resolutions, are better equipped to capture broader range of high-frequency information.
Next, we examine how frequency contributions evolve over time across layers and their alignment with the overall frequency response in the diffusion process. We calculate the frequency difference between the input and output of each layer at different time steps D = DFET (xt, h-11,, ) where refers to the time step and b refers to the block index. There are significant specialization patterns of different blocks (Figure B. and C.).
14

---

Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
dec,64x64 block1
B.
Time Stape
Log Difference
0.4
0.3
0.2
0.1
5
25
Radius
dec.Bxa block1
C.
Log Difference
0.50
0.45
Tim: 57455
0.40
0.35
0.30
1.5
2.0
3.5
4.0
Radius
Figure A. Average feature maps at different depths. The inner region corresponds the skip path, while the outer region corresponds to the convolution path consisting of normalization, activation, and convolutional layers. From deeper to shallower layers, the decoder exhibits a transition from low to high feghffreuuenyy contributions the spatial domain. Radial averaging the log difference between the output and input of shallow layer reveals that the gigh-frequency component increases over time. C. Radial averaging of the log difference between output and input of deep layer. Due the lower resolution, this layer exhibits smaller radius and frequency range.
F. Training Efficiency Comparison
We provide further training efficiency comparisons between methods and the other approaches ImageNet 64 64. The name in the quote is the pre-trained diffusion model. With similar teacher models (EDM and EDM2), our methods require significantly fewer training images while delivering better performance.
Table 12, Comparison of distilled one-step generators on ImageNet 64x
Method
FID |
Training Images (M)
Params (M)
BOOT (EDM)
16.30
307
280
DMD (EDM)
2.62
117
280
ECM-S (EDM2)
5.51
12
280
ECM-S, longer training (EDM2)
4.05
102
280
ECM-XL (EDM2)
3.35
12
1119
ECM-XL, longer training (EDM2)
2.49
102
1119
sCD-S (FDM2+TrriFlow)
2.97
819
280
sCD-XL. (EDM2+TrigFlow)
2.44
819
1119
D20-F (EDM)
1.16
280
G. Samples
We include EDM's results for comparison. The initial noises are the same as all models. D20 and D20-F generate images similar to the original model (EDM). But they are not identical, because our methods allow the student model to perform differently from the teacher model.
15

---

Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
Figure CIFAR-10, EDM, NFE=18, FID=1.96
Figure CIFAR-IO, 10, D20, NFE=1, FID=1.66
Figure 9. CIFAR-10, D20-F, NFF=1, FID=1.54
16

---

Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
Figure 10. CIFAR-10 (conditional), EDM (VE), NFE=18, FID=1.82
Figure 11. CIFAR-10 (conditional), D20, NFE=1, FID=1.58
Figure 12. CIFAR-10 (conditional), D20-F NFE=1, FID==1.4
17

---

Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
Figure 13, AFHQv2 64x64, VE, NFE=79, FID=2.17
Figure AFHQV2 64x64, D20, NFE=1, FID=1.23
Figure 15. AFHQV2 64x64, D20-F NFF=1, FID=1.31
18

---

Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
Figure 16. FFHQ 64x64, EDM (VE), NFE=79, FID=2.60
Figure 17. FFHQ 64x64, D20, NFE=1, FID=1.08
Figure 18. FFHQ 64x64, D20-F NFE=1, FID=0.85
19

---

Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
Figure 19. ImageNet 64x64 (conditional), EDM (VE), NFE=79, FID=2.36
Figure 20, ImageNet 64x64 (conditional), D20, NFE=1, FID=1.42
BABI 8.0
D
Figure 21. ImageNet 64x64 (conditional), D20-F, NFF=1, FID=1.16
20


Reviews:
[{"review_id": "5vk2fpjns5", "reviewer": "Reviewer_EGMJ", "summary": "The paper proposes a novel approach, D2O (Diffusion to One-Step), which uses a GAN objective to convert diffusion models (DMs) into efficient one-step generators. It identifies a key issue in previous distillation methods: the teacher and student models' distinct local minima, which enables effective knowledge transfer. The paper argues that a standalone GAN objective can bypass this issue, enabling the diffusion model to be fine-tuned for one-step generation without relying on distillation losses. The authors introduce D2O-F, where most parameters are frozen during fine-tuning, further improving the method's efficiency. Through experiments on datasets like CIFAR-10 and ImageNet, they show that D2O and D2O-F achieve competitive performance with fewer training images compared to traditional methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and positive feedback. We acknowledge that our current work is primarily empirical. Nevertheless, we believe that our work provides important insights for the community, particularly regarding how to take advantage of the capabilities within pre-trained diffusion models. We validate this idea with experiments on the one-step generation task. We plan to strengthen the theoretical foundations and extend the method to more complex datasets.", "claims_and_evidence": "The claims are generally supported by clear and convincing evidence. The authors show that D2O and D2O-F achieve competitive results on several datasets (CIFAR-10, ImageNet, etc.) with much fewer training images (0.2M to 5M). The use of a GAN objective during fine-tuning is demonstrated to overcome the local minima problem in diffusion distillation while improving efficiency. The freezing experiment further validates the claim that diffusion models provide sufficient generative capabilities through pre-training, requiring minimal adjustments during fine-tuning. The comparison with previous distillation methods also supports the proposed approach's effectiveness.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria (FID, Inception Score, Precision, and Recall) are standards for generative modeling tasks. The use of multiple datasets (CIFAR-10, AFHQv2, FFHQ, ImageNet) provides a broad assessment of the model's performance.", "theoretical_claims": "The paper makes a claim that the local minima of teacher and student models differ significantly, challenging the distillation process. And it also claims that the GAN objective leads to better convergence compared to traditional distillation. No formal proof of these theoretical claims are provided, but the empirical results look reasonable to me.", "experimental_designs_or_analyses": "The experimental designs are sound, particularly the ablation study of the D2O-F model, which freezes most parameters and shows that this strategy enhances performance. The authors also test D2O and D2O-F on multiple datasets and tasks, confirming their results across different settings. As the authors acknowledged, however, the freezing method's effect on more complex architectures (e.g., DiT) and higher-resolution datasets has not been tested, which might limit the generalizability of the findings.", "relation_to_broader_scientific_literature": "The paper positions itself as an advancement over existing methods by addressing the limitations of distillation, providing an efficient method for one-step generation. The relationship to GAN-based distillation methods (e.g., Sauer et al., 2023) is also highlighted. The proposed approach offers a promising alternative to multi-step distillation in generative modeling, which is a current challenge in the field.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "By using the freezing strategy (freezing most of the pre-trained parameters during fine-tuning), the paper demonstrates that you can achieve competitive performance with much fewer training steps (as few as 0.2M training steps) compared to traditional methods, which require tens or even hundreds of millions of training steps. \n\nThe NFE results are mentioned in the tables, but there's a lack of detailed analysis on them or discussion regarding the inference gains.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A\n\nUpdate after rebuttal: After reviewing the rebuttal discussions, I would keep the original rating.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vkoOSKMPka", "reviewer": "Reviewer_BcZC", "summary": "This work proposes a method called D2O that fine-tunes a pretrained diffusion model for one-step generation with GAN loss. Pretrained VGG-16 is used as discriminator and the specific GAN loss objective used for the discriminator is Projected GAN (Sauer et al. 2022). In addition, they use other techniques like regularization and normalization to further stabilize training. In addition, they also propose D2O-F that freezes 85% of the parameters in convolutional layers during fine-tuning. This results in a good single-step generator. In addition, the paper also presents some interesting analysis on how different frequencies are processed in diffusion model both within the UNet architecture and during sampling.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and constructive feedback. Here, we provide a point-by-point rebuttal, which we hope helps to clarify the confusion:\n\n**Claims And Evidence:**\n\n1. \"Writing and overall structure…\"  \n   - \"Consider the following two statements about the use of augmentation…\"  \n      Sorry for the confusion, but there’s no inconsistency here. We first used diffAug as the baseline’s default setting initially. Since the ablation experiments showed that augmentation led to worse performance in our GAN post-training pipeline, we chose not to use augmentation in the rest of the experiments.  \n   - \"Contributions are unclear…\"  \n      As you noted, this paper’s main aim is to develop a method for one-step image generation. The frequency analysis in Section 4 is to provide an intuition for understanding why one-step generation with the frozen weights works. We agree that it could belong to Supplementary, but we feel that including it provides a theoretical explanation for our approach, especially for those who do not have time for supplementary materials.   \n   - \"Many comparisons are not fair…\"  \n      The point of these comparisons is **not** to argue for a better training efficiency. Instead, they are to demonstrate that our models quickly learn one-step generations by adapting the pre-trained models’ innate capability (since learning from scratch needs far more data than what D2O-F needs), thereby supporting our claim that diffusion training can be viewed as a pre-training for generative capabilities. This seems to be a misunderstanding.  \n2. \"The experiment set up in Section 2.3…\"   \n   Thanks for the constructive criticism. As you correctly noted, our approach differs from traditional trajectory-alignment distillation approaches using L2/Huber losses.   \n   However, training a multi-step generator with a GAN loss is entirely feasible. We simply allowed gradients to flow through the entire multi-step sampling process. By employing the same loss function for both teacher and student, the results showing the mismatch between the two (Fig. 2\\) are more compelling.   \n   Moreover, our goal is not to show that GAN is superior to L2/Huber losses. Rather, we want to demonstrate that the teacher and student networks may converge to different local minima for the same generative task.   \n   In this experiment, if we used an L2/Huber loss for trajectory alignment, we would force the student to imitate the teacher, which undermines our goal of letting each model independently learn the target distribution and comparing their solution. Therefore, we used a pure GAN loss as a direct, efficient way to ensure both models optimize the same ultimate objective without additional constraints.   \n   Finally, as Fig. 2 supports our hypothesis. Forcing students to replicate the teacher with a L2/Huber trajectory alignment loss could lead to inefficient and performance degradation.  Our central idea proposes to bypass traditional trajectory-based sampling or distillation and to directly tune a pre-trained diffusion model with a simpler objective. Thus, choosing pure GAN is reasonable, which is further solidified by the superior performance of D2O-F. We also tested training with additional L2 (CD) loss (Section 4.4), which did not improve the performance. \n\n**Methods and Evaluation Criteria：** \t\t\t\n\n1. We admit that the CLIP-FID comparisons in the appendix are limited due to resource constraints. However, we believe that SiD (a near-SOTA method with solid theory and efficiency claims) is a strong baseline to demonstrate our method’s efficiency and prove that our method’s good performance is not due to FID leakage.  \n2. The results in Tables 4 and 5 used 4-7 M training images, depending on the dataset.  \n3. Here, we provide further evidence as requested, with more training images and one-step FID comparison between D2O-F and competing methods on ImageNet 64x64. The name in the quote is the pre-trained diffusion model.\n\n   | Methods | FID | Training Images (Millions) | MParams |\n   | :---- | :---- | :---- | :---- |\n   | BOOT (EDM) | 16.3 | 307  | 280 |\n   | DMD (EDM) | 2.62 | 117  | 280 |\n   | ECM-S (EDM2)  | 5.51 | 12  | 280 |\n   | ECM-S\\* (EDM2)  | 4.05 | 102 | 280 |\n   | ECM-XL (EDM2) | 3.35 | 12  | 1119 |\n   | ECM-XL\\* (EDM2) | 2.49 | 102  | 1119 |\n   | sCD-S (EDM2+TrigFlow) | 2.97 | 819 | 280 |\n   | sCD-XL (EDM2+TrigFlow) | 2.44  | 819 | 1119 |\n   | **D2O-F** (EDM) | **1.16** | **5** | 280 |\nAgain, we’d like to thank the reviewer for the time and effort. With the additional results and the clarification, we sincerely hope the reviewer would re-evaluate the paper accordingly.", "claims_and_evidence": "1. Writing and overall structure of the paper needs to be improved, as there are instances of contradictory statements. There's also slight overclaiming at certain places which needs to be toned down.\n- Consider the following two statements about use of augmentation. There’s clearly some logical disconnect here. \n    1. Line 217: We adopt differentiable augmentation (diffAug) without a gradient penalty by default…\n    2. Line 190-191 (second column): we find that It leads to poor results in our method… we disable all augmentation in all of our further experiments. \n- Contributions are unclear - The initial sections of this paper are devoted towards a method for single step image generation however, towards the end of Section 4, the focus shifts towards understanding how different frequencies are processed during sampling in diffusion models, as well as within the UNet architecture. While the frequency analysis is useful, it feels like a tangential direction to the main goal of this paper, which is single-step image generation.  \n- Many comparisons are not fair. \n    1. “D2O-F produces satisfying images with as few as 0.2 million training steps (FID=4.12). It reaches near SOTA performance by only 5 million steps (FID=1.54). In contrast, training a generative model with similar per- formance typically requires tens or hundreds of millions of training steps (100M for StyleGAN2-ADA, 200M for EDM, on CIFAR-10)” This statement is not fair because D2O-F uses a pre-trained diffusion model (in fact EDM), and the training budget for pretraining needs to be accounted. Similar statements have been made at many other places in the paper. These statements need to be toned down.\n2. The experimental set up in Section 2.3 is confusing. The experiment in Section 2.3 is used to argue that teacher and student models converge to different minima in traditional distillation methods, and that the student models fail to match few step teacher predictions. However, only GAN loss is used for fine-tuning both the teacher and student model in these experiments (as specified in Lines 139-140). GAN loss is more suitable for one-step generation and it is unclear why the authors used GAN loss for multistep sampling set up. It is also unclear how the multistep teacher models were trained with only GAN loss. Further, sweeping arguments across all distillation methods have been made based on the results obtained from these fine-tuned models while the set up is quite different from that of traditional distillation methods (See beginning of Section 3.1 for instance). Both the loss landscape and gradient descent dynamics will be quite different for purely GAN-based fine-tuning of pre-trained diffusion models compared to the MSE/Huber loss based distillation methods. Also, if using only GAN loss leads to different minima, as specified in Section 2.3, why would authors still choose to train purely on GAN loss anyway (See Section 3.3)?", "methods_and_evaluation_criteria": "There are two primary set of results presented in this work.\n\nThe first set of results on comparison of quality of images generated by D2O (in terms of FID as well as NFE for efficiency) against the prior methods is detailed. The authors have compared against well-known distillation methods, though I would encourage the authors to add a subset of more recent consistency-based methods like iCT [1], sCT [2], ECT[3], TCM[4] etc. some of which fine-tune from a pretrained diffusion model and might be a more fair comparison. The authors also include results on CLIP-FID in the appendix. \n\nThe second  set of results on training sample efficiency feels a bit incomplete. The comparison of training sample efficiency has been made against Consistency Distillation (CD) and SiD which feels a bit incomplete. More precisely, Section 4.1 states: “The amount of data used to train D2O is substantially smaller than that used in most prior distillation methods“ - More evidence is needed for this, especially comparison with the prior distillation based methods e.g. DMD, BOOT, sCM [1], ECT [2], etc, if that data is available. \n\n[1] Song, Yang, and Prafulla Dhariwal. \"Improved techniques for training consistency models.\" arXiv preprint arXiv:2310.14189 (2023).\n[2] Lu, Cheng, and Yang Song. \"Simplifying, stabilizing and scaling continuous-time consistency models.\" arXiv preprint arXiv:2410.11081 (2024).\n[3] Geng, Zhengyang, et al. \"Consistency models made easy.\" arXiv preprint arXiv:2406.14548 (2024).\n[4] Lee, Sangyun, et al. \"Truncated Consistency Models.\" arXiv preprint arXiv:2410.14895 (2024).", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental designs make sense. I have listed some avenues for improvement of the experimental set up under the previous questions.", "relation_to_broader_scientific_literature": "There's growing demand for efficient one-step image generators that can generate high quality images quickly. Potentially, the findings here can be extended to videos and other modalities as well.", "essential_references_not_discussed": "Related work for consistency models can also be added in the appendix. Many of the recent methods are missing. I have listed these methods above. Check my previous responses for exact references.", "other_strengths_and_weaknesses": "Strengths:\n1. The observation that a (smartly) frozen UNet can be fine-tuned efficienctly with only GAN loss to get good image quality is interesting and beneficial.\n2. The frequency analysis of the diffision sampling process as well as the UNet blocks presented both in the main paper and the appendix is interesting (but also seems a bit unrelated to the main problem of one step generation).\n\nWeaknesses: (Please check my previous responses for a detailed feedback on these points.)\n1. Quality of writing and overall structure needs improvement. \n2. Some baselines are missing and can be added.", "other_comments_or_suggestions": "How many fine-tuning steps were used for D2O and D2O-F in Table 4 and 5? Perhaps, this can be indicated in a pair of brackets in the table for easy reference.", "questions_for_authors": "Is there a typo in Section 2.3 in Lines 139-140 about how the teacher models were fine-tuned with GAN loss for multistep sampling?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aGe7ceed1l", "reviewer": "Reviewer_atu8", "summary": "This work observes that distillation-based training of diffusion models may result in a mismatch of local minima between the student and teacher models. Additionally, it demonstrates that employing a standalone GAN objective, without a distillation objective, is sufficient to transform diffusion models into efficient one-step generators.\n\n## update after rebuttal\n\nI appreciate the authors for their clarifications. After a thorough review of the work and the authors' responses, I have raised my scores. However, I believe that the presentation of the paper could be improved.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the comments. However, the reviewer may have overlooked some important content of our paper. Below is the point-by-point response.\n\n**Summary:** \n\nOur work is not merely about finding \"different local minima\" or \"distilling diffusion using only a GAN,\" as the reviewer stated, which only covers the results from Sections 2 and 3. More important contributions are detailed in Sections 4 and 5:\n\n1. To explain the efficiency of training with a GAN objective, we hypothesize that diffusion training provides a powerful generative capacity and can be viewed as a form of generative pre-training. Under this view, we can bypass iterative trajectory-based sampling/distillation, and a more direct post-training approach can transform diffusion models into one-step generative models with high efficiency (Sections 4.1-4.4).  \n2. We validate this hypothesis by freezing most of the diffusion model’s parameters during training, requiring only minimal fine-tuning (on the order of 0.2M training images). This indicates that our methods leverage the generative pre-training capacity of diffusion models (Sections 4-5).  \n3. Finally, we explore why diffusion models have this general generative capacity with a preliminary frequency analysis, showing they exhibit distinct frequency-specific patterns (Sections 4.5)\n\n**Claims And Evidence:**\n\nThe primary concern about the \"optimization landscape\" appears to reflect a misunderstanding of the results presented in Section 2\\. As shown in Figure 2, an increase in the teacher network’s parameters or sampling steps consistently leads to a wider gap in FID scores. This widening gap clearly indicates growing divergence between the learned mappings, suggesting increasingly divergent local optima. This empirical evidence strongly supports our claims of distinct local minima.\n\nFurthermore, Section 2 only serves as an inspiration for us to develop the one-step generation model, which is presented mostly in Sections 4 and 5\\. We would like to ask the reviewer to give more consideration to these sections when evaluating the paper.\n\n**Theoretical Claims:**\n\nThe reviewer stated that the paper did not have theoretical results. However, we presented theoretical analyses in Section 4.5, which explored the potential mechanism that allows one-step training in our model.\n\n**Supplementary Material:**\n\nWe kindly request the reviewer to re-examine this statement: \"The authors did not attach supplementary material, although I did browse through the appendix.\" Detailed supplementary material is indeed included in the manuscript. This is confirmed by other reviewers:\n\n1. Reviewer BcZC: \"Yes, I have reviewed all parts of Supplementary material.\"  \n2. Reviewer EGMJ: \"Yes, checked the implementation details and additional visualizations.\"\n\n**Other Strengths And Weaknesses:**\n\n1. \"Since the proposed…\"  \n   This is precisely the main point of our paper, which is mainly explained in Sections 4 and 5, but also stated clearly in the **Introduction (Line 53)**.  \n     \n2. \"GAN training is inherently unstable…\"  \n   We discussed this point in **Section 4.2 (Line 249)**: \"Notably, with the majority of parameters in both the discriminator and generator frozen, the training process of D2O-F is stable with minimal instances of mode collapse. Therefore, the freezing method circumvents the inherent instability of using GANs.\"  \n3. \"In addition, the FID evaluation of the method is questionable…\"  \n   We did provide additional CLIP-FID evaluations in **Appendix D** to address this point, which were positively acknowledged by other reviewers.\n\nIn summary, we appreciate the reviewer's valuable feedback but would like to kindly request the reviewer to consider re-evaluating the paper and incorporating the analyses and results that were unfortunately overlooked.", "claims_and_evidence": "The claim that a student model's performance is degraded compared to the teacher when using only a distillation loss is well established in the distillation literature [1,2]. However, the FID plots in Figure 2 do not provide strong empirical support for the assertion regarding the \"optimization landscape\". Specifically, the statement—\"We speculate that this may produce different optimization landscapes and different local minima between the teacher and student model\"—lacks direct evidence from the presented results. Moreover, does the phenomena hold for other distillation-based methods?\n\n\n[1] Sauer et. al., Adversarial Diffusion Distillation\n\n[2] Kim et. al., Consistency Trajectory Model", "methods_and_evaluation_criteria": "The paper employs standard evaluation metrics commonly used in image generation, including FID, IS, and Precision/Recall scores.", "theoretical_claims": "The paper does not have theoretical results.", "experimental_designs_or_analyses": "The overall validity of the experimental design appears valid and reasonable.", "relation_to_broader_scientific_literature": "This work proposes eliminating the distillation losses commonly used in diffusion distillation methods and relying solely on adversarial loss. The idea is straightforward, and I am concerned about the inherent and classic challenges associated with GAN training.", "essential_references_not_discussed": "Overall, the authors discuss essential references to some extent. However, I suggest including discussions on the following points:\n\n- The observation that GAN-distillation-based methods may require less training data was previously noted in [1] (see their Section 3.2). Additionally, this reference provides further arguments regarding GAN-based training on top of diffusion models.\n\n- While [1] is a concurrent work, it adopts a similar approach and extends the methodology to video generation. A conceptual discussion of this work would enhance the paper’s contextual positioning.\n\n[1] Ki, et. al., PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher\n\n[2] Lin et. al., Diffusion Adversarial Post-Training for One-Step Video Generation", "other_strengths_and_weaknesses": "- Since the proposed method relies solely on GAN loss for training, does this essentially reduce the entire pipeline to standard GAN training, with the pre-trained diffusion model merely serving as a better initialization? If so, could one achieve similar performance by initializing with a pre-trained GAN instead, assuming successful training?\n\n- GAN training is inherently unstable—it requires careful architectural choices and specialized training techniques, as also evidenced in Table 1. Even if the proposed method achieves reasonably good performance, it remains unconvincing compared to distillation-based or consistency training approaches, which offer more stability and theoretical grounding.\n\n- In addition, the FID evaluation of the method is questionable. It is well known that using a GAN loss with a discriminator pretrained on ImageNet significantly biases the FID metric [1]. I suggest evaluating the method using Fréchet distances with DINOv2, following the approach in the EDM2 paper, to provide a more reliable assessment of generative quality.\n\n- Diffusion models and their distilled variants naturally support classifier-free guidance (CFG) by leveraging conditional and unconditional score estimates during sampling. However, a GAN-only distillation framework raises questions about how conditional generation can be effectively incorporated--especially for text-to-image generations.\n\n\n[1] Kynkäänniemi, et al. The role of imagenet classes in fr'echet inception distance", "other_comments_or_suggestions": "Please refer to the comments above.", "questions_for_authors": "Please refer to the comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a novel and empirically validated approach—D2O and D2O-F—to convert diffusion models into efficient one-step generators via GAN-based fine-tuning, reframing diffusion pretraining as a powerful generative foundation. The authors identify and empirically support a key limitation in prior distillation-based methods: the mismatch of local minima between teacher and student models due to architectural and procedural differences. By eliminating the need for trajectory-aligned distillation losses and instead relying solely on adversarial objectives, the proposed method significantly reduces inference complexity while maintaining strong generative performance.

strengths:
* demonstrates state-of-the-art or near-SOTA FID scores on multiple datasets with drastically fewer training images.
* introduces a compelling perspective that diffusion training acts as generative pretraining.
* the freezing technique (D2O-F) shows that only minimal parameter tuning is needed to achieve high-quality results, enhancing stability and efficiency.
* reviewers appreciate the breadth of experiments, ablations, and the initial frequency analysis explaining the efficacy of one-step generation.

weaknesses:
* reviewers highlighted presentation issues (e.g., writing clarity, structural flow, some overclaiming), but the authors addressed these in the rebuttal and committed to improving them.
* clarifications were requested around training setups and comparison fairness, especially regarding pretraining budgets, which the authors provided in detail.
* reviewer atu8 raised concerns about generality, GAN instability, and evaluation biases. However, many points appear to stem from partial engagement with the full content, and the authors responded with clarifying evidence and results that were acknowledged by other reviewers.

While not without imperfections, the paper introduces a novel and promising method that offers both practical impact and conceptual insight into diffusion-based generative modeling. Therefore it would benefit the community to further discuss the work.
Decision: Accept (poster)