Improved Distribution Matching Distillation for Fast Image Synthesis
Authors: Tianwei Yin
Michaël Gharbi
Taesung Park
Richard Zhang
Eli Shechtman
Fredo Durand
William T. Freeman
Venue: NeurIPS 2024 oral (2024)
Keywords: Image Generation
diffusion based models
model distillation
Primary area: diffusion_based_models
TL;DR: We distill diffusion models into few-step generators that produce images with superior quality.

Abstract: Recent approaches have shown promises distilling expensive diffusion models into efficient one-step generators.
Amongst them, Distribution Matching Distillation (DMD) produces one-step generators that match their teacher in distribution, i.e., the distillation process does not enforce a one-to-one correspondence with the sampling trajectories of their teachers.
However, to ensure stable training in practice, DMD requires an additional regression loss computed using a large set of noise--image pairs, generated by the teacher with many steps of a deterministic sampler.
This is not only computationally expensive for large-scale text-to-image synthesis, but it also limits the student's quality, tying it too closely to the teacher's original sampling paths.
We introduce DMD2, a set of techniques that lift this limitation and improve DMD training.
First, we eliminate the regression loss and the need for expensive dataset construction.
We show that the resulting instability is due to the "fake" critic not estimating the distribution 
of generated samples with sufficient accuracy and propose a two time-scale update rule as a remedy.
Second, we integrate a GAN loss into the distillation procedure, discriminating between generated samples and real images.
This lets us train the student model on real data, thus mitigating the imperfect "real" score estimation from the teacher model, and thereby enhancing quality.
Third, we introduce a new training procedure that enables multi-step sampling in the student, and
addresses the training--inference input mismatch of previous work, by simulating inference-time generator samples during training. 
Taken together, our improvements set new benchmarks in one-step image generation, with FID scores of 1.28 on ImageNet-64×64 and 8.35 on zero-shot COCO 2014, surpassing the original teacher despite a 500X reduction in inference cost.
Further, we show our approach can generate megapixel images by distilling SDXL, demonstrating exceptional visual quality among few-step methods, and surpassing the teacher. 
We release our code and pretrained models.

Improved Distribution Matching Distillation for Fast Image Synthesis
Tianwei Yin'
Michaël Gharbi²
Taesung Park
Richard Zhang?
Eli Shechtman
Frédo Durand
William T. Freeman'
Massachusetts Institute of Technology
Adobe Research
http::/ffanveiyyghhbbb.dddd////
Abstract
Recent approaches have shown promises distilling expensive diffusion models into efficient one-step generators. Amongst them, Distribution Matching Distillation (DMD) produces one-step generators that match their teacher in distribution, i.e., the distillation process does not enforce one-to-one correspondence with the sampling trajectories of their teachers. However, to ensure stable training practice. DMD requires noise-imaae pairs, generated by the teacher with many steps of deterministic sampler. This is not only computationally expensive for large-scale text-to- image synthesis, but it also limits the student's quality, tying it too closely to the teacher's s original sampling paths. We introduce DMD2, set of techniques that lift this limitation and improve DMD training. First, we eliminate the regression loss and the need for expensive dataset construction. We show that the resulting instability is due to the "fake" critic not estimating the distribution of generated samples with sufficient accuracy and propose a two time-scale update rule remedy. Second, we integrate a GAN loss into the distillation procedure, discriminating between generated samples and real images. This lets train the student model on real data, thus mitigating the imperfect "real" score estimation from the teacher model, and thereby enhancing quality. Third, we introduce a new training procedure that enables multi-step sampling in the student, and addresses the training- inferrncc input mismatch of previous work, by simulating inference-time generator samples during training. Taken together, our improvements set new benchmarks in one- step image generation, with FID scores of 1.28 on ImageNet-64xtte and 8.35 on zero-shot COCO 2014, surpassing the original teacher despite 500x reduction in inference cost. Further, we show our approach can generate megapixel images by distilling SDXL, demonstrating exceptional visual quality among few-step methods, and surpassing the teacher. release our code and pretrained models.
additional regression loss computed using large set
1 Introduction
Diffusion models have achieved unprecedented quality in visual generation tasks (1-8). But their sampling procedure typically requires dozens of iterative denoising steps, each of which a forward pass through neural network. This makes high resolution text-to-image synthesis slow and expensive. To address this issue, numerous distillation methods have been developed to convert a teacher diffusion model into an efficient, few-step student generator (9)20). However, they often result in degraded quality, as the student model typically trained with a loss to learn the pairwise noise-to to-imae mapping of the teacher, but struggles to perfectly mimic its behavior.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Asha Ttat
7A
Figure 1: 1024x 1024 samples produced by our 4-step generator distilled from SDXL, Please zoom in for details.
Nevertheless, it should be noted that loss functions aimed at matching distributions, such as the GAN (21) or the DMD [22 loss, are not burdened with the complexity of precisely learning the specific paths from noise to image because their goal is align with the teacher model in terms of distribution-- minimizing either a Saanenn-h (JS) or an approximate Hallback-Leibler (KL) divergence between the student and teacher output distributions.
In particular, DMD [22 has demonstrated state-of-ther -het-rtt results in distilling Stable Diffusion 1.5, yet it remains less investigated than GAN-based methods (23 A likely reason is that DMD still requires an additional regression loss to ensure stable training. In turn, this necessitates creating millions of noise-image pairs by rumning the full sampling steps of the teacher model, which is particularly costly for text-to-image synthesis. The regression loss also negates the key benefit of DMD's unpaired distribution matching objective, because it causes the student's 's quality to be upper-bounded by the teacher's.
In this paper, we show how to do away with DMD's regression loss, without compromising training stability. We then push the limits of distribution matching by integrating the GAN framework into DMD, and enable few-steps sampling with novel training procedure, which we termed 'backward simulation'. Taken together, our contributions lead to state-of-the-art fast generative models that outperform their teacher, using as few as 4 sampling steps. Our method, which we call DMD2,

---

achieves tfate-ofthhe-att results in one-step image generation, setting a new benchmark with FID scores of 1.28 on ImageNet-64xie 64x64 and 8.35 on zero-shot COCO 2014. We demonstrate our approach's scalability by distilling from SDXI, to produce high-quality megapixel images, establishing new standards among few-step methods.
In short, our contributions are as follows:
We propose a new distribution matching distillation technique that does not require regression loss for stable training, thereby climinating the need for costly data collection, and allowing for more flexible and scalable training.
We show that training instability in DMD [22] without regression loss stems from an insufficiently trained fake diffusion critic, and implement a two time-scale update rule to address this issue.
We integrate a GAN objective into the DMD framework, where the discriminator is trained to distinguish samples from the student generator vs. real images. This additional supervision operates at the distribution level, which better aligns with DMD's distribution-matchinn philosophy than the original regression loss. It mitigates approximation errors the teacher diffusion model and enhances image quality.
While the original DMD only supports one-step students, we introduce a technique to support multi-step generators. Unlike previous multi-step distillation methods, we avoid the domain mismatch between training and inference by simulating inference- iffrrence-time generator inputs during training, thus improving overall performance.
2 Related Work
Diffusion Distillation. Recent diffusion acceleration techniques have focused on speeding up the generation process through distillation (9 10 |3 22
22 23 30). They typically train a generator approximate the ordinary differential equation (ODE) sampling trajectory of teacher model, in fewer sampling steps. Notably, Luhman et al. (6) precompute a dataset of noise and images pairs, generated by the teacher using an ODE sampler, and use it train the student to regress the mapping in single network evaluation. Follow-up works like Progressive Distillation [TO] 13 eliminate the need to precompute this paired dataset offline. They iteratively train a sequence of student models, each halving the number of sampling steps of its predecessor. A complementary technique, Instaflow (TT) straightens the ODE trajectories. so they are easier to approximate with one-step student. Consistency Distillation 19 26 31 32]. and TRACT (33), train student models so their outputs are self-consistent at any timesteps along the ODE trajectory, and thus consistent with the teacher.
GANs. Another line of research employs adversarial training align the student with the teacher at a broader distribution level. In ADD [23). the generator, initialized with weights from a diffusion model, trained using projected GAN objective with an mmage-space classifier (34). Building on this,
LADD [24] utilizes a pre-trained diffusion model as the discriminator and operates in latent space, thus improving scalability and enabling figher-resolution synthesis. Inspired by DiffusionGAN [28 29). UFOGen (25) introduces noise injection prior to the real vs. vs. fake classification in the discriminator, to smooth out the distributions, which stabilizes the training dynamics. However, purely GAN- based methods often struggle to integrate classifier-free guidance directly. For instance, LADD uses images with classifier-free guidance as real data in GAN discriminator. Other approaches combine adversarial objectives with distillation loss to preserve the original guided sampling trajectory. For instance, SDXL-Lightning (27) integrates a DiffusionGAN loss 255 with a
progressive distillation objective 13) while the Consistency Trajectory Model [26] combines
a GAN (35) with an improved consistency distillation (9]. In contrast, our approach based on
distribution matching [22 36 37) inherently integrates classifier-free guidance into the training
supervision, significantly simplifying the training process.
Score Distillation was initially introduced in the context of text-to-3D synthesis [37 (37-40), utilizing a
pre-trained text-to-image diffusion model as distribution matching loss. These methods optimize a 3D object by aligning rendered views with a text-conditioned image distribution, using the scores predicted by a pretrained diffusion model. Recent works have extended score distillation (37 38 41-
43 to diffusion distillation [22 30 36 44 46). Notably, DMD 22 minimizes approximate KL
3

---

Figure 2: 1024x 1024 samples produced by our 4-step generator distilled from SDXL. Please zoom for details.

---

divergence, with its gradient represented as the difference between two score functions: one, fixed and pretrained, for the target distribution and another, trained dynamically, for the output distribution of the generator.
DMD parameterizes both score functions using diffusion models. This training objective proved more stable than GAN-based methods and has demonstrated superior performance in one-step image synthesis. An important caveat, DMD requires a regression loss for stability, calculated using precomputed noise-image pairs, similar to Luhman et al. [6]. Our work does away with this requirement. We introduce techniques to stabilize the DMD training procedure without the regression regularizer, thus significantly reducing the computational costs incurred by paired data precomputation. Furthermore, we extend DMD to support multi-ster generation and integrate the
strengths of both GANs and distribution matching approaches [22 30 36 (5), leading to state-of-the art results in text-to-image synthesis.
3 Background: Diffusion and Distribution Matching Distillation
This section gives brief overview of diffusion models and distribution matching distillation (DMD).
Diffusion Models generate images through iterative denoising. In the forward diffuusion process, noise is progressively added to corrupt a sample x ~ Preal from the data distribution into pure Gaussian noise over a predetermined number of steps T, that, at each timestep f, the diffused samples follow the distribution P = f Preat (z)g(æz (1 [r) (r),r, with qe (I2 (z) ~ N (aaz, o²1), where (2 og > 0 are scalars determined by the noise schedule [7 48). The diffusion model learns to iteratively reverse the corruption process by predicting a denoised estimate p(z,, t), conditioned on the current noisy sample 2, and the timestep t. ultimately leading an image from the data distribution Preal- After training, the denoised estimate relates to the gradient of the data likelihood function, or score function [88 of the diffused distribution:
( (Ia, t) Vz, log Preal, (x:) 3t. Ort/lreal
Sreal
of
Sampling an image typically requires dozens to hundreds of denoising steps
49 52
Distribution Matching Distillation (DMD) distills a many-step diffusion models into a one-step generator G (22) by minimizing the expectation over t of approximate Kullback-Liebler (KL. diver- gences between the diffused target distribution Preal. and the diffused generator output distribution Prake, Since DMD trains G by gradient descent, only requires the gradient of this loss, which can be computed as the difference of 2 score functions:
V LoMD = B. (V KL (leaat.)) = -B. ( (
State (F(G ((saa(F((G(())))
dGo(z) do
de
where (0, I) a random Gaussian noise input, @ are the generator parameters, F is the forward diffusion process (i.e., noise injection) with noise level corresponding to time step t. and Syeal and Stake are scores approximated using diffusion models Preat and Mtake trained on their respective distributions (Eq. (.) DMD) uses a frozen pre-trained diffusion model as /real (the teacher), and dynamically updates Mlake while training G. using a denoising sore-matching loss on samples from the one-step generator, i.e., fake data [2)
Yin et al. [22] found that an additional regression term [16] was needed regularize the distribution
matching gradient (Eq. (2) and achieve high-quality one-step models. For this, they collect a dataset
of noise-image pairs (z,y) y) where the image y generated using the teacher diffusion model, and deterministic sampler 53), starting from the noise map z. Given the same input noise the regression loss compares the generator output with the teacher's prediction:
Lre = (y ((Go(2),)))
(3)
where is a distance function, such as LPIPS (54) in their implementation. While gathering this data incurs negligible cost for small datasets like CIFAR-10, it becomes a significant bottleneck with large-scale text-to-image synthesis tasks, or models with complex conditioning (55(57). For instance, generating one noise-image pair for SDXL (58 takes around 5 seconds, amounting to about 700 A A100 days to cover the 12 million prompts in the LAION 6.0 dataset [59], as utilized by Yin et al. 22 This dataset construction cost alone is already more than 4x our total training compute (as
5

---

detailed in Appendix M This regularization objective is also at odds with DMD's s goal of matching the student and teacher distribution, since encourages adherence to the teacher's 's sampling paths.
4 Improved Distribution Matching Distillation
We revisit multiple design choices in the DMD algorithm [2] and identify significant improvements.
distribution masching gradiesi
FoAl SCOVO
Tew-step generator
iaage
|+
ditlusion
real lunetion lake score function
fake
imput
computed gradient
ditfusion
diiTussio
discriminstor
GAN loss
Figure 3: Our method distills a costly diffusion model (gray, right) into one- or multi -tep generator (red, left). Our training alternates between 2 steps: 1. optimizing the generator using the gradient of implicit distribution matching objective (red arrow) and a GAN loss (green), and 2. training score function (blue) to model the distribution of "fake" samples produced by the generator, as well as GAN discriminator (green) to discriminate between fake samples and real images. The student generator can be one-step or multi-step model, as shown here, with an intermediate step input.
4.1 Removing the regression loss: true distribution matching and easier large-scale training
The regression loss [6 used in DMD 22 ensures mode coverage and training stability, but as we discussed in Section g makes large-scale distillation cumbersome, and is ar odds with the distribution matching idea, thus inherently limiting the performance of the distilled generator that of the teacher model. Our first improvement is to remove this loss.
4.2 Stabilizing pure distribution matching with Two Time-scale Update Rule
Naively omitting the regression objective, shown in Eq. 3 from DMD leads to training instabilities and significantly degrades quality (Tab. For example, we observed that the average brightness, along with other statistics, of generated samples fluctuates significantly, without converging to stable point (See Appendix G). We attribute this instability to approximation errors in the fake diffusion model Mraee which does not track the fake score accurately, since it is dynamically optimized on the non-stationary output distribution the generator. This causes approximation errors and biased generator gradients (as also discussed in (30).
We address this using the two time-scale update rule inspired by Heusel al. (60) Specifically, we train jiaae and the generator G' at different frequencies to ensure that Plaee accurately tracks the generator's output distribution. We find that using fake score updates per generator update, without the regression loss, provides good stability and matches the quality of the original DMD on ImageNet (Tab. while achieving much faster convergence. Further analysis are included in Appendix G
4.3 Surpassing the teacher model using GAN loss and real data
Our model so far achieves comparable training stability and performance to DMD (22 without the need for costly dataset construction (Tab. 3). However, a performance gap remains between the distilled generator and the teacher diffusion model. We hypothesize this gap could be attributed to approximation errors in the real score function Peaa used in DMD, which would propagate to the generator and lead to suboptimal results. Since DMD's distilled model never trained with real data, cannot recover from these errors.
We address this issue by incorporating an additional GAN objective into our pipeline, where the discriminator is trained to distinguish between real images and images produced by our generator.

---

With the proposed improvements, we are able to match the performance of teacher diffusion models on ImageNet and COCO (see Tab. and Tab.5), However, found that larger scale models like SDXL (58) remain challenging to distill into a one-step generator because of limited model capacity and a complex optimization landscape to learn the direct mapping from noise highly diverse and Trained using real data, the GAN classifier does not suffer from the teacher's limitation, potentially allowing our student generator surpass it in sample quality, Our integration GAN classifier into DMD follows minimalist design: we add a classification branch on top of the bottleneck of the fake diffusion denoiser (see Fig. 3). The classification branch and upstream encoder features in the UNet are trained by maximizing the standard non-saturing GAN objective:
LGAN B [log (x, t))] + E, P (oggg(oooo(((((G((()))))
(4)
where D the discriminator, and the forward diffusion process (i.e., noise injection) defined in Section B with noise level corresponding time step 6. The generator G minimizes this objective. Our design is inspired by prior works that use diffusion models as discriminators [24 27). We
note that this GAN objective is more consistent with the distribution matching philosophy since does not require paired data, and is independent of the teacher's 's sampling trajectories.
4.4 Multi-step generator
detailed images. This motivated us to extend DMD to support multi-step sampling.
We fix predetermined schedule with N timestep (2.-42---0 [N). identical during training and inference. During inference, at each step, we alternate between denoising and noise injection steps, following the consistency model [9]. to improve sample quality. Specifically, starting from Gaussian noise 2g ~ (0, I), we alternate between denoising updates &t, (Te, : fz), and forward diffusion steps 2t,+1 0tt,, St, + oues c with c (O, I), until we obtain our final image ăty- Our 4-sepp model uses the following schedule: 999, 749, 499, 249, for teacher model trained with 1000 steps.
4.5 Multi-step generator simulation to avoid training/inference mismatch
Previous multi-step generators are typically trained to denoise noisy real images 23 27 However, during inference, except for the first step, which starts from pure noise, the generator input come from a previous generator sampling step λt, This creates training-inferene mismatch that adversely impacts quality (Fig. 4p. We address this issue by replacing the noisy real images during training, with noisy synthetic images Tt, produced by the current student generator running several steps, similar to our inference pipeline ($ 4.4). This tractable because, unlike the teacher diffusion model, for few steps. Our generator then denoises these simulated images and the outputs are supervised with the proposed loss functions. Using noisy synthetic images avoids the mismatch and improves overall performance (See Sec. 5
generator only
real image
forward didTusion: train lest domain gap
"Gake" sample
backward simulation: triin alignement
Figure 4: Most multi-step distillation methods simulate intermediate steps using forward diffusion during training (left). This creates a mismatch with the inputs the model sees during inference. Our proposed solution (right) remedies the problem by simulating the inference-ting backward process during training.
A concurrent work, Imagine Flash [61] proposed a similar technique. Their backward distillation algorithm shares our motivation of reducing the training and testing gap by using the student-generant images as the input to the subsequent sampling steps at training time. However, they do not entirely resolve the mismatch issue, because the teacher model of the regression loss now suffers from the training-test gap: it is never trained with synthetic images. This error is accumulated along the sampling path. contrast, our distribution matching loss is not dependent on the input the student model, alleviating this issuc.
7

---

4.6 Putting everything together
In summary, our distillation method lifts DMD 22 stringent requirements for precomputed noise- image pairs. It further integrates the strength of GANs and supports multi-step generators. As shown in Fig. 3 starting from pretrained diffusion model, we alternate between optimizing the generator Gg to minimize the original distribution matching objective as well as a GAN objective, and optimizing the fake score estimator jakee using both denoising score matching objective on the fake data, and the GAN classification loss. To ensure the fake score estimate accurate and stable, despite being optimized on-inne, update it with higher frequency than the generator (5 steps 1). comparison of the training algorithms between DMD and DMD2 (Ours) is shown in Appendix Alg.
Experiments
We evaluate approach, DMD2, using several benchmarks, including class-conditional image generation on ImageNet-64x64 [62] and text-to-image synthesis on COCO 2014 [63] with various
teacher models (58). We use the Fréchet Inception Distance (FID) 660 to measure image quality and diversity, and the CLIP Score [64] to evaluate text-to-image alignment. For SDXL models,
additionally report patch FID 65). which measures FID on 299x center-cropped patches of each image, to assess high-resolution details. Finally, we conduct human evaluations to compare our approach with other state-of-the-ee methods. Comprehensive evaluations confirm that distilled models trained using our approach outperform previous work, and even rival the performance of the teacher models. Detailed training and evaluation procedures are provided in the appendix.
5.1 Class-conditional Image Generation
Table D compares our model with recent baselines on ImageNet-64x64. With single forward pass, method significantly outperforms existing distillation techniques and even outperforms the teacher model using ODE sampler [53]- We attribute this remarkable performance to the removal of DMD's regression loss (Sec. 4.1 and 4.2), which eliminates the performance upper bound imposed by the ODE sampler, as well as our additional GAN term (Sec. 4.3), which mitigates the adverse impact of the teacher diffusion model's 's score approximation error.
Table Image quality comparison on 1m4geNet-64x64.
Table Image quality comparison with SDXL back- bone on 10K prompts from COCO 2014.
Fwd Pass (U)
FID
Fwd Pass (L)
FID
Patch
CLIP
Method
Method
())
(J)
FID (L)
(t)
BigGAN-deep 66
1
4.06
LCM-SDXL.
32
81.62
154.40
0.275
ADM 67
250
2.07
4 22.16
33.92
0.317
RIN 668
1000
1.23
SDXL- Turbo
23
24.57
23.94
0.337
StyleGAN -XL 35
1.52
4 23.19
23.27
0.334
Progress. Distill.
15.39
SDXL Lightning
23.92
31.65
0.316
DFNO 69
7.83
27
24.46
24.56
0.323
BOOT
20
1
16.30
TRACT 33
7.43
DMD2 (Ours)
19.01
26.98
0.336
Meng al.
13
7.54
19.32
20.86
0.332
Diff-Instruct
36
5.57
SDXL
Consistency Model
9
6.20
100
19.36
21.38
0.332
Teacher, cfg=6
iCT-deep
12
3.25
SDXL
CTM 26
1.92
100
20.39
23.21
0.335
Teacher, cfg=8
58
DMD
22
2,62
DMD2 (Ours)
+longer training (Ours)
1,28
EDM (Teacher, ODE
53
511
2,22
EDM (Teacher. SDE
33
511
1.36
8

---

5.2 Text-to-Image Synthesis
evaluate DMD2's text-to-image generation performance on zero-shot COCO 2014 [63| Our generators are trained by distiiliigg SDXL (58] and SD v1.5 [ respectively, using subset of million prompts from AAION-Acsthetin Aesthetic as training data for the GAN discriminator. Table summarizes distillation results for the SDXL model. Our 4-step generator produces high quality and diverse samples, achieving FID score of 19.32 and CLIP score of 0.332, rivaling the teacher diffusion model for both image quality and prompt coherence. To further verify our method's s effectiivness, we conduct an extensive user study comparing model'ss output with those from the teacher model and existing distillation methods. We use subset of 128 prompts from PartiPrompts (70) following LADD (24). For each comparison, visually appealing, as well as the one that better represents the text prompt. Details about the human evaluation are included in Appendix D As shown in Figure [ our model achieves much higher user preferences than baseline approaches, Notably, our model outperforms its teacher in image quality for 24% of samples and achieves comparable prompt alignment, while requiring 25 fewer forward passes (4 vs 100). Qualitative comparisons are shown in Figure 6 Results for SDv1.5 are provided Table in Appendix(D Similarly, one-step model trained using DMD2 outperforms all previous diffusion acceleration approaches, achieving a FID score of 8.35, representing a significant 3. 14-point improvement over the original DMD method [22
[59]. Additionally, we collect 500k images from LAION-
ask random set of five evaluators choose the image that is more
Our results also surpass the teacher models that
uses 50-step PNDM sampler (50).
Image Quality
Prompt Alignment
DMD2
SDXL-LCM
DMD2
SDXL-LCM
DMD2
SDXL-Turbo
DMD2
SDXL-Turbo
DMD2
LDxh-Lightnnng DMD2
LiXh-Lightning
DMD2
SDXL-Teacher
DMD2
SDXL-Teacher
20
40
60
60
100
20
40 60
80
100
Preference Rate (%)
Preference Rate (%)
Figure 5:
User study comparing our distilled model with its teacher and competing distillation All distilled models use 4 sampling steps, the teacher uses 50. Our model achieves the best performance for both image quality and prompt alignment.
baselines 23 27 31..
5.3
Ablation Studies
Table 3:
Ablation studies on ImageNet. TTUR stands for two-timescale update rule.
Table 4: Ablation studies with SDXL backbone on 10K prompts from COCO 2014.
DMD No Regress. TTUR GAN FID (L)
Method
FID (L) Patch FID (L) CLIP (1)
2.62
w/o GAN
26.90
27.66
0.328
3.48
w/o Distribution
13.77
27.96
0.307
2.61
Matching
V
1.51
w/o Backward
Simulation 20.66
24.21
0.332
2.56
2.52
DMD2 (Ours)
19.32
20.86
0.332
Table 3 ablates different components of our proposed method on ImageNet. Simply removing the ODE regression loss from the original DMD results in degraded FID of 3.48 due to training instability (see further analysis in Appendix G.. However, incorporating our Two Time-scale Update Rule (TTUR, Sec. Sec. mitigates this performance drop, matching the DMD baseline performance without requiring additional dataset construction. Adding our GAN loss achieves a further 1.1- point improvement in FID. Our integrated approach surpasses the performance of using GAN alone (without distribution matching objective), and adding the two-timescale update rule to GAN alone does not improve highlighting the effectiveness of combining distribution matching with GANs in unified framework.

---

photo of Ilama wearing sunglasses standing the deck of spaceship with the Earth the background.
shiba wearing and black turtleneck
a young girl playing piano
train ride the rain Kerala, With Koala beur wearing hat looking the window. There lot of coconut trees out the window.
DMD2 (Ours)
LCM
Turbo
Lightning
Teacher
Figure Visual comparison between our model, the SDXI. teacher, and selected competing methods 23 27 [T). All distilled models use 4 sampling steps while the teacher model uses 50 sampling steps with classifier-fre guidance. All images are generated using identical noise and text prompts. Our model produces images with superior realism and text alignment. (Zoom in for details.) More comparisons are available Appendix Figure] [I
In Table 4 ablate the influence of the GAN term (Sec. distribution matching objective (Eq. and backward simulation (Sec. 4.4) for distilling the SDXL model into four-step generator. Qualita- tive results shown in Appendix Figure. [ In the absence of the GAN loss, baseline model produces oversaturated and oversmoothed images (Appendix Fig. g third column). Similarly, elimi- nating distribution matching objective (Eq. (2 reduces approach to pure GAN-based method, which struggles with training stability (2]. Moreover, pure GAN-based methods also lack natural way incorporate classifier-free guidance (73), essential for high-quality text-to-image syn- thesis 2 Consequently, while GAN-based methods achieve the lowest FID by closely matching the real distribution, they sigmificantly underperform in text alignment and aesthetic quality (Appendix Fig. [8] second column). Likewise, omitting the backward simulation leads to worse image quality, as indicated by the degraded patch FID score.
10

---

6 Acknowledgements
We extend our gratitude Minguk Kang and Seungwook Kim for their assistance in setting up the human evaluation. We also thank Zeqiang Lai for suggesting the timestep shift technique used in our one-step generator. Additionally, we are grateful our friends and colleagues for their insightful discussions and valuable comments. This work was supported by the National Science Foundation under Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http://iaifi.org/ by NSF Grant 2 105811,, by NSF CISE award 1955864, and by funding from Google, GIST, Amazon, and Quanta Computer.
References
[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. CVPR, 2022.
[2] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.0230: 2022.
[3] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204 06125, 2022.
[4] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding. In NeurlPS, 2022.
[5] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.00 1,01324, 2022.
[6] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023.
[7] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, 2022.
[8] Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. CVPR, 2023.
[9] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In ICML, 2023.
[10] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. ICLR. 2022.
[11] Xingchao Liu, Xiwen Zhang, Jianzhu Ma. Jian Peng, and Qiang Liu. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. arXiv preprint arX1v:2309.06388, 2023.
[12] Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In ICLR, 2024.
[13] Chenlin Meng, Robin Rombach. Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In CVPR, 2023.
[14] Jonathan Heek, Emiel Hoogeboom, and Tim Salimans. Multistep consistency models. arXiv preprint arXiv:2403. 06807, 2024.
[15] Hanshu Yan, Xingchao Liu, Jiachun Pan, Jun Hao Liew, Qiang Liu, and Jiashi Feng. Perflow: Piecewise rectified flow as universal plug-and-play accelerator. arXiv preprint arXiv:2405.075 2024.
[16] Eric Luhman and Troy Luhman. Knowledge distillation in itcrative generative models for improved sampling speed. arXiv preprint 1oXiv:2101.02
2021.
[17] Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang. Jie Wu, Pan Xie, Xing Wang, and Xuefeng Xiao, Hyper-sd: Trajectory segmented consistency model for efficient image synthesis. arXiv preprint arXiv:2404. 13686, 2024.
[18] Chen Xu, Tianhui Song, Weixin Feng. Xubin Li, Tiezheng Ge. Bo Zheng, and Limin Wang. Accelerating image generation with sub-path linear approximation model. arXiv preprint arXiv:2404. 13903. 2024.
[19] Jianbin Zheng, Minghui Hu, Zhongyi Fan, Chaoyue Wang, Changxing Ding, Dacheng Tao, and Tat-Jen Cham. Trajectory consistency distillation. arXiv preprint arXiv:2402.199 19159, 2024.
[20] Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, and Joshua M Susskind. Boot: Data-free distillation of denoising diffuuson models with bootstrapping. In ICML 2023 Workshop on Structured Probabilistic Inference & Generative Modeling, 2023.
11

---

[21] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In NIPS, 2014.
[22]
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Frédo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In CVPR, 2024.
[23] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation.
arXiv preprint arXiv:23311 17004, 2023.
[24] Axel Sauer, Frederic Boesel, Tim Dockhom, Andreas Blattmann, Patrick Esser, and Robin Rom- bach. Fast high-resolution image synthesis with latent adversarial diftusion distillation, arXiv preprint arXiv:2403. 12015, 2024.
[25] Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. Ufogen: You forward once large scale text-to- image generation via diffusion gans. In CVPR, 2024.
[26] Dongjun Kim, Chich-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. In ICLR, 2024.
[27] Shanchuan Lin, Anran Wang, and Xiao Yang. Sdxl-lighting: Progressive adversarial diffusion distillation. arXiv, 2024.
[28] Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion gan: Training gans with diffusion. In ICLR, 2023.
[29] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffuusin gans. In ICLR, 2022.
[30] Mingyuan Zhou, Huangjie Zheng, Zhendong Wang. Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In ICML, 2024.
[31] Simian Luo. Yiqin Tan, Longbo Huang. Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with ste- inference. arXiv preprint arXiv:2310.043755 2023.
[32] Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao. Lem-lora: A universal stable-diffusion acceleration module. arXiv preprint arXiv:2310.04375. 2023.
[33] David Berthelot, Arnaud Autef, Jierui Lin, Dian Ang Yap. Shuangfei Zhai, Siyuan Hu, Daniel Zheng, Walter Talbot, and Eric Gu. Tract: Denoising diffusion models with transitive closure time-distillation. arXiv preprint arXiv:2303.04244 2023.
[34] Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. In NeurlPS, 2021.
[35] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datascts. In SIGGRAPHI 2022.
[36] Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models. In NeurlPS, 2023.
[37] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305. 16213, 2023.
[38] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2023.
[39] Amir Hertz, Ktir Aberman, and Daniel Cohen-Or. Delta denoising score. In ICCV, 2023.
[40] Haochen Wang, Xiaodan Du Jiahao Li. Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In CVPR, 2023.
[41] Mingxuan Yi, Zhanxing Zhu, and Song Liu. Monoflow: Rethinking divergence gans via the perspective of wasserstein gradient flows. In /CML, 2023.
[42] Siddarth Asokan, Nishanth Shetty, Aadithya Srikanth, and Chandra Sekhar Seelamantula. Gans settle scores! arXiv preprint arXiv:2306.016554 2023.
[43] Romann M Weber. The score-difference flow for implicit generative modeling. In TMLR, 2023.
[44] Jean- Yves Franceschi, Mike Gartrell, Ludovic Dos Santos, Thibaut Issenhuth, Emmanuel de Bézenac, Mickaël Chen, and Alain Rakotomamonjy. Unifying gans and score-based diffusion generative particle models. In NeurlPS, 2023.
[45] Thuan Hoang Nguyen and Anh Tran. Swiftbrush: One-step text-to-imaa to-iito-imeeeeem diffusion model with variational score distillation. In CVPR, 2024.
12

---

[46] Mingyuan Zhou, Zhendong Wang. Huangjje Zheng, and Hai Huang. Long and short guidance in score identity distillation for one-step lext-to-image generation. arXiv preprini arXiv:2406.0156. 2024.
[47] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
[48] Yang Song, Jascha Sohl- Dickkiicnt,,n, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[49] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[50] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. ICLR, 2022.
[51] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. In arXiv preprint arXiv:2211.01095 2022.
[52] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In NeurlPS, 2022.
[53] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In NeurlPS, 2022.
[54] Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
[55] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
[56] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
[57] Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar. Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. arXiv preprint arXiv:2311 10089, 2023.
[58] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
[59] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-Sb: An open large-scale scale dataset for training next generation image-text models. NeurIPS, 2022.
[60] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a time-scale update rule converge to local nash equilibrium. In NeurlPS, 2017.
[61] Jonas Kohler, Albert Pumarola, Edgar Schönfeld, Artsiom Sanakoyeu. Roshan Sumbaly, Peter Vajda, and Ali Thabet. Imagine flash: Accelerating emu diffusion models with backward distillation. arXiv preprint arXiv:2405.0522 000000, 2024.
[62] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
[63] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[64] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[65] Lucy Chai, Michael Gharbi, Eli Shechtman, Phillip Isola, and Richard Zhang. Any-resolution training for high-resolution image synthesis. In ECCV, 2022.
[66] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In ICLR, 2019.
[67]
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS, 2021.
[68] Allan Jabri, David Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. In ICML, 2023.
[69] Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. In ICML, 2023.
[70] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint 221.:2006110788 2022.
13

---

[71] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In ICML, 2018.
[72] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image image synthesis. In CVPR, 2023.
[73] Jonathan Ho and Tim Salimans. Classifier- free diffusion guidance. arXiv preprint arXiv:200..12598,
2022.
[74] Bram Wallace, Mcihua Dang, Rafacl Rafailov, Lingi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. arXiv preprint vrxii:2311.12999
2023.
[75] Zhendong Wang. Jianmin Bao, Wengang Zhou, Weilun Wang, Hezhen Hu, Hong Chen, and Houqiang Li. Dire for diffusion--eenrraee image detection. In ICCV, 2023.
[76] Sheng- Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros. Cnn-generated images surprisingly casy to spot... for now. In CVPR, 2020.
[77] Xudong Shen, Chao Du, Tianyu Pang. Min Lin, Yongkang Wong, and Mohan Kankanhalli, Finetuning text-to-image diffusion models for fairness. In ICLR, 2024.
[78] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-lo-image generation. ICML, 2021.
[79] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. In ECCV, 2022.
[80] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML, 2022.
[81] Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Towards language-free free training for text-to-image generation. In CVPR, 2022.
[82] Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-to- to-image synthesis. ICML, 2023.
[83] Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: A unified predictor- framework for fast sampling of diffusion models. arXiv preprint arXiv:2302.044 048677 2023.
[84] Yifan Zhang and Bryan Hooi. Hipa: Enabling one-step text-to- text-to-image diffusion models via high-frequency- promoting adaptation. arXiv preprint arXiv:2311.18/55 2023.
[85] Xun Huang, Ming- Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. In ECCV, 2018.
[86] Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei Ffros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In NeurIPS, 2017.
[87] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. In NeurlPS, 2024.
[88] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[89] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- (sigam: Weak-to-stroon training of diffusion transformer for 4k text-to-image generation. arXiv preprini arXiv:2403. 04692, 2024.
[90] Zeqiang Lai. Opendmd: Open source implementation and models of one-step diffusion with distribution matching distillation, https::// //gi thub. com www.com//ggge Zegiang- Lai /OpenDMD 2024. Accessed: 2024-05-22
[91] Gaurav Parmar, Richard Zhang, and Jun Zhu. On aliased resizing and surprising subtleties in gan evaluation. In CVPR, 2022.
14

---

Table of Contents
Introduction
2
Related Work
3
Background: Diffuusion and Distribution Matching Distillation
4
Improved Distribution Matching Distillation
4.1
Removing the regression loss: true distribution matching and easier large-scale training
4.2
Stabilizing pure distribution matching with Two Time-scale Update Rule
4.3
Surpassing the teacher model using a GAN loss and real data
4.4
Multi-step generator
4.5
Multi-step generator simulation avoid training baaning/infereeee inference mismatch
4.6
Putting everything together
5
Experiments
5.1
Class-conditional Image Generation
5.2
Text-to-Image Synthesis
5.3
Ablation Studies
Acknowledgements
11
Limitations
15
B
Code, dataset, and more results
15
C
Broader Impact
16
D
SD v1.5 Results
16
Comparison with DMD1
16
Text-to-Image Synthesis Further Analysis
16
Two Time-scale Update Rule Further Analysis
17
H
Additional Text-to-lmage Synthesis Results
17
1
ImageNet Visual Results
18
J
Implementation Details
24
J.1
GAN Classifier Design
24
J.2
ImageNet
24
1.3
SD
24
J.4
SDXL
24
K
Evaluation Details
25
User Study Details
25
M
Prompts for Figure
Figure 2] and Figure 12
26
A
Limitations
While achieving superior image quality and text alignment, distilled generator experiences slight degradation in image diversity compared to the teacher models (see Appendix F Additionally, generator still requires four steps to match the quality of the largest SDXL model. These limitations, while not unique to our model, highlight areas for further improvement. Like most previous distillation methods, Introducing variable guidance scale [13,31] could be a promising direction for future research. Furthermore, methods optimized for distribution matching; incorporating human feedback or other reward functions could further enhance performance [17 74). Lastly, training large-scale generative models is computationally intensive, making inaccessible for most researchers. We hope our efficient approach and optimized, user-friendly codebase will help democratize future research in this field.
fixed guidance scale during training, limiting flexibility.
B Code, dataset, and more results
We are continually updating the paper and related materials. For access our code, model, and the latest version of the paper with expanded results and analysis, please visit our project website.
15

---

Table 5: Sample quality comparison on 30K prompts from COCO 2014.
Family
Method
Resolution (1) Latency (4) FID (1)
DALL-E
27.5
DAILE
256
10.39
Parti-750M
70
256
10.71
Original,
Parti-3B
70
256
6.4s
8.10
Make-A-Scene
256
25.0s
11.84
unaccelerated
GLIDE 80
256
15.0s
12.24
LDM
256
3.7s
12.63
Imagen
256
9.1s
7.27
cDiff-1
256
32.0s
6.95
LAFITE 81
256
0.02s
26.94
GANs
StyleGAN-T 82
512
0.10s
13.90
GigaGAN 72
512
0.13s
9.09
DPM++ (4 step)
0.26s
22.36
UniPC (4 step)
512
0.26s
19.57
I.CM-LoRA (4 step
132
512
0.19%
23.62
InstaFlow-0.9p
512
0.09s
13.10
Accelerated
SwiftBrush 45
0.09s
16.67
diffusion
HiPA 84
512
0.09s
13,91
UFOGen 25
512
0.09s
12.78
SLAM (4 step) 18
512
0.19s
10.06
DMD [22
512
0.09s
11.49
DMD2
512
0.09s
8.35
SDv1.5 (50 step, cfg=3, ODE)
50
512
2.59s
8.59
Teacher
SDv1.5 (200 step, cfg=2, SDE)
512
10.25s
7.21
C
Broader Impact
Our work on improving the efficiency and quality diffusion model has several potential societal impacts, both positive and negative. On the positive side, the advancements in fast image synthesis can significantly benefit various creative industries. These models can enhance graphic design, animation, and digital art by providing artists with powerful tools to generate high-quality visuals efficiently. Additionally, improved text-to-imas to-ta-eemmee synthesis capabilities can be used education and entertainment, enabling the creation of personalized learning materials and immersive experiences.
However, potential negative societal impacts must be considered. Misuse risks include generating misinformation and creating fake profiles, which could spread false information and manipulate public opinion. Deploying these technologies could result in biases that unfairly impact specific groups, especially if models are trained on biased datasets, potentially perpetuating amplifying existing societal biases. To mitigate these risks, we are interested in developing monitoring mechanisms to detect and prevent misuse [75 76 and methods to enhance output diversity and fairness (77).
SD v1.5 Results D
Table 5 presents detailed comparisons between our one-step generator distilled from SD v1.5 and competing approaches.
E
Comparison with DMD1
A comparison between DMD and DMD2 is shown in Algorithm 7
F
Text-to-Image Synthesis Further Analysis
Qualitative ablation results using SDXL. backbone are shown Figure Additionally, we compare the image diversity of our 4-step generator with other competing approaches distilled from SDXL 23
16

---

Table 6: Image quality and diversity comparison with SDXL backbone.
# Fwd
FID (L)
Patch FID (L)
CLIP
Diversity
Method
Pass (1)
(1)
Score (t)
LCM-SDXL
32
4
22.16
33.92
0.317
0.61
LDXI--Turhe
4
23.19
23.27
0.334
0.58
SDXL-Lightning
27
4
24.46
24.56
0.323
0.63
DMD2 (Ours)
4
19.32
20.86
0.332
0.61
SDXL cfg=6
58
100
19.36
21.38
0.332
0.64
SDXL Teacher, ofg=8
58
100
20.39
23.21
0.335
0.64
Method SDXL.
ImageReward 0.86
Aesthetic Score
6. 16
DMD2
1.07
6.30
Table 7:
Comparison of ImageReward and Aesthetic Score for Different Methods
We employ LPIPS-based diversity score, similar to that used in multi-modal image-to- image translation 88 86). Specifically, we generate four images per prompt and calculate the average pairwise LPIPS distance [54] For this evaluation, we use the LADD (2 subset of PartiPrompts [70].
31J.
We also report the FID and CLIP score measured on 10K prompts from COCO 2014 on the side. T summarizes the results. Table provides further comparisons using image reward
[87 and
aesthetic score metrics [59]. Our model achieves the best image quality, indicated by the lowest FID and Patch FID scores. We also achieve text alignment comparable to SDXL-Turbo while attaining better diversity score. While SDXL-Lightning [27] exhibits higher diversity score than our approach, it suffers from considerably worse text alignment, reflected by the lower CLIP score and human evaluation (Fig. 5). This suggests that the improved diversity is partially due to random outputs lacking prompt coherence. We note that it is possible increase the diversity of our model by raising the weights for the GAN objective, which aligns with the more diverse unguided distribution. Further investigation into finding the optimal balance between distribution matching and the GAN objective is left for future work.
G Two Time-scale Update Rule Further Analysis
In Section we discuss that updating the fake score multiple times (5 updates) per generator update leads to better stability. Here, we provide further analysis. Figure visualizes pixel brightness variations throughout training. The baseline approach, which omits the regression objective from DMD and uses just fake score update, results in significant training instability, as evidenced by periodic fluctuations in pixel brightness. In contrast, two time-scale update rule with 5 fake score updates per generator update stabilizes the training and leads to better sample quality, as shown in Tab.
We further examine the influence of the update frequency for the fake diffusion model jlake in Figure TO An update frequency of fake diffusion update per generator update corresponds to the naive baseline (red line) and suffers from training instability. Although a frequency of 10 updates (magenta line) provides excellent stability, it significantly slows down the training process. We found that a moderate frequency of 5 updates (green line) achieves the best balance between stability and convergence speed on ImageNet. Our approach proves more effective than using asynchronous learning rates [60] (cyan line) and converges significantly faster than the original DMD method that employs regression loss [22] (dark blue line). For new models and datasets, recommend adjusting the iteration number to the smallest value that ensures the stability of general image statistics, such as pixel brightness.
H Additional Text-to-Image Synthesis Results
Additional visual comparisons the 4-step distilled models are shown in Figure Sample outputs from one-step generator are presented in Figure 12
17

---

Algorithm 1: DMD (original)
Algorithm 2: DMD2 (ours)
Input:
Protrained real diffusion model Mesl paired ODE
Input:
Pretrained real diffusion model Medl: real image dataset Drad.
solution pairs 2 {Art She))
Output: Trained generator G
updato_G_froe __freq: Irequency of generalor updales
Initialize generator Iake score model
Output: Trained generator
G copy "upyWeights(Haaa)
// Initialize generator, Take score model, discriminator
Make +- copyWeigghie (HeyWWeighs((HHe))
while maii do
G copyWWighhts(Hea))
// Generate batch of inagee
Msae f copyyWiighs(Htma))
Sample 2 No0, n) (Zet: Merl ~ D. x + G(z): Ind +- G(za)):
intisalizeDascciniiiaott))
5 for itenation J to maxhers do // main training loop
// (a) Generate batch images
// 1. Updare generator
7 Sample = ~ (0,
CKL distribationMotchingl..c
Minkes
musti--stp then
10
Lis LPIPS Mer)::
- G G(muliiSepSammliggGG..
11
La - LKL + Auy Leg.
simulaee multi step
12
G updale((G Le);
end ebe
13
2. Update fake score model
14
Sample 1(00, 1];
<- G(=)
15
+ forvaddifffsssii (stpifiisioo((soppgaaaaa))))) t);
end if // (b) Update only once per update_G_free iterations
16
Camss + denoisingLosd Mine (*t, (), st)p
17
+- update upaet Laman):
18
end while
15
ieeation % updale, G. freg== then
// (b1) Distribution natching
CKL + distrikuiinMMuccinnloeeese Mide; x);
// (b2)
--E(llog D(F(G(s)) e)]]
// (b3) Final generator leaa
21
La + LKL + AeAN Coax:
(b4) Apply gradient update
23
G + updale(G,, Col:
24
end
Sample seal images Rnd Duni
26
Sample 1 ~ 14(0, 1);
Update prase (fake denoising fake data
Fe +- CawaanDiillusionssttrgalll
t):
Lamme + denoisingLoss Maste (Rt, t), stopgrad(z)):
Peae f- update updatee (Mate, Lemue):
1/ Update discriminator
cllasiificatioo)
CaaND +- Ellog ((F(m.a)))) +))] E[lo((1
D D(F(stopgrn((a) t))0]
33
D + update(D, LEAND):
34 end for
Figure 7: Side-by-side comparison of DMD (left) and our improved DMD2 (right).
I ImageNet Visual Results
In Figure 13 we present qualitative results obtained from our one-step distilled model trained on the ImageNet dataset.
18

---

girl examining an ammonite fossil
a teddy bear on skateboard in times square
A soft beam of light shines down an armored granite wombat warrior statue holding broad sword. The statue stands ornate pedestal the cella temple. wide-angle lens. anime oil painting.
A close-up a woman's face, lit by the soft glow of neon sign in a dimly lit, retro diner, hinting narrative of longing and nostalgia.
Cinematic photo of beautiful girl riding a dinosaur in jungle with mud, sunny day shiny clear sky. 35mm photograph, film, professional, 4k, highly detailed.
w/o Distribution Matching
w/o Backward Simulation
DMD2 (Ours)
w/o GAN
Figure 8: SDXL Qualitative Ablations. All images generated using identical noise and text prompts. Removing the distribution matching objective significantly degrades aesthetic quality and text alignment. Omitting the GAN loss results in oversaturated and overly smoothed images. The baseline without backward simulation produces images of lower quality.
19

---

[iteration-1] Ours (iteration-5)
0.50
Pixel Brightness
0.30
0.50
0.30
Training Steps
Figure 9: Visualization of pixel brightness variations throughout training. The baseline approach, which naively removes the regression loss from the original DMD [22], suffers from significant training instability, leading to fluctuating general image statistics like the overall pixel brightness. In contrast, our time-scale update rule, which optimizes the fake diffusion model five times per generator update, significantly stabilizes training and enhances sample quality.
DMD with Regression Loss (Iteration-1)
DMD w/o Regression Loss (iteration=1)
Regression (Iteration-5i
DMD Regression Loss (iteration= 10] Regression (async
FID Sceres
0
Training Hours
Figure 10: Visualization of FID score progression during training. Naively removing the regression loss leads training instability (red line). two time-scale update rule with five fake diffusion critic updates per generator update stabilizes training and is more effective than using a larger number of fake diffusion updates learning rate 5 times larger than the generator. The model trained with our two time-scale update rule (green) also converges significantly faster than the original DMD method with regression loss (dark blue), even though TTUR performs less number of the generator weight updates.
asynchronous learning rate where the fake diffusion model uses
20

---

an orunge wearing cow boy hat
A bald eagle made of chocolate powder, and whipped cream
pumpkin man's head
punk rock squirrel a studded leather jacket shouting into a microphone hile standing on a boulder
cut reading newspaper
DMD2 (Ours)
LCM
Turbo
Lightning
Teacher
Figure 11: Additional visual comparison between our model, the SDXL teacher, and selected competing methods [23 31) All distilled models use sampling steps while the teacher model uses 50 sampling steps with classifier-fi guidance. All images are generated using identical noise and text prompts. Our model produces images with superior realism and text alignment. Please zoom for details.
21

---

Figure 12: Additional 1024x 1024 samples produced by our 1-step generator distilled from SDXL. Please zoom in for details.
22

---

Figure 13: One-step samples from our generator trained on ImageNet (FID=1.28). Please zoom in for details.
23

---

Implementation Details
This section outlines key aspects of the implementation. All results are reproducible using our open-source training and evaluation code. Generally, we employ the maximum batch size supported by our compute resources. The learning rate is set to the highest stable value, ensuring no divergent loss within the initial 500 iterations. We determine the TTUR iteration count based on the minimum needed for stability, while the guidance scale is selected to optimize performance for the teacher model.
J.1 GAN Classifier Design
Our GAN classifier design inspired by SDXL--Lighming [27]. Specifically, we attach a prediction head to the middle block output of the fake diffusion model. The prediction head consists of a stack of 4 4 convolutions with a stride of 2, group normalization, and SiLU activations. All feature maps are downsampled to 4 x 4 resolution, followed by single convolutional layer with a kernel size and stride of 4. This layer pools the feature maps into a single vector, which is then passed to a linear projection layer to predict the classification result.
J.2 ImageNet
Our ImageNet implementation closely follows the DMD paper [22]. Specifically, we distill a one-step generator from the EDM pretrained model [53). released under the BY-NC-SA 4.0 License. For the standard training setup, we use the Adam optimizer (88) with learning rate of 2 x 10-6, weight decay of 0.01, and beta parameters (0.9, 0.999). We use batch size of 280 and train the model on 7 A100 GPUs for 200K iterations, which takes approximately fake diffusion model update per generator update is set to 5. The weight for the GAN loss is set to 3 x 10-3 For the extended training setup shown in Table | we first pretrain the model without GAN loss for 400K iterations. We then resume from the best checkpoint (as measured by FID), enable the GAN loss with a weight of 3 x 10 reduce the learning rate to 5 x 10-7, and continue training for an additional 150K iterations. The total training time for this run is approximately 5 days.
days. The number of
J.3
SD v1.5
We distill a one-step generator from the SD v1.5 model [T. released under the CreativeML Open RAIL-M license, using prompts from the LAION- Acsthetic 6.25+ dataset (59]. Additionally, we collect 500K images from LAION-Aesthete 5.5+ as training data for the GAN discriminator, filtering out images smaller than 1024 1024 and those containing unsafe content. Our training process involves two stages. In the first stage, we disable the GAN loss and use the Adam optimizer with learning rate of 10-5 weight decay of 0.01, and beta parameters of (0.9, 0.999). The fake diffusion model is updated 10 times per generator update. We set the guidance scale for the real diffusion model to be 1.75. We use batch size of 2048 and train the model 64 A100 GPUs for 40K iterations. In the second stage, we enable the GAN loss with a weight of 10-3 33 reduce the learning rate to 5 x 10-? and continue training for an additional 5K iterations. The total training time is approximately 26 hours.
J.4 SDXL
We train both one-step and four-step generators by distilling from the SDXL model (58), released
under the CreativeMI. Open RAIL++MM License. For the one-step generator, we observed similar block noise artifacts as reported in SDXL-Lightning (27) and Pixart-Sigma [89]. We addressed this by adopting the timestep shift technique from OMD [90] and Pixart-Sigma [89], setting the conditioning timestep to 399, Additionally, we initialized the one-step generator by pretraining with regression loss using small set of 10K pairs for a short period. These adjustments are not necessary for the multi-step stepi--tep model or other backbones, suggesting this issue might be specific to SDXL. Similar to SD v1 v1.5, we use prompts from the LAION-Aestheti 6.25+ dataset (59) and collect 500K images from LAION- -AstheAst 5.5+ as training data for the GAN discriminator, filtering out images smaller than 1024 x 1024 and those containing unsafe content. The generator is trained using the AdamW optimizer with a learning rate of x 10-7 a weight decay of 0.01, and beta parameters of (0.9, 0.999). The fake diffusion model is updated 5 times per generator update. We set the guidance
24

---

scale for the real diffusion model to be 8. We use batch size of 128 and train the model on 64 A100 GPUs for 20K iterations for the 4-step generator and 25K iterations for the 1-step generator, taking approximately 60 hours.
K
Evaluation Details
For the COCO experiments, we follow the exact evaluation setup as GigaGAN 722 and DMD [22] For the results presented in Table we use 30K prompts from the COCO 2014 validation set and generate the corresponding images. The outputs are downsampled to x and compared with 40,504 real images from the same validation set using clean-FII [91]. For the results presented in T we use a random set of 10K prompts from the COCO 2014 validation set and generate the corresponding images. The outputs downsampled to 512x512 and compared with the corresponding 10K real images from the validation set with the same prompts. We compute the CLIP score using the OpenCLIP-G backbone. For the ImageNet results, generate 50,000 images and calculate the FID statistics using EDM's evaluation code [53]
L User Study Details
To conduct the human preference study, we use the Prolific platform (httss//www.prrolfficcco We use 128 prompts from the LADD (24) subset of PartiPrompts (70]. All approaches generate corresponding images, which are presented pairs to human evaluators measure aesthetic and prompt alignment preference. The specific questions and interface are shown in Figure Consent
is obtained from the voluntary participants. We manually verify that all generated images contain standard visual content that poses no risks to the study participants.
which image looks more representativve of the text shown above and faithfully follows it?
Which image of more aesthetic and visual pleasing?
Wote
Wyloree
Figure 14: sample interface for our user preference study, where images are presented random left/right order.
25

---

M Prompts for Figure 1 Figure 2 and Figure 12
We use the following prompts for Figure
D From left to right, top to bottom:
a girl examining an ammonite fossil
A photo of an astronaut riding horse in the forest.
a giant gorilla at the top of the Empire State Building
A close-up photo of wombat wearing a red backpack and raising both arms in the air.
Mount Rushmore in the background.
An oil painting of two rabbits in the style of American Gothic, wearing the same clothes as in the original.
portrait of old man
a watermelon chair
A sloth go kart on race track. The sloth is holding a banana one hand. There is banana peel on the track in the background.
penguin standing on a sidewalk
teddy bear on skateboard in times square
We the following prompts for Figure From left to right, top to bottom:
a chimpanzee sitting on a wooden bench
cat reading newspaper
A television made water that displays an image of a cityscape at night.
portrait of statue of the Egyptian god Anubis wearing aviator goggles, white t-shirt and leather jacket. The city of Los Angeles is in the background.
squirrell driving toy car
an elephant walking on the Great Wall
capybara made of voxels sitting in a field
Cinematic photo of a beautiful girl riding dinosaur in jungle with mud, sunny day shiny clear sky. 35mm photograph, film, professional, 4k, highly detailed. A still image of humanoid cat posing with a hat and jacket in a bar. A soft beam of light shines down on an armored granite wombat warrior statue holding broad sword. The statue stands an ornate pedestal in the cella of a temple. widc-angle lens. anime oil painting.
children
A photograph of the inside of a subway train. There are red pandas sitting on the seats. One of them is reading a newspaper. The window shows the jungle in the background.
a goat wearing headphones motion
A close-up of a woman's face, by the soft glow of neon sign dimly lit, retro diner, hinting at narrative of longing and nostalgia.
We use the following prompts for Figure 12
From left right, top bottom:
A close-up of a woman's 's face, lit by the soft glow of neon sign a dimly lit, retro diner, hinting at narrative of longing and nostalgia.
a cat reading a newspaper
A television made of water that displays an image of cityscape night.
a portrait of a statue of the Egyptian god Anubis wearing aviator goggles, white t-shirt and leather jacket. The city of Los Angeles is in the background.
a squirrell driving a toy car
elephant walking on the Great Wall
a capybara made of voxels sitting in a field
A soft beam of light shines down on an armored granite wombat warrior statue holding broad sword. The statue stands an ornate pedestal in the cella of temple. wide-angle lens.
anime oil painting.
goat wearing headphones
An oil painting two rabbits in the style of American Gothic, wearing the same clothes as in the original.
girl examining an ammonite fossil
chimpanzee sitting wooden bench
26

---

children
A still image of humanoid cat posing with a hat and jacket in a bar.
motion
27

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We accurately state our contribution and scope in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
28

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all experimental details in the main paper (Section 5 and the Appendix (Sections[] K). Additionally, we release all our code and models to reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution primarily new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (c.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
29

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
We release them at our project website.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ mripss.cc/ubllic/gui.ee CoodeSumeeieeeeee ssi iessonPolll molyicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In the main paper (Section 5) and Appendix (Section J and K
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Reporting error bars is computationally prohibitive due the high cost of large-scale generative model training. However, to the best of our knowledge, the variance across runs and seeds is minimal.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
30

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes)
Justification: In Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We fully comply with the NeurlPS code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
31

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper does not pose significant risks, and future production systems could implement an output classifier to detect and prevent unsafe content.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cow/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
32

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We release code and dataset website.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: In Appendix
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our human study only requires participants to compare image outputs from different models and poses no risks them. More details are available in Appendix [
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "9BHV0SNzrT", "reviewer": "Reviewer_Yut7", "summary": "The paper introduces an upgraded version of Distribution Matching Distillation (DMD), i.e., DMD2, which addresses the limitation and inefficiency of previous DMD and improves the performance of efficient and high-quality image synthesis using diffusion models. Specifically, the authors identify the limitations of the original Distribution Matching Distillation (DMD), such as the need for a regression loss and extensive dataset construction. DMD2 eliminates the regression loss, integrates a Generative Adversarial Network (GAN) loss, and introduces a two-time-scale update rule to stabilize training. Additionally, a new training procedure is implemented to simulate multi-step sampling, addressing the training-inference mismatch. Experimental results demonstrate that DMD2 achieves state-of-the-art performance, surpassing the original DMD and other competitive models in image quality and efficiency.", "questions": "- Training with GAN often entails numerical instability. Does DMD2 have such concerns? If it is true, could the author provide some details in overcoming the instability of DMD2?\n- Besides the evaluation metric such as FID and Inception scores, how about some human-related metrics such as ImageReward or aesthetic scores? Does DMD2 show comparable results to teacher SDXL?", "limitations": "The paper includes limitations of the proposed method, e.g., requires multiple steps to generate on par with teacher model, e.g., SDXL.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. **Elimination of Regression Loss**: By removing the regression loss, DMD2 simplifies the training process and reduces computational costs, making it more scalable and flexible for large-scale applications.\n\n2. **Integration of GAN Loss**: The incorporation of a GAN loss improves the quality of generated images by discriminating between real and generated samples, enhancing the overall distribution matching objective.\n\n3. **Two-Time-Scale Update Rule**: This technique addresses training instability issues, ensuring that the fake score accurately tracks the generator’s output distribution, leading to stable and high-quality image generation.\n\n4. **Multi-Step Sampling**: The introduction of multi-step sampling allows DMD2 to produce high-quality images in fewer steps, addressing the inefficiency of one-step generation while maintaining performance.\n\n5. **Comprehensive Evaluation**: The paper provides extensive experimental results on various benchmarks, demonstrating DMD2's superior performance in both class-conditional and text-to-image synthesis tasks.", "weaknesses": "I do not find a specific weakness of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer Yut7's constructive feedback. Below, we address the remaining questions regarding numerical instability and human-related metrics. \n\n**Training with GAN often entails numerical instability. Does DMD2 have such concerns? If it is true, could the author provide some details in overcoming the instability of DMD2?**\n\nThe GAN component in DMD2 does not introduce numerical instability. This stability can be attributed to our use of a diffusion GAN framework, where both the generators and discriminators are initialized from a pretrained diffusion model. Before classification, images are also treated with noise injection, enhancing stability. This method has proven more stable than traditional GANs that utilize pixel space discriminators without noise injection, as supported by several recent studies [1, 2]. Furthermore, DMD2 includes a two-timestep-scale update rule that bolsters stability. Additionally, DMD2 utilizes a weighted combination of DMD and GAN losses. The DMD loss consistently guides the overall structure of the images, preventing mode collapse and ensuring training stability even with the added GAN components.\n\n**Besides the evaluation metric such as FID and Inception scores, how about some human-related metrics such as ImageReward or aesthetic scores? Does DMD2 show comparable results to teacher SDXL?**\n\nThank you for this insightful suggestion! We have extended our evaluation of DMD2 and SDXL to include both ImageReward and aesthetic scores, using PartiPrompts [3] for consistency with our human evaluation. Below is a summary of our findings:\n\n| Method | ImageReward | Aesthetic Score | \n| - | - | - |\n|DMD2 | 1.07 | 6.30 |\n| SDXL | 0.86 | 6.16 |  \n\nThese results demonstrate that DMD2 consistently outperforms SDXL, corroborating the findings from our main paper's FID and human evaluation results. We will incorporate these additional metrics into the revised version of our paper. \n\n[1] Wang, Zhendong, et al. \"Diffusion-gan: Training gans with diffusion.\" ICLR 2023 \n\n[2] Xu, Yanwu, et al. \"Ufogen: You forward once large scale text-to-image generation via diffusion gans.\" CVPR 2024.\n\n[3] Yu, Jiahui, et al. \"Scaling autoregressive models for content-rich text-to-image generation.\" arXiv preprint arXiv:2206.10789 2.3 (2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oEVEjCh8Bf", "reviewer": "Reviewer_aF5v", "summary": "This work addresses identifies reasons for training instability of one of competitive diffusion distillation approaches based on distribution matching, using bi-level optimization and also adopt a GAN based feature space feedback for improved quality. Overall demonstrate very good performance on SDXL, SD checkpoints demonstrating effectiveness on large models too.", "questions": "What is setup of ablation without backward simulation in multi-step setting? Do forward diffusion and query the student generator and based on that query fake score function estimator? If so is the fake score function also trained on equivalent forward diffuse + generator's predictive distribution? As it is currently unclear is it alignment of fake score function alignment to generator or the backward simulation which is resulting in improved performance.\n\nAlso given DMD has implicit assumption that the fake score function is capturing predictive distribution of student generator and authors identify its fitting being one of reasons for instability. It might be useful to understand how sensitive is alignment of fake score estimation to student's generator. Also, how good is quality of fake score fun at different stages of training and its implications? \n\nAs at implementation level we are starting with pretrained model and in the limit if distilled student model matches pre-trained model's weights we are asking fake score function to match pre-trained model again.", "limitations": "This work build on DMD and improves stability, good engineering practice as primary contribution of this work with limited formulation novelty or insights. So more insights on hyperparameter sensitivity, why and how are different design choices effect final performance, diversity etc as discussed above would make it more useful for community and strong contribution!", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Paper is well written and easy to understand, with good benchmarking for large scale models and also comparing to other distillation techniques.  Overall, DMD puts lesser constraints on distillation w.r.t underlying map from noise to data space enabling more good formulation for distillation. And improving stability is useful for broader adoption of DMD style formulation for distillation towards practical diffusion based applications.", "weaknesses": "As the authors discuss on not using real-data within current formulation and setup, there could be a tendency for model to have model collapse? \n\nAs proposed distillation objective is not sampling w.r.t teacher's marginal predictive distribution nor data-distribution. It would be useful to get some diversity metric at per-prompt level using  e.g., LPIPS Diversity or etc comparing to other Distillation approaches and teacher model.\n\nAlso it would be useful to understand what stage of training causes this mode collapse, i.e., does small scale training preserve diversity at cost of some quality drop or we observe a consistent drop in diversity?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer aF5v's constructive feedback. Below, we address the remaining concerns.\n\n**As the authors discuss on not using real-data within current formulation and setup, there could be a tendency for model to have model collapse?** \n\nThere may be some misunderstanding regarding our use of real data. In fact, our model does incorporate real data during training via the GAN loss component. This inclusion enhances diversity and helps to prevent mode collapse. The positive impact of integrating GAN loss with real data is evident in the improved performance detailed in Tables 3 and 4.\n\n**As proposed distillation objective is not sampling w.r.t teacher's marginal predictive distribution nor data-distribution. It would be useful to get some diversity metric at per-prompt level using e.g., LPIPS Diversity or etc comparing to other Distillation approaches and teacher model.**\n\nThank you for your suggestion! We have indeed conducted a diversity assessment using LPIPS diversity metrics, which is presented in Table 6 of the appendix. Our results indicate that our model's diversity is on par with other distillation approaches, such as the latent consistency model, and significantly exceeds that of purely GAN-based methods like SDXL-Turbo. While our model shows slightly worse diversity compared to the teacher model, it offers substantially better image quality, as illustrated in Figure 5. We believe this often represents a more advantageous trade-off for practical text-to-image applications.\n\n**Also it would be useful to understand what stage of training causes this mode collapse, i.e., does small scale training preserve diversity at cost of some quality drop or we observe a consistent drop in diversity?**\n\nThank you for your suggestion! In response, we retrained our SDXL model and monitored diversity metrics throughout the training process. The results from this new run showed a small diversity improvement over those reported in our paper. Specifically, the model displays higher diversity at the beginning, which gradually diminishes as image quality improves. Despite this trend, the final diversity scores of the model remain closely comparable to those of the teacher model (0.63 vs 0.64 for the teacher), indicating a well-maintained balance between diversity and image quality.\n\n| Train Iter | LPIPS Diversity |\n| - | - | \n| 0 | 0.25 |\n| 4k | 0.65 |\n| 8k | 0.64 | \n| 12k | 0.65 | \n| 16k | 0.64 | \n| 20k | 0.63 |  \n| SD Teacher Baseline | 0.64 | \n\n\n\n**Setting without backward simulation**\n\nThe reviewer’s interpretation is correct. In the setting without backward simulation, we add noise to real images and then feed these noisy images into our generator. The output from the generator is then supervised using both the DMD and GAN loss. Additionally, the fake score function is trained based on this generator output. We will clarify this process more thoroughly in the revised version of our paper.\n\n**how sensitive is alignment of fake score estimation to student's generator and how good is the quality of fake score at different stages of training and its implications?**\n\nThank you for suggesting this set of further analyses. Due to time constraints, these will be included in the revised version of our paper. Regarding the first part, we believe achieving proper alignment between the fake score estimation and the generator’s output distribution is crucial. As shown in Figure 9 of the main paper, more frequent training of the fake score—aimed at enhancing its accuracy—leads to more stable generator training and overall improved performance. We will provide a more comprehensive analysis in the revised version. For the second part, we plan to retrain an ImageNet model and monitor the performance of the fake score model at various training stages by assessing the quality of its sample outputs. We appreciate the reviewers’ valuable suggestions and look forward to incorporating these insights!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xDWT33Illg", "reviewer": "Reviewer_xYYF", "summary": "This paper introduces DMD2, a few-step distilled generator to achieve fast sampling while maintaining the decent generation quality of the multi-step diffusion models. DMD2 proposes several new improvements to the training procedure of the original DMD, including (1) replacing the regression loss with the Two-Time scale Update Rule (TTUR) to stabilize the training process, (2) incorporating the standard GAN loss to achieve better quality, (3) and utilizing the backward simulation to alleviate the potential mismatch of training and inference. Built upon these modifications, DMD2 achieves excellent results on few-step image generation.", "questions": "1. In the original DMD paper, the regression loss is capable of mitigating the issue of mode collapse. Would DMD2 also suffer from this issue as the regression loss is removed and an extra GAN loss is introduced?\n2. Why the intermediate outputs of DMD2 shown in the right subfigure of Figure 3 are so similar and seem to follow a certain trajectory? As far as I know, distribution matching-based methods do not guarantee the specific paths of the teacher diffusion model and the student generation model are aligned (as mentioned in Lines 35-39). Could authors provide more explanations about this phenomenon (Fig 3)? Note that the samples generated by few-step consistency models may also switch between different paths and generate very different images.", "limitations": "Please refer to the weakness and question sections above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The source-intensive process of generating noise-image pairs is replaced by a simple TTUR strategy.\n2. DMD2 achieves SOTA results on one-step image generation on ImageNet64 and shows its effectiveness on distilling SDXL into few steps.", "weaknesses": "1. It would be better to include a detailed training algorithm to clearly showcase the modifications over the original DMD training process.\n2. In practice, the real data used to train the (teacher) diffusion models may not be accessible to the users who hope to distill a small (student) generation model (due to privacy, storage, …). In that case, one limitation of DMD2 is the calculation of GAN loss may become inapplicable. Could you share your opinions about this matter?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer aF5v's constructive feedback. Below, we address the remaining concerns.\n\n**It would be better to include a detailed training algorithm to clearly showcase the modifications over the original DMD training process.**\n\nThank you for your suggestion. As you accurately summarized, our DMD2 model eliminates the need for the regression loss and incorporates a diffusion GAN loss along with backward simulation to mitigate training-inference mismatches. To clearly illustrate these modifications over the original DMD training process, we will include a detailed comparison of the training algorithms in the revised version of our paper. \n\n**In practice, the real data used to train the (teacher) diffusion models may not be accessible to the users who hope to distill a small (student) generation model (due to privacy, storage, …). In that case, one limitation of DMD2 is the calculation of GAN loss may become inapplicable. Could you share your opinions about this matter?**\n\nWe acknowledge that DMD2 relies on access to real data to enhance diversity and image quality. However, it is important to note that the exact dataset used to train the teacher model is not required. For our training, we utilized a random set of 500,000 images from the LAION database, which is generally of lower quality than the curated aesthetic dataset of SDXL. This demonstrates that our GAN loss can be effectively applied using an alternative dataset, underscoring its versatility and universal applicability, regardless of the specific model being distilled.\n\n\n**In the original DMD paper, the regression loss is capable of mitigating the issue of mode collapse. Would DMD2 also suffer from this issue as the regression loss is removed and an extra GAN loss is introduced?**\n\nThank you for your question. In DMD2, we demonstrate that much of the mode collapse observed in the original DMD method relates more to training issues than to an inherent inability of the DMD loss to support diverse generator training. Practically, we can assess the final performance under two scenarios:\n\n- Class-Conditional Image Generation: We observed no mode collapse, as evidenced by the state-of-the-art FID scores for image generation (see Table 1).\n- Text-to-Image Generation: This scenario is more complex. We utilize classifier-free guidance, which typically trades diversity for image quality. At high guidance settings, using the SDXL baseline, we achieved superior image quality with diversity comparable to or better than other distillation methods but slightly worse than the teacher model (see Table 6 and the new results in our Response to Reviewer **aF5v**). \n\nOur current framework trains stably and produces excellent image quality along with comparable diversity. However, we are open to exploring future methods that might enhance this diversity/quality tradeoff by better integrating trajectory-preserving techniques (such as the more efficient consistency distillation) with distribution matching methods. We believe this is a promising direction for further research.\n\n**Why intermediate outputs of DMD2 follow a certain trajectory?**\n\nThis observation was initially surprising to us as well. Although our generator does not follow the teacher diffusion model’s sampling trajectory, our training methods lead to few-step generators where the first output significantly shapes the general structure. Subsequent images tend to closely resemble this initial output. This effect is likely influenced by the relatively high signal-to-noise ratios in subsequent sampling steps, which preserve much of the structure even after noise injection.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4HUvroOypm", "reviewer": "Reviewer_uheV", "summary": "This work proposed an improved training method for distribution matching distillation, named DMD2. Notably, compared to DMD, it does need the regression loss which relies on constructing the synthetic noise-data pairs. Instead, DMD2 introduces three new features: 1) a two time-scale update rule for fake score and generator training, 2) a GAN loss which extracts the bottleneck features of fake score network for a training prediction head as the discriminator, and 3) backward simulation for few-step distillation that produces inference-time generator inputs during training.", "questions": "- I’m curious about the extra cost of backward simulation, i.e., producing synthetic images with the current student generator running several steps. Is it possible to compare the training time per iteration with and without backward simulation?\n- From the numbers in Table 2, it looks like the 4-step distillation improves Patch FID but gets worse FID and CLIP, compared to 1-step distillation. Any justification?", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and easy to read.\n- The new distribution matching distillation technique (DMD2) significantly improves over DMD with several well-justified innovations, such as TTUR, GAN loss and backward simulation.\n- DMD2 achieves SOTA performance on ImageNet-64 and COCO 2014 with one-step generation, and can achieve SOTA performance in distilling SDXL to a 4-step generator, measured by sufficient automatic metrics and human studies.\n- Ablation studies have been well executed to highlight the importance of each introduced feature.", "weaknesses": "- The proposed method introduces many hyperparameters and seems to be sensitive to these hyperparameters, such as batch size, guidance scale for teacher score, GAN loss weighting and fake score updating frequency. For instance, in different distillation tasks (EDM on ImageNet, SDv1.5 and SDXL), these hyperparameters are different (as shown in Appendix G). I’m not sure if the hyperparameters need to be specifically tuned for good performance in each distillation task. \n- The authors claimed that “SDXL remains challenging to distill into a one-step generator because of limited model capacity and a complex optimization landscape to the direct mapping from noise to highly diverse and detailed images”. For the point of “limited model capacity”, does it mean that if the student has the same network with SDXL, by any means, we are not able to achieve a one-step generation that matches SDXL’s performance? For the point of “a complex optimization landscape”, it seems that both SDv1.5 and SDXL are trained on LAION dataset, which means they are both trying to learn the same mapping from noise to data. Does it mean the higher-quality generation of SDXL (rather the training data of teacher models) hinders the one-step distillation? On the other hand, I wonder if it is possible to tune the hyperparameters of distilling SDXL for a better one-step generator. For example, in Appendix G, the batch size for distilling SDXL is only 128 while the batch size for distilling SDv1.5 is only 2048. If DMD2 is sensitive to batch size in the large-scale text-to-image case, can we increase the batch size for distilling SDXL to 2048 during training for improved performance? \n- There are some inconsistencies: 1) In Figure 4, the caption says the teacher uses 50 sampling steps while the main text says “while requiring 25x fewer forward passes (4 vs 100)”. 2) EDM (Teacher ODE) originally reported their FID as 2.22, while this work reports 2.32 in Table 1. 3) Both $\\mu_{\\text{real}}$ and $\\mu_{\\text{fake}}$ are introduced without definition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer uheV's constructive feedback. Below, we address the remaining concerns.\n\n**How to select the set of hyperparameters?**\n\nThank you for your question. Our approach to selecting hyperparameters is straightforward and consistent across all datasets. We utilize the maximum batch size our compute setup allows. The learning rate is determined as the highest value that does not cause divergent loss within the first 500 iterations. We set the number of TTUR iterations to the minimum required for stability. The guidance scale is chosen based on what yields the best results for the teacher model. For the GAN weight, our method works well across a wide range of values, as demonstrated by the following ImageNet FID scores:\n\n| Weight  | ImaegNet FID |\n| - | - |\n| 2e-3 | 1.31 | \n| 3e-3 | 1.28 | \n| 4e-3 | 1.28 | \n| 5e-3 | 1.26 | \n| 1e-2 | 1.30 |\n\nWe will include these guidelines in our updated version. \n\n**For the point of “limited model capacity”, does it mean that if the student has the same network with SDXL, by any means, we are not able to achieve a one-step generation that matches SDXL’s performance?**\n\nWe are open to the possibility of a one-step generation that matches the performance of the teacher model. However, we want to emphasize that achieving this in a single-step generation is substantially more challenging than in a multi-step process, especially when distilling complex networks like SDXL. While our one-step generator can match the teacher model in quantitative metrics such as FID, qualitative aspects of image quality present greater challenges. Visually, the one-step process still produces some artifacts that are difficult to eliminate (See Figure 11). An exciting direction for future research could involve refining our training objectives to address these issues more directly, as seen in recent developments like those explored in HyperSD [1].\n\n**For the point of “a complex optimization landscape”, it seems that both SDv1.5 and SDXL are trained on LAION dataset, which means they are both trying to learn the same mapping from noise to data. Does it mean the higher-quality generation of SDXL (rather the training data of teacher models) hinders the one-step distillation?**\n\nWhile both models are trained on the LAION dataset, they likely utilize different subsets, leading to variations in the noise-to-data mapping. The higher quality and particularly the subtler details found in SDXL are indeed more challenging to capture, potentially requiring more advancements in loss design. \n\n **I wonder if it is possible to tune the hyperparameters of distilling SDXL for a better one-step generator. For example, in Appendix G, the batch size for distilling SDXL is only 128 while the batch size for distilling SDv1.5 is only 2048. If DMD2 is sensitive to batch size in the large-scale text-to-image case, can we increase the batch size for distilling SDXL to 2048 during training for improved performance?**\n\nWe have observed improved performance with increased batch sizes and compute. We used a batch size of 128 for SDXL, as this is the maximum our current setup can accommodate within our compute budget of 3 days—especially considering that the SDXL model is three times larger than SDv1.5. Exploring a larger batch size, such as 2048, with additional resources in the future would indeed be an interesting experiment to potentially enhance performance further.\n\n**Some Writing Inconsistency**\n\nThank you for pointing out these inconsistencies. Regarding the discrepancy between 50 and 100 steps in Figure 4, the teacher model uses 50 steps, but effectively has 100 forward passes due to the application of classifier-free guidance. For EDM, upon reevaluating the released model, we recorded a FID of 2.32, which we will update to 2.22 in our revised version to reflect the most accurate data. We appreciate your attention to detail and will correct the remaining issues. Thank you again for your valuable feedback.\n\n**Extra cost per training iteration of backward simulation**\n\nThank you for raising the issue of training computational cost. The training iteration time for the model with backward simulation is approximately 9.2 seconds per iteration, compared to 7.8 seconds for the model without backward simulation. We also discovered that enabling backward simulation only during the last 20% of the training epochs yields comparable results. This approach offers a more computationally efficient option when resources are limited. \n\n**From the numbers in Table 2, it looks like the 4-step distillation improves Patch FID but gets worse FID and CLIP, compared to 1-step distillation. Any justification?**\n\nFrom the data in Table 2, the difference in FID between the one-step and four-step distillation (0.3) is generally within the range of variability observed across different runs, indicating comparable performance. The significant improvement in Patch FID for the four-step distillation reflects better local image detail, aligning with qualitative improvements we observed. However, the CLIP score did decrease, which may suggest that the four-step method slightly sacrifices prompt alignment for image quality. This also happens for previous approaches with good text to image alignment like SDXL-Turbo.\n\n[1] Ren, Yuxi, et al. \"Hyper-sd: Trajectory segmented consistency model for efficient image synthesis.\" arXiv preprint arXiv:2404.13686 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jdg6f4HO2G", "reviewer": "Reviewer_xaYh", "summary": "The authors propose an improved way of distilling image-generating diffusion models into fast models capable of generating high-quality images with as few as 1-4 steps. Compared to prior work using distribution-matching (DMD), they do away with the regression loss term that tied the teacher path to the student path. They also introduce a GAN-style loss in the pipeline, and introduce a few other tricks to squeeze out extra performance. They demonstrate SOTA performance among efficient models, and even beat the teacher model (made possible by training using real images and a GAN loss).", "questions": "What is the most significant difference between this work and the line of work by Sauer et al ([23, 24])? Could one sentence about that be added to the \"related work\" section?\n\nThe original DMD paper [22] demonstrated examples of mode collapse when omitting the regression loss. GANs (at least some) are also known to be prone to mode collapse. Could you mention anything about the mode collapse situation in this method? (Since the FID numbers are good, I assume that this is also good, but since this was a main point of analysis in [22]?)", "limitations": "Yes, limitation and potential social impact are well-described in section 6 and A.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Overall, I found this to be an excellent paper. The work is well-motivated, addressing an important problem, and will likely be interesting to a large audience. The writing is excellent, with all concepts being well explained. The experimental coverage is excellent, with evaluation on two datasets and with a user study, leaving nothing more to be desired. Their performance numbers are also convincing. Finally, their attention to detail on experimental parameters also looks very thorough, giving confidence that their results can be reproduced.\n\nOn a more detailed level, I personally read the original DMD paper not too long ago, and found their regression loss to be slightly unsatisfactory, since it ties the student generation paths to the teacher paths in a way that seems contrary to the idea of the distribution matching loss. Therefore, I was happy to see in this paper that one can do away with this term.", "weaknesses": "There's not so much to say here, since I found most aspects of the paper to be excellent. However, I would have liked to see some more details in how their approach relates to competing approaches. Most notably, the line of work by Sauer et al [23, 24] use SDS and a GAN-style loss. Now that a GAN-style loss is introduced in the DMD framework, the gap between these two approaches gets smaller, and it would be nice to see some more explanation about what the core difference is.\n\nThe progress is mostly empirical in nature. For example, the authors note that the training becomes more stable using the two-scale update rule, but they don't present any theoretical convergence guarantees (which is perfectly fine, the empirical progress is definitely good enough in my opinion).\n\nThere are a few minor typos (e.g. line 122, should be \"gradient of the data log likelihood\"), but few enough not to impair the overall understanding (and I trust the authors to do a final proof reading for the camera ready version).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer xaYh's constructive feedback. We will fix all typos. Below, we address the remaining concerns.\n\n**How is DMD2 related to ADD and LADD? What is the most significant difference?**\n\nThank you for the opportunity to discuss how DMD2 relates to other concurrent GAN based methods such as ADD and LADD. The ADD paper [23] utilizes a pretrained DINO-based classifier in pixel space, which we found to be less efficient in terms of training and leading to reduced diversity, as detailed in Table 6. More recent efforts like LADD, UFOGEN, and SDXL-Lightning employ latent diffusion GAN-based networks akin to our DMD2. A significant difference, which is often overlooked, is that DMD-style training inherently integrates classifier-free guidance directly into the real score of the DMD gradient. This integration simplifies the training process of our model. In contrast, purely GAN-based methods typically struggle to incorporate classifier-free guidance directly. For example, SDXL-Lightning needs to combine GAN with progressive distillation to enable CFG, while LADD relies on diffusion generated images with CFG as real data in its GAN discriminator. These approaches often complicate the training process. We will elaborate further on these distinctions in the related work section of our revised paper.  \n\n**The original DMD paper [22] demonstrated examples of mode collapse when omitting the regression loss. GANs (at least some) are also known to be prone to mode collapse. Could you mention anything about the mode collapse situation in this method? (Since the FID numbers are good, I assume that this is also good, but since this was a main point of analysis in [22]?)**\n\nSimilar to our response to Reviewer xYYF, we assess the mode collapse situation under two scenarios:\n\n- Class-Conditional Image Generation: We observed no mode collapse, as evidenced by the state-of-the-art FID scores for image generation (see Table 1).\n- Text-to-Image Generation: This scenario is more complex. We utilize classifier-free guidance, which typically trades diversity for image quality. At high guidance settings, using the SDXL baseline, we achieved superior image quality with diversity comparable to or better than other distillation methods but slightly worse than the teacher model ((see Table 6 and the new results in our Response to Reviewer aF5v). \n\nOur current framework trains stably and produces excellent image quality. However, there remains a small gap in output diversity compared to the original teacher model and we are open to exploring future methods that might enhance this diversity/quality tradeoff by better integrating trajectory-preserving techniques (such as the more efficient consistency distillation) with distribution matching methods. We believe this is a promising direction for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their constructive feedback. We are grateful for the positive reception of our work, which has been recognized for its well-founded innovations and outstanding quality. Our DMD2 model facilitates the training of a few-step generator that delivers superior image quality and diversity comparable to the teacher SDXL models. We have incorporated additional evaluations concerning human-related and diversity metrics as requested by Reviewer **Yut7** and Reviewer **aF5v**. Detailed responses to each reviewer’s specific concerns are provided below.

Meta-review: This paper presents DMD2, a few-step distilled generator designed to achieve fast sampling while preserving the high-generation quality of multi-step diffusion models. The paper introduces several key enhancements to the original DMD's training procedure, including: (1) replacing the regression loss with the Two-Time Scale Update Rule to stabilize training, (2) integrating GAN loss to improve generation quality, and (3) employing backward simulation to address potential mismatches between training and inference. Experimental results show that DMD2 achieves state-of-the-art performance, outperforming the original DMD and other competitive models in both image quality and efficiency.

All reviewers unanimously acknowledged the novelty and effectiveness of this work, leading to its acceptance. The authors should also revise their paper according to the reviewers' suggestions in the final version.
Decision: Accept (oral)