One-Step Diffusion Distillation through Score Implicit Matching
Authors: Weijian Luo
Zemin Huang
Zhengyang Geng
J Zico Kolter
Guo-Jun Qi
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion mode
Diffusion Distillation
Text-to-Image Generation
Generative Adversarial Network
Primary area: diffusion_based_models
TL;DR: The submission propose an general one-step diffusion distillation framework that induces efficient and stable instances with very strong performances.

Abstract: Despite their strong performances on many generative tasks, diffusion models require a large number of sampling steps in order to generate realistic samples.  This has motivated the community to develop effective methods to distill pre-trained diffusion models into more efficient models, but these methods still typically require few-step inference or perform substantially worse than the underlying model.  In this paper, we present Score Implicit Matching (SIM) a new approach to distilling pre-trained diffusion models into single-step generator models, while maintaining almost the same sample generation ability as the original model as well as being data-free with no need of training samples for distillation.  The method rests upon the fact that, although the traditional score-based loss is intractable to minimize for generator models, under certain conditions we \emph{can} efficiently compute the \emph{gradients} for a wide class of score-based divergences between a diffusion model and a generator. SIM shows strong empirical performances for one-step generators: on the CIFAR10 dataset, it achieves an FID of 2.06 for unconditional generation and 1.96 for class-conditional generation.  Moreover, by applying SIM to a leading transformer-based diffusion model, we distill a single-step generator for text-to-image (T2I) generation that attains an aesthetic score of 6.42 with no performance decline over the original multi-step counterpart, clearly outperforming the other one-step generators including SDXL-TURBO of 5.33, SDXL-LIGHTNING of 5.34 and HYPER-SDXL of 5.85. We will release this industry-ready one-step transformer-based T2I generator along with this paper.

One-Step Diffusion Distillation through Score Implicit Matching
Weijian Luo"
Zemin Huang Westlake University
Peking University
luoweijian pku. edu. en
muangzemin@westlakee edu.eeeeducc. en
Zhengyang Geng
J. Zico Kolter
Guo-jun Qii
Carnegie Mellon University zgeng20cs. cmu eddu
Carnegie Mellon University
Westlake University guo gua com
zkolteråcs cmu.
ht ht http:///gttuub.///m/.. hecplle-reeeerchhee 1ab/ SIM
Abstract
Despite their strong performances many generative tasks, diffusion models require a large number of sampling steps in order to generate realistic samples. This has motivated the community to develop effective methods to distill pre-trained diffusion models into more efficient models, but these methods still typically require few-step inference perform substantially worse than the underlying model. In this paper, we present Score Implicit Matching (SIM) a new approach to distilling pre-trained diffusion models into single-step generator models, while maintaining almost the same sample generation ability as the original model as well as being data-free with need of training samples for distillation. The method rests upon the fact that, although the traditional score-based loss is intractable to minimize for generator models, under certain conditions we can efficiently compute the gradients for wide class of score-base score-based divergences between diffusion model and a generator. SIM shows strong empirical performances for one-step generators: on the CIFAR 10 dataset, it achieves an FID of 2.06 for unconditional generation and 1.96 for Blss-condiitonn generation. Moreover, by applying SIM to a leading transformer-bast
diffusion model, we distill a single-step generator for text-to- image (T2I) generation that attains an aesthetic score of 6.42 with no performance decline over the original multi-step counterpart, clearly outperforming the other one-step generators including BURBO 5.33, SDXI-LIGHTNING of 5.34 and HYPER-SDX of 5.85. We will release this industry-reade one-step transformer-based T2I generator along with this paper.
Introduction
Over the past years, diffusion models (DMs) [2 67 65]
have shown significant advancements across
a broad spectrum of applications, ranging from data synthesis [25]
26
51
52
22
[66]
23
31
to
density estimation
332
8), text-to-image generation 54 60
7 text-to-3D creation |6
74
34
image editing
47
9 19
1 30 49), and beyond 83 79 5
85
18
59
14|
73
89
72
42
7
84
46
16
55
10 From high level point of view. diffusion models, also framed as
score-based diffusion models, use diffusion processes to corrupt the data distribution. They are then
trained to approximate the score functions the noisy data distributions across varying noise levels.
Alternative email: Muulyj199@@icood.cco
t *Correspondence to Guo-jun Qi. The project was initiated and supported by the MAPLE lab of Westlake University.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

hateyound
aftroeaaiitt hop Syunne poots""
tlen stw wost soachig ngy copped aboy San prirt anatinn nurter moatt sirte topued mosts colder higo contey moeutty the
dif's
orgiet brolicgp of Tadil lord Begeman."
T0.Shinn Ziro wng Noter uy tatbook.
Figure 1: Time for Human Preference Study! Could you please tell which one is better? Hint: rightmost column one-step Latent Consistency Model PixelArt-o; The left two columns randomly placed, with generated from one-step SIM-DiT-600M model, and another generated from the 14-step PixelArt-a teacher diffusion model. We put the answer in Appendix B.I
Diffusion models have multiple advantages, such as training flexibility, scalability, and the ability to produce high-quality samples, making them favored choice for modern AIGC models. After training, the learned score functions can used to reverse the data corruption process, which can be implemented by numerically solving the associated stochastic differential equation. Such data generation mechanism usually requires many neural network evaluations, which leads to significant limitation of DMs: the generation performance of DMs degrades substantially when the number of sampling steps is reduced. This shortcoming restricts the practical deployment DMs, particularly where quick inference is crucial, such as devices with limited computational capacities like mobile phones and edge devices, or in applications requiring rapid response times.
This challenge has spurred a variety of approaches aimed at expediting the sampling process of diffusion models while preserving their robust generative capabilities. Distillation approaches, in particular, focus on applying distillation algorithms to transition the knowledge from pre-trained, teacher diffusion models to efficient mudent--enneaate models which are capable of producing high-quality samples within few generation steps.

---

Some works have studied the diffusion distillation algorithm through the lens of probability divergence minimization. For instance, Luo al. 49 Yin et al. [82] have studied the algorithms that minimize the KI. divergence between teacher and one-step student models. Zhou et al. 9 have explored distilling with Fisher divergences, resulting in impressive empirical performances. Though these studies have contributed to the community in both theoretical and empirical aspects with applicable single-step generator models, their theories are built upon specific divergences, namely the Kullback- Leibler divergence and the Fisher divergence, which potentially restrict the distillation performances. A more general framework for understanding and improving diffusion distillation is still lacking.
In this work, we introduce Score Implicit Matching (SIM), a novel framework for distilling pre- pre-trained diffusion models into one-step generator networks while maintaining high-quality generations. To do so, we propose a wide and flexible class of score-based divergences between the (intractable) score function of the generator model and that of the original diffusion model, for arbitrary distance functions between the two score functions. The key technical insight of this work is that although such divergences cannot be computed explicitly, the gradient of these divergences can be computed exactly using a result we call the score-gradient theorem, leading to an implicit minimization of the divergence. This lets us efficiently train models based on such divergences.
We evaluate the performance of SIM compared to previous approaches, using different choices of distance functions to define the divergence. Most relatedly, we compare SIM with the (DI) 43) method, which uses a KL-based divergence term, and the Score Identity Distillation (SiD) method (93] which we show be a special case of our approach when the distance function is simply chosen to be the squared T22 distance (though derived in an entirely different fashion). We also show empirically that SIM with a specially-desinned Pseudo-Hube distance function shows faster convergences and stronger robustness to hyper-paramest than L2 distance, making the resulting method substantially strong than previous approaches.
Finally, we show that SIM obtains very strong empirical performance in absolute terms relative to past work in the field on CIFARIO image generation and text-to-image generation. On the CIFARIO dataset, SIM shows one-step generative performance with Frechet Inception Distance (FID) of 2.06 for unconditional generation and 1.96 for class-conditional generation. More qualitatively. distilling a leading diffusion wwatanttttaaor-bbbbbbeeee extremely capable one-step text-to sox--to-mmage generator which we show almost lossless in terms of generative performances as teacher diffusion model. Particularly, by applying SIM to PixelArt-o- 77 a single-step generator is distilled that reaches an outstanding aesthetic score of 6.42 with no performance decline over the original multi-step diffusion model. This remarkably outperforms the other one-step lext-to-image generators including SDXL-TURBO (44 of 5.33, SDXL-LIGHTNING 335 of 5.34 and HYPER-SSDLL (57 of 5.85. Such result not only marks new direction for one-step text-to-image generation but also motivates further studies of distilling diffussoontrranffomeer-- transformer-based AIGC models in other domains such video generation.
[53] text-to-image diffusion model results in an
2 Diffusion Models
In this section, we introduce preliminary knowledge and notations about diffusion models and diffusion distillation. Assume we observe data from the underlying distribution (a(x)) The goal of generative modeling is to train models to generate new samples æ ~ (z) The forward diffusion process of DM transforms any initial distribution qo qu towards some simple noise distribution,
dæ, Fat,, t) + G(t)dw:
(2.1)
where F' is a pre-defined drift function, G(t) is pre-defined rcalar-value diffusion coefficient, and w; denotes independent Wiener process. A muntinoous-indexe score network s,, t) is employed to approximate marginal score functions of the forward diffusion process (2.).. The learning of score
networks is achieved by minimizing weighted denoising score matching objective (70 67).
1--0 M())E / (2.,)a)))))) 1 (x,, Va, ax log (x
(DSM(()) ---
(2.2)
Here the weighting function (() controls the importance of the learning at different time levels and g: (2))) denotes the conditional transition of the forward diffusion (2. After training, the score network (x, 1) Ri Vn, log q: (ax)) is a good approximation of the marginal score function of the diffused data distribution. High-quality samples from a DM can be drawn by simulating SDE which
3

---

is implemented by the learned score network [67] However, the simulation of an SDE is significantly slower than that of other models such as one-step generator models.
Score Implicit Matching
In this section, we introduce Score Implicit Matching which general method tailored for the one-step distillation of score-based diffusion models. We first introduce the problem setup and notations, then introduce a general family of score-based probability divergences and show how SIM can be used to minimize the mentioned divergences, We finally discuss specific choices of the method, such as the choice of distance function, and explore the effect this has on the distillation.
Problem setup.
Our starting point is pre-trained diffusion model specified by the score function
8q. (c,) = V log (xxe)
(3.1)
where qe (xbe)) 's are the underlying distribution diffused at time t according to (2.1). We assume that the pre-trained diffusion model provides a sufficiently good approximation of data distribution, and thus will be the only item consideration for our approach.
The student model of interest is single-step generator network go. which can transform an initial random noise ~ P= to obtain a sample æ = go =>); this network parameterized by network parameters 0. Let Pa o denote the data distribution of the student model, and Pa.: denote the marginal diffused data distribution of the student model with the same diffusion process (2.I). The student distribution implicitly induces score function
Spa,e (zt) = Vz, log (xx))
(3.2)
and evaluating it is generally performed by training an alternative score network as elaborated later.
3.1 General Score-based Divergences
The goal of one-step diffusion distillation is to let the student distribution p8,0 match the data distribution Jo- To do so, we propose to match the diffused marginal distribution Po,: and at all diffusion time levels. We can define such objective via the following general acore-based divergence. Assume
Rd - IR is scalar-valued proper distance function (i.e., a function that obeys Va, d(x) 2 0 and dlxx) 0 if and only if æ 0). Given sampling distribution n, that has larger distribution support than p: and we can formally define a time-integral score divergence as
1" w(t w()))mmmt [d(sp.. (xx)) S.. (2x)) jat,
D Diu.I(
(3.3)
where Pt and d denote the marginal densities of the diffusion process 2.1 at time & initialized with and p respectively. w(t) is an integral weighting function. Clearly, we have DIO.TI q) 0 if and only if all marginal score functions agree, which implies that po pa(x)) go(x);), a.s. mg-
3.2 Score Implicit Matching
Based upon this motivation, would like to minimize the integral score-based divergence between Pe and in order to train the student model, i.e..
[ w(t) [d [d(spo. ) Szg (2xe) J.=0
(()) DIO.T (po.
(3.4)
where we assume that the distribution ne has no parameter dependence of 0, such as vh(()))) = PaKla] (x:e) Taking the gradient with respect we have
a a0 ((0) [' (()) d' d'(spa
θ $ po, ( dt.
Sy. (2xe) 20
(3.5)
where d' denotes the derivative of wrt. its inputs, i.e. Vyd(y). Unfortunately. because the score function not tractable, it is impossible to compute 50 po. (wte) directly, rendering such a direct approach impractical.

---

Algorithm 1: Score Implicit Matching for Diffusion Distillation. (Pseudo-code in Appendix (A.2)
Input:
pre-trained DM &q. (.), generator go- prior distribution Pz, online DM (.)::);
differentiable distance function d(),, and forward diffusion 2 while not converge do with frozen 0, update &b using SGD with gradient
Grad (1))
(t) smp so moote), as (m
[18p Va,, log qe (xx (2))
Orp L=
with frozen ub, update using SGD with the gradient
0
{ (-d'(vi)) me 00
} (ou((e)))) t)- -) log (x)) fat,
Grad(O) = 00 1=0
w(t)B xmpa mopo)))
where yt = sy (Ic, t) (x)e.
end
return 0, vb.
Fortunately, a key finding of our paper is if we choose the sampling distribution to the diffused implicit distribution, i.e. n, = Ps where the notation sg[0] denotes the stop gradient operator that cuts off the parameter dependence of 6, the loss function (3.4p along with its intractable gradient (3.5) can be minimized efficiently via an gradient-equivalent loss. This relies on our Theorem 3.1
Theorem 3.1 (Score-divergene gradient Theorem). If distribution po. satisfies some mild regularity conditions, we have for any score function $q. (.), the following equation holds for all parameter 6:
d' d'(pp..c Ex--Pid (xet) 89. (zze) 30 pp, (ume)
(3.6)
[d (xt) Sq, (2x1)
{
Speeto, (xt) V log (2)2) }}
so Ro-PPO0 wy (igg (ieg//o
30
The key observation here is that we replace the intractable gradient of the score function on the left-hand side of (3.6) with much affordable evaluation of the score function the right-hand side, the latter of which can be accomplished much more easily using a separate approximation network. This theorem bridge denoising score matching with denoising auto-encoders. However, the key in proving Theorem 3.1 is a proper choice of 0-parameter (in)dependence by appropriately stopping the gradients shown in this theorem. We provide the detailed proof in Appendix A.
be proved by using acore-projection identity [70 [3] which first introduced to
Now it is ready to reveal the objective we will use train the implicit generator ga- A direct result of 3.6 is the gradient (3.5)
can be realized via minimizing a tractable loss function
(S)M((0=
w()EE
d' (zv))
Peglo) (xx)) Van, log qi (xx (1x jol (3.7)
awop dng (x) me me ong)
with y: == 8 Pegto) (xx)) Sq, (.)). By Theorem 3.1 this alternative loss has an identical gradient to that the original loss without the need access the gradient of the score network.
In practice, we can use another online diffusion model s ( t) to approximate the generator model's score function settee, (axz) pointwise, which was also done in previous works such Luo et al. (43, Zhou et al. (93], and Yin et al. 82 We name the distillation method that minimizes the objective LSIM (0) in 3.7pp the Score Implicit Matching (SIM) because the learning process implicitly matches the intractable marginal score function pp. (-) of the implicit student model with the explicit score function of the pre- -rettrined diffusion model $ (.)...
The complete algorithm for SIM is shown in Algorithm which trains the student model through two alternative phases between learning the marginal score function 8y, and updating the generator model with gradient (3.). The former phase follows the standard DM learning procedure, i.e., minimizing the denoising score matching loss function (2.2), with slight change that the sample is generated from the generator. The resulting sy (x t) provides good pointwise estimation of Spant) (x)).
5

---

The latter phase updates the generator's parameter & by minimizing the loss function (3.7 where two needed functions are provided by pretrained DM sy. (2x) and learned DM sy (x
3.3 Instances of Score Implicit Matching.
The previous section introduced the SIM algorithm without choosing a specific distance function d(.). Here discuss different choices and their influence on the distillation process. We also show that in the SIM framework, the SiD can be viewed as a special case.
The Design Choice of Distance Function d(.). Clearly, various choices of distance function d()) result in different distillation algorithms. Perhaps the most natural choice of the distance function is simple squared distance, i.e. ( = 3 12. The corresponding derivative term writes d'y))) = 2y.- In fact, such a loss function recovers the delta loss studied in SiD [93], which the authors empirically find that such a loss function works satisfactorily (though through very different derivation). Thus, SiD is in fact a special case of SIM, though the derivation of SiD there does not suggest how alternative losses may be employed. A direct generalization of the quadratic form is the a-power of the a-norm where a > and a is even. In this case, the distance function writes ( = oy, (a-1) and the resulting loss function is summarized in Table in Appendix A.3
The Sgeudo-Huber distance function. Different from powered norms, we introduce SIM with the Pseudo-Huber distance function, which is defined with d(y) == v /ly.112 c² C, where is pre-defined positive constant. The corresponding distillation objective writes
(SIM =
y
{
(02, Van, log (1)
(3.8)
yı 1
In the rest of this paper, we will use the Greudo-Huber distance as the default choice of the distance, unless specified otherwise. Due to the limited space, we summarize different choices of distance function and the corresponding loss functions in TableAaas as well as their derivations, along with more discussions in Appendix A.3
Particularly, unlike SiD (the T,2 case in Table 4) with the Pseudo-Huber distance in the SIM, we observe that the vector y/ is naturally normalized adaptively by dividing by squared root of the vector. Such a normalization can stabilize the training loss, resulting in a robust and fast-converging distillation process. In section 4.1 we conduct empirical experiments to show three advantages: robustness to large-learning rate, fast convergence, and improved performances.
3.4 Related Works
Diffusion distillation 41 is a research area that aims to reduce generation costs using teacher diffusion models. It involves three primary distillation methods: 1) Trajectory Distillation: This method trains student model to mimic the generation process of diffusion models with fewer steps. Direct distillation (139 (5)) and progressive distillation (161
48 variants predict less noisy data minimize the self-consistencc metric. These require true data samples for training. 2) Distributional Matching: It focuses on aligning the student's generation distribution with that of a teacher diffusion model. Among them are adversarial training methods (176) (77) requiring real data for distilling diffusion models. Another important line of methods attempts to minimize divergences like KL (82) such as Diff-Instruct (DI) 45 82 and Fisher divergence such as Score identity Distillation (SiD)
from noisy inputs. Consistency-based methods (168) 29 66 36 17)
[33] often without
needing real data. Though SIM has gotten inspiration from SiD and DI, the gap between SIM and SiD and DI is significant. SIM not only offers solid mathematical foundations which may lead to deep understanding of diffusion distillation, but also provides substantial flexibility in using different distance functions, resulting strong empirical performances when using specific Pseudo-Huber distance. 3) Other Methods: Methods like operator learning (186), ReFlow ((37), and FMM 3 provide alternative insights into distillation. Moreover, many works made outstanding efforts to scale up diffusion distillation to one-step text-to-image generation and beyond 50 69 82 92

---

Table 1: Unconditional sample quality CIFAR- 10. f means method we reproduced.
Table 2: Class-conditional sample quality CI-
FAR10 dataset. f means method we reproduced.
METHOD
NFE (U)
FID
METHOD
NFE (J)
FID (U)
DIFTERENT ARCHITECTURE AS EDM MODEL
DIFFERENT ARCHITECTURE EDM MODEL
3.17
BIGGAN
GAN(T=N) 776
BIGGAN+TUNE
38
9.36
STYLEGAN2
6.96
TDPM 90
8.91
MULTIHINGE
6,40
DENO
4.12
FQ-GAN S7
5.59
3-REFLDW (+DISTILL)
371
5.21
STYLEGAN2-ANE AAAAAAAA 2
2.42
STYLEGAN2-ADA
2.92
STYLEGAN2-ADA-DI
2.27
STYLEGAN2-ADA-D
2.71
STYLEGAN2 + SMART
66
2,06
SAME ARCHITECTURE AS EDM 26) MODEL
STYLEGAN-XI 63
1.85
28
SAME ARCHITECTURE EDM 26 MODEL
5.13
EDM 26
2.93
EDM 26
10
15.56
GET 15
6.91
EDM 26
314.81
CT 68
8.70
GET 15
6.25
ICT-DEEP 66
2.24
DIRF-INSS INSTRICT
4.19
WIRI-NNSTRUC
4,53
DMD (W.D. RBG)
5.58
DMD X
3.77
DMD (W.D. KL)
82
3.82
CTM 25
1.98
DMD 2
2.66
CTM 20
1.87
CTM 20
1.73
SID (o
1.0)
9
1.92
CTM 20
2
1.63
SID (o-
1.2) 93
2.02
SID
1.0) 93
1.93
D1
3.70
StD
1.2)
1.71
SIM (OURS)
2.06
SIM (OURS)
1.96
Experiments
4.1 One-step CIFARIO Generation
Experiment Settings. In this experiment, we apply SIM distill the pre-trained EDM [26 diffusion models into one-step generator models on the CIFAR10 [33] dataset. We follow the same setting as DI 43 and SiD 93 to distill the diffusion model into a one-step generator. Details can be found in Appendix B.2 We refer to the high-quality codebase of SiD [93 to reproduce its results by closely referring to configurations on our devices. We also re-implement the DI under the same experiment settings.
Performances. We evaluate the performance of the trained generator via Frechet Inception Distance (FID) 200 which is the lower the better. We refer to the evaluation protocols in [43) for comparison Table I and summarize the FID of generative models CIFARIO datasets. We reproduce the SiD and the DI with the same computing environments and evaluation protocol as SIM for fair comparison. Models in the upper part of the table have different architectures diffusion models from the EDM model, while the models in the lower part of the tables share exactly the same architecture and the teacher EDM diffusion models, which thus directly comparable.
As shown in Table for the CIFARIO 10 unconditional generation task, the proposed SIM achieves a decent FID of 2.06 with only one generation step, outperforming SiD and DI with the same training compute. is on par with the CTM and the SiD's official implementation which are trained to fully converge with training costs of hundreds of GPU days. For the class-conditional generation in Table 2. the SIM achieves an FID of 1.96, acting among top-performing models.
The CIFAR-10 10 generation tasks are much toyish as merely performed with diffusion models of limited capacities on a simple dataset. We will perform experiments to distill from top-per cop-performing transformer-base diffusion models for text-to- image generation tasks. We will show that the one-step T2I generator distilled by SIM demonstrates state-of of-the-art results over other industry-lever models.
instruce
7

---

Figure 2: Left Two: Comparison of distillation methods with batch size of 256 and learning rate of -4. 4. (Left): the FID value. (Right): the Inception Scores. Right Two: Comparison of distillation methods with a batch size of 256 and learning rate of 1e 5. (Left): the FID value. (Right): the Inception Scores. All methods are constrained to the same settings except for the distillation methods.
Before that let further look into some advantages of SIM robustness to large learning rate and faster convergences - over SiD and DI on CIFAR-10, 10, which will shed some light on how distillation methods scale up to more complex tasks with much larger neural networks.
Robustness to large learning rate. We apply SIM, SiD, and DI under the same settings to distill from EDM (details in Appendix) the CIFARIO 10 unconditional generation task, with learning rate
of le - 4, and plot the Fretchet Inception Distance (FID) (0) and the Inception Score (62 in Figure
2 Both the DI and the SiD are unstable even in the early training phase, while the SIM can steadily
converge even with a large learning rate. The potential reason is that SIM naturally normalizes the
loss objective to keep its scale from changing abruptly along the training process. This distinguishes SIM from SiD in practice for training large models, because training modern large models is so expensive that researchers often have few chances to adjust the hyperparameters within budget.
Fast convergence. The second advantage of SIM is its faster convergence than SiD F To show this, we follow the same setting as SiD on CIFAR 10 unconditional generation. As shown Figure and Figure ??, under all configurations, the SIM consistently shows better FID and Inception Scores under the same training iterations. Due to page limitations, we put more details Appendix B.2
Experiments on CIFARIO generation show that SIM is a strong, robust, yet fast converging one-step diffusion distillation algorithm. However, the power of SIM is not restricted a toy CIFAR-10 benchmark. In section 4.2] we apply the SIM to distill a 0.6B DiT (53)) based text-to-image diffusion model and obtain the state-of-the-at transformer-bas based one-step generator.
4.2 Transformer-based One-step Text-to-Image Generator
Experiment Settings. In recent years, wansformer-basee text-to-X generation models have gained great attention across image generations such as Stable Diffusion V3 [12] and video generation such as Sora (6. In this section, we apply SIM to distill one of the leading open-sourced DiT-based diffusion models that have gained lots of attention recently: the 0.6B PixelArt-a model 77, which built upon with DiT model [53), resulting in the state-of-the-art one-step generator in terms of both quantitative evaluation metric and subjective user studics.
Experiment Settings and Evaluation Metries. The goal of one-step distillation is to accelerate the diffusion model into one-generation steps while maintaining or even outperforming the teacher diffusion model's 's performances. To verify the performance gap between our one-step model and the diffusion model, we compare four quantitative values: the aesthetic score, the PickScore, the Image Reward, and our user-studied comparison score. On the SAM-LLaVA-Captionn 10M, which is one of the datasets the original PixelArt-a model is trained on, we compare the SIM one-step model, which we called the SIM-DiT-600M, with the PixelArt-a model with a 14-step DPM-Solver[SM (38 to evaluate the in-data performance gap. We also compare the HIM-DIT-600N and Art-ax with other few-step models, such as LCM [40), TCD (91) PeReflow 81). and Hyper-SD (57] series on the widely used COCO-2017 validation dataset. We refer to Hyper-SD' SD's evaluation protocols to compute evaluation metrics. Table summarizes the evaluation performances of all models. For the human preference study against PixArt-a and SIM-DiT-6OMS from the SAM Caption dataset and generate images with both PixArt-a and SIM-DiT-600M, then
randomly select 17 prompts
We find that the DI converges fast but suffers from mode-collapse issues. So do not compare with it.
8

---

SIM (1 (T STEg)
80AL Hypor stog)
SCAL-TCD 29055
SDXL-Ture 5tog)
SE9g.nnghing stop!
LCM PBA 1-3
Figure 3: Qualitative comparison SIM-DiT-600M against other few-step text-to-image models. Please zoom in to check details, lighting, and aesthetic performances. Prompts in Appendix B.7
crode dhenew rull toce phors Alnen apart apartteee. Bewn clores micup
wor slowing satt and ciy sqaage She
eich other authay dillin aillindd dea caffee.
Figure 4: Visualization of bad generation cases of one-step SIM-DiT model.
ask the studied user to choose their preference according image quality and alignments with the prompts. Figure T shows visualization of our user study cases, in which it is difficult to distinguish the images from PixArt-a and SIM-DIT-600M.
surprising that SIM-DiT-600M achieves almost no performance loss compared teacher diffusion models. For instance, the SAM Caption dataset in Table SIM-DiT-600D recovers 99.6% aesthetic score of PixArt-a model and 100% PickScore. However, the SIM-DiT-600M shows a slightly smaller Image Reward, which can be potentially optimized with more training computes. When compared with leading few-step text-to-image models such SDXL-TTurbo SDXL.-lightning. and Hyper-SDXIL, the SIM-DİT-600M shows dominant aesthetic score with significant margin, together with decent Image Reward and Pick Score.
Almost lossless one-step distillation.
Besides the top performance, the training cost of SIM-DIT-600N is surprisingly cheap. Our best model is trained (data-freely) with 4 A100-80G GPUs for 2 days, while other models in Table require hundreds of A100 GPU days. We summarize the distillation costs in Table marking that SIM is a super efficient distillation method with astonishing scaling ability. We believe such efficiency comes from two properties of SIM. First, the SIM is data-free, making the distillation process not need ground truth image data. Second, the use of the Pseudo-Huber distance function (3.3) adaptively normalizes the loss function, resulting in robustness
pyper--arameters and training stability.
Qualitative comparison. Figure qualitatively compares SIM-DiT-600N against other leading few-step text-to-image generative models. It is obvious that SIM-DiT-60RE generates images with higher aesthetic performances than other models. This reflects the quantitative results Table where the SIM-DiT-600M reaches high aesthetic score. Both the quantitative and qualitative results showease SIM-DiT-600M as the por-perfogning one-step text-to-image generator. Please check supplementary materials for more qualitative evaluations.

---

MODEL
STEPS TYPE
PARAMS
AES SCORE
IMAGE REWARD
PICK SCORE
USER PREF
DISTILL
COST
SDIS-BASE
25
UNET
860 M
5.26
0.18
0.217
SD15-LCM
50
UNET
5.66
-0.37
0.212
A100x DAYS
SDI5-TCD
4
UNET
860 M
5.45
-0.15
0.214
8 A800X 5.8 DAYS
PERFLOW
[S]
4
UNET
860 M
5.64
-0.35
0.208
M GPUX N DAYS
HYPER-SDI5(57
5.79
0.215
A100X
SDXL-BASE 58
25
UNET
2.6
5.54
0.87
0.229
SDXL-LCM
UNET
0.224
A100X 4 DAYS
SDXL-TCD 9
UNET
2.6 B
5.42
0.67
0.226
8 A800 S.8 DAYS
HYPER-SDXI 57
UNET
2.6
5.74
0.93
0.232
32 x N DAYS
SDXL-TURBO 64
UNET
2.6
5.33
0.78
0.228
M GPUx N DAYS
SDXL-LIGHTNING 35
UNET
2.6
5.34
0.54
0.223
64 A100X DAYS
LYYPER SDXL S7
UNET
2.6
5.85
1.19
0.231
32 A100X DAYS
PIXART-a
SIM-DIT-600M
DIT
610 M
6.42
0.67
0.223
A100X 2 DAYS
PIXART-a
7
30
5,93
0.53
0.223
54.88%
SIM-D1I-600M'
DrT
610 M
5.91
0.44
0.223
45.12%
A100x 2 DAYS
Table 3: Quantitative comparisons with frontier text-to-image models on COCO-2017 validation
dataset. The user preference is the winning rate of our user study on SIM-DiT-600M against 20-step PixelArt-a. * means the results evaluated on the SAM-LLaVA-Caption I 1OM dataset, and SIM-DiT- 600M means the SIM generator distilled from PixelArt-a-600M, excluding those in the T5 text encoder. The distillation cost M GPUx N Days means the model did not report the cost.
Failure Cases of One-step SIM-DIT Model. Though the SIM-DiT one-step model shows impres- sive performances, it inevitably has limitations. For instance, we find that the 0.6B SIM-DiT one-step model sometimes fails to generate high-quality tiny human faces and proper human arms and fingers. Besides, the model sometimes generates a wrong number of objects and contents that do not strictly follow the prompts. We believe that scaling up the model size and teacher diffusion models will help to address these issues. Please refer to Figure for visualization of failure cases.
Conclusion and Future Works
This paper presents a novel diffusion distillation method, the score implicit matching (SIM), which enables to transform pre-trained trained multi-step diffusion models into one-step generators a tata-free fashion. The theoretical foundations and practical algorithms introduced in this paper enable more affordable deployment of single-step generators across various domains and applications at scale without compromising the performance of underlying generative models.
Nonetheless, SIM has its limitations that call for further research. First, with the abundance of other powerful pre-trained generative models such as flow-matching models, it is worth exploring to reveal SIM to such a broader family of generative models.
it is possible to generalize the application Second, even though data-free is an important feature of SIM, incorporating new data in the SIM can further boost the quality of generated images failed by the teacher model. This potential benefit has yet to be explored. We hope this could case the training of large generative models.
Acknowledgement
Zhengyang Geng supported by funding from the Bosch Center for AI. Zico Kolter gratefully acknowledges Bosch's funding for the lab.
We would like to acknowledge constructive suggestions from reviewers and ACs/SACs/PCs of NeurlPS 2024. We acknowledge Dr. Mingyuan Zhou for his constructive suggestions the representation of our theoretical results. We also acknowledge the authors Diff- Instrunstttt and Score-identity Distillation for their great contributions to high-quality diffusion distillation Python code. We appreciate the authors of PixelArt-a for making their DiT-based diffusion model public.
10

---

References
[1] Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM Transactions on Graphics (TOG), 42(4):1-11, 2023.
[2] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Mikk Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yo Yu Liu, ediff-i: Text- to- to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211.01324, 2022.
[3] Nicholas M Boffi, Michael S Albergo, and Eric Vanden- Vanden-Eijnden. Flow map matching. arXiv preprint arXiv:2406.07500 07507 2024.
[4] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. International Conference on Learning Representations, 2019, URL
https://openrevii
net stte:/oorumidddBBBsssss
9Fm
[5] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392-18402, 2023.
[6] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue. Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024. URL https: //tp://///p//////// com/ research/ wor 1 mu lators
vi www.:eeenneaatttomomooooeemeemme
[7] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James T. Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. ArXiv, abs/2310.004266. 2023. URL wttp/oorpps..:233333223333
https: semant semanticscholas
[8] Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling, In Advances in Neural Information Processing Systems, pages 1916-99288 2019.
[9] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion- based semantic image editing with mask guidance. ArXiv, abs/2210.11427. 2022.
[10] Wei Deng. Weijian Luo, Yixin Tan, Marin Biloš, Yu Chen, Yuriy Nevmyvaka, and Ricky TQ Chen. Variational schr\," odinger diffusion models. arXiv preprint arX.v:2405.04475, 2024.
[11] Wei Deng, Weijian Luo, Yixin Tan, Marin Biloš, Yu Chen, Yuriy Nevmyvaka, and Ricky TQ Chen. Variational schrödinger diffusion models. Forty-first International Conference on Machine Learning, 2024.
[12] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. arXiv preprini urXiv:2403.03200 2024.
[13] Yasong Feng, Weijian Luo, Yimin Huang, and Tianyu Wang. A lipschitz bandits approach for continuous hyperparameter optimization. arXiv preprint arXiv:2302.01553) 2023.
[14] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint ar Xi0: 221.0..0000... 2022.
[15] Zhengyang Geng, Ashwini Pokle, and J Zico Kolter. One-step diffusion distillation via deep equilibrium models. In Seirty-seventh Conference on Neural Information Processing Systems, 2023. URL http:::t////. //pennre///peeeevv/
rumm? wtt.:rummmmmmmmoo.momo. d=b6xvK2de
[16] Zhengyang Geng, Ashwini Pokle, William Luo, Justin Lin, and Zico Kolter. Consistency models made easy. arXiv preprint 1rXXi:2446.14544 2024.

---

[17] Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, and Josh Susskind. Boot: Data-free distillation of denoising diffusion models with bootstrapping. arXiv preprint 22Xv:22306.05544. 2023.
[18] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.
Prompt-to-prompt image editing with cross attention control. arXiv preprint wrXiv:2208.01666, 2022.
[19] Amir Hertz, Kfir Aberman, and Daniel Cohen-Or. Delta denoising score. arXiv preprint Stti:23304.00990 2023.
[20] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems, pages 6626-6637, 2017.
[21] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6833 2020.
[22] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv preprint arXiv:2204.03453 2022.
[23] Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning, pages 8867-8887. PMLR, 2022.
[24] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data, Advances in Neural Information Processing Systems, 33, 2020.
[25] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110-8119, 81,,, 2020.
[26] Tero Karras, Mika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Proc. NeurIPS, 2022.
[27] Ilya Kavalerov, Wojeiech Czaja, and Rama Chellappa. A multi-class hinge loss for conditional gans. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1290-1299, 2021.
[28] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri. and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/VVE Conference on Computer Vision and Pattern Recognition, pages 6007-6017, 2023.
[29] Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279, 2023.
[30] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3426-2435 2022.
[31] Heeseung Kim, Sungwon Kim, and Sungroh Yoon. Guided-tts: A diffusion model for text- to-speech via classifier guidance. In International Conference on Machine Learning, pages 11119-11133. PMLR, 2022.
[32] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 10215-10224, 2018.
[33] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The CIFAR-10 Dataset. online: http://ww.. cs. toronto. edu/kriz/cfarr html, 55, 2014.
12

---

[34] Chen- Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 300-309, 2023.
[35] Shanchuan Lin, Anran Wang. and Xiao Yang. Sdxl-lightning: Progressive adversarial diffusion distillation. arXiv preprint arXiv:2402.
2024.
[36] Hongjian Liu, Qingsong Xie, Zhijie Deng, Chen Chen, Shixiang Tang, Fueyang Fu, Zheng- jun Zha, and Haonan Lu. Scott: Accelerating diffusion models with stochastic consistency distillation. arXiv preprint arXiv:2403.01 505, 2024.
[37] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003; 2022.
[38] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.0080 00927, 2022.
[39] Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02330 2021.
[40] Simian Luo, Yigin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing nigh-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023.
[41] Weijian Luo. A comprehensive survey on knowledge distillation of diffusion models. arXiv preprint arXiv:2304.00222, 2023.
[42] Weijian Luo and Zhihua Zhang. Data prediction denoising models: The pupil outdoes the master, 2024. URL htt.htts////peeeeiiiiee www.:/wmmeeeeoeumeeeeeee htw.:/uummm..ewwwueuuur.3333
[43] Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL http://ennreeeeeeeeeeeootoo
[44] Weijian Luo, Hao Jiang, Tianyang Hu, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Training energy-based models with diffusion contrastive divergences. arXiv preprint rxXiv:2377.01666 2023.
[45] Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang, Diff- instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[46] Weijian Luo, Boya Zhang, and Zhihua Zhang. Entropy-based training methods for scalable neural implicit samplers. Advances in Neural Information Processing Systems, 36, 2024.
[47] Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073. 2021.
[48] Chenlin Meng, Ruiqi Gao, Diederik P Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. arXiv preprint arXiv:2210.03112, 2022.
[49] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038-6047, 2023.
[50] Thuan Hoang Nguyen and Anh Tran. Swiftbrush: One-step text-to-image diffusion model with variational score distillation. arXiv preprint arXiv:2312.055.:9 2023.
[51] Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion prohabilistic models, arXiv preprint arXiv: 09672, 2021.
13

---

[52] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint (619:0360033999 2016.
[53] William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint 0rX.0222...09748, 2022.
[54] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach, Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
[55] Ashwini Pokle, Zhengyang Geng, and Zico Kolter. Deep equilibrium approaches to diffusion models. Advances in Neural Information Processing Systems, 35:37975-37990, 2022.
[56] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall, Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations, 2022.
[57] Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, and Xuefeng Xiao. Hyper-sd: Trajectory segmented consistency model for efficient image synthesis. arXiv preprint arXiv:2404. I 13686, 2024.
[58] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[59] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22500-22510 10, 2023.
[60] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022.
[61] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022. URL https: / //ppnreview net/ FetoormmiddTIdIiII
[62] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung. Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances neural information processing systems, pages 2234 2242, 2016.
[63] Axel Sauer, Katja Schwarz. and Andreas Geiger. xly:eganxx:: Scaling stylegan to large diverse datasets. ACM SIGGRAPH 2022 Conference Proceedings, 2022.
[64] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint 1rXiv:2311.17000 2.211.1170444 2023.
[65] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[66] Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint arXiv:23 14189, 2023.
[67] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
[68] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.011669 2023.
[69] Yuda Song, Zehao Sun, and Xuanwu Yin. Sdxs: Real-time one-step latent diffusion models with image conditions. arXiv preprint arXiv:2403.16622 2024.
14

---

[70] Pascal Vincent. A Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 23(7):161-1167, 2011.
[71] Yifei Wang, Weimin Bai, Weijian Luo, Wenzheng Chen, and He Sun. Integrating amortized inference with diffusion models for learning clean distribution from corrupted images. arXiv preprint arXiv:2407.11162, 2024.
[72] Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion-gan: Training gans with diffusion. arXiv preprint arXiv:2206.02262. 2022.
[73] Zhendong Wang. Yifan Jiang, Huangjie Zheng, Peihao Wang. Pengcheng He, Zhangyang "At- las" Wang, Weizhu Chen, and Mingyuan Zhou. Patch diffusion: Faster and more data-efficient training of diffusion models. In A. Oh, Naumann, A. Glober- son, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 72137-72154. Curran Associates, Inc., 2023. URL http:ppss///rcceeedddteeec/ ngs neurips cc/ e e1667d60a555447400000 22010/44400112222277///888888
Paper- Conference pdf
[74] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305. 16213, 2023.
[75] Mengfei Xia, Yujun Shen, Ceyuan Yang, Ran Yi, Wenping Wang, and Yong-jin Liu. Smart: Improving gans with score matching regularity. arXiv preprint wrxiv2231..18206 2023.
[76] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. In International Conference on Learning Representations, 2021.
[77] Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingho Hou. Ufogen: You forward once large scale text-to-image generation via diffusion gans. arXiv preprint arXiv:231 2023.
[78] Shuchen Xue, Mingyang Yi, Weijian Luo, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhi-Ming Ma. Sa-solver: Stochastic adams solver for fast sampling of diffusion models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[79] Shuchen Xue, Mingyang Yi, Weijian Luo, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and
Zhi-Ming Ma. SA-solver: Stochastic adams solver for fast sampling of diffusion models. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL https / wwwpnnreeweeweeeefooorrre
9XVEYIO
[80] Zeyue Xue, Guanglu Song. Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diffusion paths. ArXiv, abs/2305. 13055.899. 2023. URL WtRL:ttts::///
semant wwwanticschooll............. htmpaaaaecoccollooooooooooo/oooo 258959002
[81] Hanshu Yan, Xingehao Liu, Jiachun Pan, Jun Hao Liew, Qiang Liu, and Jiashi Feng. Perflow: Piecewise rectified flow as universal plug-and-play accelerator. arXiv preprint 10 2024.
[82] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. arXiv preprint arXiv:2311. 18828, 2023.
[83] Boya Zhang, Weijian Luo, and Zhihua Zhang. Enhancing adversarial robustness via score-based optimization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL http::::://peeeeeee...o
www.oruummm---ooio-------------- MOAHXR
[84] Boya Zhang, Weijian Luo, and Zhihua Zhang. Purify++: Improving diffusion- with advanced diffusion models and control of randomness. arXiv preprint arXiv:2310. 10.1222... 18762, 2023.
[85] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
15

---

[86] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:224..13902 2022.
[87] Yang Zhao, Chunyuan Li, Ping Yu, Jianfeng Gao, and Changyou Chen. Feature quantization improves gan training. arXiv preprint arXiv:2004.02088, 2020.
[88] Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. arXiv preprint arXiv:2211.134422 2022.
[89] Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusion proba- bilistic models and diffusion-based adversarial auto-encoders. arXiv preprint (rxXi:2202.096771 2022.
[90] Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusion probabilistic models and diffusion-based adversarial auto-encoders. In The Eleventh Interna- tional Conference on Learning Representations, 2023. URthttps::/ wpwpreveeeee...ee/ net/
Horum2id-HDxcckk555
[91] Jianbin Zheng. Minghui Hu, Zhongyi Fan, Chaoyue Wang, Changxing Ding, Dacheng Tao, and Tat-Jen Cham. Trajectory consistency distillation. arXiv preprint arXiv:2402.19159 2024.
[92] Mingyuan Zhou, Zhendong Wang. Huangjie Zheng, and Hai Huang. Long and short guidance in score identity distillation for one-step text-to-imaae generation. ArXiv 2406.0/ 56/, 2024. URL
wtt.:ooggg/bb/aogoe 01561
[93] Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. arXiv preprint arXiv:2404.04057 2024.
16

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] is generally preferable to "[No]" it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claim is that we propose method, Score Implicit Matching (SIM), that can distill a pre-trained diffusion model into a one-step generator with very strong performances compared the other SoTA models. We have highlighted important contri- butions of our method in 3.2 and have provided sufficient evidence to support our central claims in the experiments section and 4.2
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
17

---

Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have explcitly stated the limitations of our method in conclusion, and have suggested some potential future work to mitigate these limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Though this paper is more empirically strong paper, also provide clear assumptions or detailed supporting references for the thcoretical statements of the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
18

---

We have included details of experimental settings and hyperparameters in B.2 We also plan release our code ensure transparency and reproducibility of the results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No].
Justification: Since our code has a business policy, we can not release the code at this time. But we plan release the code once the acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp) cc/
pub lice/ii//a quidessiss..eeeeee Cote//ueeeeeeeeeeeeeeeeeee ssiont wolooooolicyy olicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NewrIPS code and data submission guidelines (https?
ssi www.:mmwmmssssoossssoooooo oonooiicy for more details. The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
/nnips cc/ /nnpps.cc//pullic//uuiieeee wwwppplccccleeeccooppopppoe
19

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have included all relevant experimental details, including details of datasets, hyperparameters, optimizer, etc. both in the main paper, as well as in B.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: provide the statistical significance test for the scaling results. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of computing workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
20

---

Justification: We have stated GPU requirements for training our models in We have also included other relevant details of computational requirements in B.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute worker CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httlic///ttt:::s//ttttee httpa//wpp...c/bbbblooccccccccae
Answer: [Yes]
Justification:
We follow the Code of Ethics from all the perspectives stated.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper presents a diffusion distillation approach to strengthen the efficiency of diffusion models. We do not see social impacts within subject.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
21

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes)
Justification: We use the pre-trained PixelArt-a sext-to-image model as teacher distillation model. along with its open-soureed training datasets. We have described the details in section 4.2 Since currently we don't plan to open source our text-to-image model, so there is concern for to release unsafe contents.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Major assets in our work are:
Datasets CIFAR- 10 and SAM-recaptioned: we have cited the original paper that proposed these datasets.
Models PixelArt-a: we have cited the original paper that proposed these datasets.
Our codebase is adapted by modifying code available on github by the authors of EDM 266 and Diff-Instruct [43]. The relevant license Atrrbution-NonComneriiaa Ahake 4.0 International. Our code includes and correctly attributes the relevant copyright information.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided.
popular datasets, ppapersw's thoode. com / dat a asets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
22

---

Justification:
We do not release new assets in the submission phase.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We have one experiment including human preference studies. We describe the detailed instructions for conducting such a user study in section B.4
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: We have one experiment including human preference studies. We have acquired the approval of users under study help fill our forms. No potential threats to human subjects were detected, and all results were anonymized to prevent any potential exposure of human identities. All human subjects were properly paid. B.4
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23

---

A Theory Parts
A.1 Proof of Theorem
The proof Theorem 3.1 is based on the so-called Score-projection identity which was first found in Vincent [70] to bridge denoising score matching and denoising auto-encoders. Later the identity applied by Zhou al. [93] for deriving distillation methods based on Fisher divergences. We appreciate the efforts of Zhou et al. [93] and re re-write the score score- projection identity here without proof. Readers can check Zhou et al. for complete proof of score-projection identity.
Theorem A.I (Score-projection identity). Let ul. 0) be a vector-valued function, using the notations of Theorem 3.1 under mild conditions, the identity holds:
2 (1.2, 0)7 Spo, (x:)) Vx. log (x) 0, AO.
* (o (mg)
Next, we turn to prove the Theorem
Proof. We prove a more general result. Let u-) be a vector-valued function, the so-called score-
projection identity 93 70 holds,
log qe (2x, } } 0, VO.
E
7P000 dtg mige City (mn)
ux 0)7
(pe)((0s) Vx
(A.1)
Taking 0 gradient on both sides of identity
A.I we have
}
Sxc
0 = E
b0g (mclm)
#0-30,0
dx
u (xt () (xx)) Va, log qe (xxe) (3z/|2200
00
axc
E
u u(xi, 0)
me log ae (ave [
(A.2)
my (a (mg) dan
00
u(x;
0)T
&pu,e (
+ og
ular, 0) (00ssoo)) (a:e)
(A.3)
00
E
u(x, 0)²
8po. (are)
(A.4)
log (m
00
0x.
D E
u1 0) (3p),((xt)) - Vmn, log 2e (mx [m))
(A.5)
(e lmg)
00
0xo
u(xi ()²{
Va, log (20)
(A.6)
Exo
00
+
u uL, 0)T ss((zz))
(A.7)
00
Ex Pp. u u(x,, 0)T
00 8po. (2x)
(A.8)
+
PPoPP (mggmg)
u(x,, 0)T
(0p)) Va, mx log de (2ll
(A.9)
30
Therefore we have the following identity:
Exropo, u(x, 8)" 00 spo.. (zxi)
② {
E
u(z,, 8)7
(xx)) - Va, log ge (2) (zo)
00 (p (mg)
(A.10)
which holds for arbitrary function ul: 0) and parameter O. If we set
u (x 0) d' d'(y))
y = Spsst0), (Ixz) - 8g. (xx.)
24

---

Then we formally have
{ Spo.. (20o)
d' d((sr))
E
00
Pecloli
d' d((we))
E
mg log (x
spo,, -
Vx, log (o
(A.11)
00
A.2 Pytorch style pseudo-code of Score Implicit Matching
In this section, we give a Py Torch style pseudo-code for algorithm | with the Pseudo-Huber distance function. For a detailed algorithm on CIFARIO with EDM model, please check Algorithm 2
import torch
import torch. nn as mn import torch.optim as optim
Initialize generator
G Generator
# load teacher DM
Sd DisfusicnModel Load tt__cttttt_ckpt evall () requires__grad (False)
10 5g copy. copy.deepcopy (Sd) # initialize online DM with teacher DM
11
12 Define opt imizers
Aeda opt optim. Adam paramramtt 1r=0.001, betas= (0.0, 0.999)]
14 opt Sg opt im, Adam (Sg. parameters() 1r=0. 001, betas= (0.0, 0.999))
15
16 Training leop
while Truc:
18
## update Sg
19
Sg.train requirrererrrrrrreeeeeere (Tarr (True)
20
G.eval () reepiirss_grad_Pallll
Loop for times to update Sg for range
23
24
z torch. randn (2000, 2) .o (device)
25
with torch.no_grad
fake_x (2)
27
28
torch. from humpyyyppyyaaammm randem. choice (np.arange Sd. size= shke_exshapplle replace=True) to (device) .1ong() fake fake_xt, xt, t, noise, sigma_t, t, g2_t = Sd (fake_x,, t=t, return_t=True
sigma_t sigma_t.view (-1,1) (- (device)
31
g2_t (dvvic))
(device)
32
score = (torch..cat ( ([fake_ xt. t.w.t /dd. -1) /sigmaat
33
batch_sg_lan score + ncise/sigma_t
batch_sg_ (g2_t=batch_sgg los.
(-1)) .mean xSd.7
36
37
optimizer_Sg.cc zead_graadd ()
38
batch_sg__oon loss. backward ()
optimizer_Sg.
40
41
42
# update G
Sg.eval () requires seaerirrr_rrd____e G.train() .requires_graa. (True)
45
16
2 torch. ( (2000, 2) ) to (device)
47
fake_x ((2)
25

---

t = torch. numpy (np. randem. choice (np. arange (1, diffusion. size= fake x.shape [0], replace=True) to (device) .1ong))
50
fake xt, t, noise, sigma_t, g2 t diffusion (fake_x, x, t=t, return t= Truc)
51
sigma_t = sigma_t.view (-1, .t (device)
g2_t to
54
score_tree true sá (torchh. cat ( (-1, od -1))
sigma_t
55
score_iat fake (torchh. cat ([take_xt, wtw.::tttttt..wttti__ee /ttll//difffuioooo -1))
sigma_t
56
57
score_diff score_true sccre_fake
58
offset_cceff cenoise_diff / sqrr (ddnoossedf square sum
(11,2,3), keepdims-True) self. =elf. p weight 1.0
60
61
62
batch_g_loss = weight offsct_cooff (fake_denolse images) latch_g_loss batch_g sum
(ml.mmaaattmi............. [1, 3]) meaan
64
65
aeriiorr_..ero_raadd s__loss.backwarr ()
66
opt aptimizer_G.sepp
Listing 1: Pytorch Style Pseudo-code of SIM
A.3 Instances SIM with different distance functions
In section 3.3 have discussed the powered normed distance functions. Other choices, such as the Huber distance, which is defined as
//2/2 [8(uua) 6/2)
for ya 2 8 otherwise
vI v D, Ls(y)a
For other choices of distance functions, such norm and exponential with powered norms, we put them in Table
Table 4: Instances of Score Implicit Matching loss with different distance functions. The notations aligned with the Algorithm
CHOICE d(-)
d'(y:)
FUNCTION
1lax
28c
2n
N (1. Ve, log a (mz (mma)
Iau 10 421. Koam
(e-1)
au(me, r) Vay lope or (oea (mea
( 121, even
exp(() ye (lll))l( (--1)
exp (.91 v1llv111?> (12)
vi (o-1) (ate, V. log de (we, (wa))
liaclla
sign(y:)
siga(
au (301, () Va, log ou (20, (so))
Lilut))
Lo(vs)
Lo(u))
(10g, 1) Va, log (ang (an, (mm))
Lg(-) Hearn
11 -
2/2 13 13+03
B (so, 1) Vor log go tet (20)
y +i
B Empirical Parts
B.1 Answer for the human preference study
The answer to the human preference study in Figure is
the middle image of the first row is generated by one-step SIM-DiT 1MM-DIT-00M::
26

---

the leftmost image of the second row is generated by one step SIM-DiT-600M;
the leftmost image of the third row is generated by one-step SIM-DIT-600M.
B.2 Experiment details on CIFAR10 dataset
We follow the experiment setting of SiD and DI on CIFAR1O. We start with brief introduction to the EDM model [26].
The EDM model depends on the diffusion process
dæ Idwr, [0,
(B.1)
Samples from the forward process can be generated by adding random noise to the output of the generator function, i.e., 2t 2n le where (NO, I) is Gaussian vector. The EDM model also reformulates the diffusion model's score matching objective as a denoising regression objective, which writes,
((6))
An pmo-pp (mz ( [mm) ( -==
xo [2dt.
(B.2)
Where (.).)) is a denoiser network that tries to predict the clean sample by taking noisy samples inputs. Minimizing the loss (B.2) leads to trained denoiser, which has simple relation to the marginal score functions as:
(x 1)
(w t)
(B.3)
t²
Under such formulation, we actually have pre-trained denoiser models for experiments. Therefore, we use the EDM notations later parts.
Construction of the one-step generator. Let d- be pretrained EDM denoiser models. Owing to the denoiser formulation of the EDM model, construct the generator to have the architecture as the pre-trained EDM denoiser with pre-selected index t", which writes
3D go (z) d(z,t" t)), (O, (I")
(B.4)
We initialize the generator with the same parameter as the teacher EDM denoiser model.
Time index distribution. When training both the EDM diffusion model and the generator, need to randomly select time t in order to approximate the integral of the loss function (B.2). The EDM model has default choice of distribution log-normal when training the diffusion (denoiser) model, i.e.
t ~ (t) t= exp(s)
(B.5)
((Pmmans P2t)), Pmean -12,, Pard 1.2.
(B.6)
And weighting function
(12 oZataa)
AEDM(() (t x dddata)
(B.7)
In our algorithm, we follow the same setting as the EDM model when updating the online diffusion (denoiser) model.
In SiD, they propose to use a special discrete time distribution, which writes
(amas K - (o min
o kmaz ))P,
Fmar 80.0, Jmin 0.002, p= 7.0, K 1000
They proposed to choose t uniformly from
L~ I>ps((() & (0, (- dj:
(B.8)
We name such a time distribution the Karr distribution in Figure because such schedule was originally proposed in Karras' EDM work for sampling.
27

---

Table Hyperparameters used for SIM on CIFARIO EDM Distillation
Hyperparameter
CIFAR-TO (Unnoo))
CIFAR-TO(Coni)))
DM 8
Generator Gi
DM 8
Generator Oe
Learning rate
le-5
le-5
le-5
le-5
Batch size
256
256
256
256
att))
2.5
2.5
2.5
Adom 8o
0.0
0.0
0.0
0.0
Adam 81
0.999
0.999
0,999
0,999
Time Distribution
PEDM(() B.5
PSIM (f) B.9
(E (B.5
Psras (f) B.99
Weighting
Agnu (t) B.7
AgDar (t)
B.7
Loss function
B.
B.I
B2
(B.13
Number of GPUs
4XAT00-40C
4x AT00-400
4XAT00-400
A MWAT000-4GG
However, in practice, we find that Karr distribution (.S)) empirically does not work well. Instead, we find that a modified log normnllaal time distribution when updating the generation with SIM works better than Karr distribution. Our SIM time distribution writes:
t ~ PSIM t cxp(ss
(B.9)
Pmean 335, Patd 2.5.
(B.
Weighting function. As we have said, we use the same (EDM(t) (B.7) weighting function as EDM when updating the denoiser model. When updating the generator, SiD uses specially designed weighting function, which writes:
C x 1s
W WS:D(t) = ||:0 dg. (xt))l1s8g
(B.11)
23 = 3o + le, 6 6~ ((.,T)
(B.12)
The notation sg means stop-gradient, and C is the data dimensions. They claim such weighting function helps to stabilize the training. However, in our experiments, since the SIM itself has normalized the loss (see section 4). we do not use such ad-hoc weighting functions. Instead, we just set the weighting function to be I for all time. We call the SiD's weighting function the sidwgt in Figure 2 and our weighting the nowgt F
In Figure 2, we compare the SiD and SIM with different time distribution and weighting functions. We find that SIM=nowgt+lognommal time distribution gives the best performances significantly, therefore our final experiment tasks such a configuration. 55 the detailed configurations we use for SIM CIFARIO EDM distillation.
With the optimal setting and EDM formulation, we can rewrite our algorithm in an EDM style in Algorithm 2
B.3 Experiment details on Text-to-lmage Distillation
In the Text-to-Image distillation part, in order to align our experiment with that CIFARIO, rewrite the PixArt-a model in EDM formulation:
(x=tt) = x -
(B.14)
Here, following the IDDPM+DDIM preconditioning in EDM, PixArt-a is denoted by Fe. is the image data plus noise with a standard deviation of t, for the remaining parameters such as C₁ and C2. we kept them unchanged to match those defined in EDM. Unlike the original model, we only retained the image channels for the output of this model. Since we employed the preconditioning of iDDPM+DDIM in the EDM, each value is rounded to the nearest 1000 bins after being passed into the model. For the actual values used in PixArt-o, beta_star start set to 0.0001, and beta_ snndeeddll is set to 0.02. Therefore, according the formulation of EDM, the range of our noise distribution is (0.01, 156.6155). which will be used to truncate our sampled f. For our one-step generator, it formulated as:
(x(z)= = =
(B.15)
Here following SiD 1" 2.5 and ~ (0,, (1)) we observed in practice that larger values of t* lead to faster convergence of the model, but the difference convergence speed negligible for the complete model training process and has minimal impact the final results.
28

---

Algorithm 2: SIM with Pseudo-Huber distance for distilling EDM teacher [Pytorch Style].
Input: pre-trained EDDM denoiser d, (.), generator go. prior distribution Pz, online EDM denoiser (.); differentiable distance function d(.), and forward diffusion 2.1
while not converge do
// freeze 0, update v.:
2o (d)tach),,,,) z~ P=
8 ~ (t) 2t = 2o + te, ~ 1,
[(v) (BDM()) [1d,
zoll
C bockward(); update v
1/ freeze v. update 8:
Zo = g0(z), 2~ Pz
1 PSIM(()) at Do + le, e <~ I)(,)
y (2 1) wa mo)
[(0)
where y/: == d, 1) du (3rs)
(B.13)
<(0).backward(;
update
end
return 0,
We utilized the SAM-LLaVA-Captionn Cop dataset, which comprises prompts generated by the LLaVA model on the SAM dataset. These prompts provide detailed descriptions for the images, thereby offering us a challenging set of samples for our distillation experiments.
All experiments in this section were conducted on 4 A100-40G GPUs with 6 precision, using the PixAnt-XL-2-51 12x5 12 model version, employing the same hyperparameters. For both optimizers, we utilized Adam with learning rate of 5c-6 and betas=[O, 0.999]. Additionally, to enable a batch size of 1024, we employed gradient checkpointing and set the gradient accumulation to 8. Finally, regarding the training noise distribution, instead of adhering to the original iDDPM schedule, we sample the a from log-normal distribution with a mean of -2.0 and standard deviation of 2.0, we use the same noise distribution for both optimization steps and set the two loss weighting to constant 1. Our best model was trained on the SAM Caption dataset for approximately 16k iterations, which is equivalent to less than epochs. This training process took about 2 days on 4 A A100-40G GPUs.
We also tested the impact of different noise distributions on the distillation process. When the noise distribution highly concentrated around smaller values, we observed a phenomenon where the generated samples appear excessively dark. On the other hand, when we used slightly larger noise distributions, we found that the structure of the generated samples tended to be unstable.
B.4 Instruction for Human Preference Study
Our user study primarily focuses on comparing the outputs of the distilled model and the teacher model. Each image has undergone rigorous manual review to ensure the safety of survey participants. We conducted the study using questionnaires, where users were presented with two randomly ordered images generated by the distilled model and teacher model and asked to select the sample that best matched the text description and had higher image quality. Finally, we used the collected votes for the distilled model and the teacher model as indicators of user preference. The questionnaire website used for conducting these evaluations are shown in Figure
To be more specific, we randomly selected 17 prompt words and generated images of resolution 512x512 using both the student model and the teacher model. To facilitate comparison, we presented the two images side by side in random order. In the questionnaire, we provided the complete prompt words for reference in addition the generated images. In the end, we collected approximately 30 survey responses in total.
29

---

AIGC Texl-to-lmage User Study
Bhem fell wing peanpys went beat by out oi dlarty and rudly (17 wutkeage palk
Figure Demonstration of our human preference user study interface.
Figure 6: One-step SIM model on CIFARI0--onniiionn FID=1.96.
30

---

Figure 7: One-step SIM model on CIEARR1--unooddtiionl FID-2.06.
Ma
Figure 8: The comparison of FID convergence between SIM and SiD.
B.5
Generated Samples on CIFAR10
B.6
FID Convergence on CIFAR10 Unconditional Generation
B.7
Prompts for Figure 3
prompt for first row of Figure 3 A small cactus with happy face in the Sahara desert.
prompt for second row of Figure An image of jade green and gold coloured Fabergé egg, 16k resolution, highly detailed, product photography, trending artstation, sharp
31

---

focus, studio photo, intricate details, fairly dark background. perfect lighting, perfect com- position, sharp features, Miki Asai Macro photographty, close-up, hyper detailed, trending aristation, sharp focus, studio photo, intricate details, highly detailed, by greg rutkowski. prompt for third row of Figure 3
Baby playing with toys in the snow.
32


Reviews:
[{"review_id": "awO0Si2XqC", "reviewer": "Reviewer_G1sA", "summary": "The paper ‘One-Step Diffusion Distillation through Score Implicit Matching’ introduces a novel framework Score Implicit Matching (SIM), to distill pre-trained diffusion models into single-step generator models. This approach achieves almost no performance loss compared to the teacher diffusion model while being data-free i.e. this approach does not require ground truth data. \n\n**Key Insights**\n- The paper proposes a flexible class of score-based divergences between the generator model (student) and the diffusion model (teacher). - Though such divergences cannot be computed explicitly, the gradients of these divergences can be computed via the score-gradient theorem, allowing for efficient training.\n- The Pseudo-Huber distance function in SIM results in stronger robustness to hyper-parameters such as learning rate and faster convergence.\n- The paper provides a detailed algorithm for implementing SIM, which involves alternating phases of learning the marginal score function and updating the generator model.\n\n**Empirical Performance**\n- On CIFAR10, SIM achieves an FID of 2.17 for unconditional generation and 1.96 for class-conditional generation.\n- For text-to-image generation, a single-step generator distilled using SIM from a leading diffusion-transformer-based model attains an aesthetic score of 6.42, outperforming other one-step generators like SDXL-TURBO, SDXL-LIGHTNING, and HYPER-SDXL.\n\n**Comparative Analysis**\n- SIM is compared to other methods like Diff-Instruct (DI) and Score Identity Distillation (SiD), demonstrating better performance in terms of robustness, convergence speed, and generative quality.\n- The empirical results highlight SIM’s superiority, particularly in maintaining the performance of the original multi-step models in a one-step generation framework.\n\n**Practical Applications**\n- The data-free nature and robust convergence make SIM a highly efficient method for distilling diffusion models.\n- The method shows strong potential for scaling to more complex tasks and larger neural networks.\n\n**Limitations and Future Work**\n- The applicability of SIM to other generative models, such as flow-matching models, is yet to be unexplored.\n- While SIM is data-free, incorporating new data might further enhance performance, which is a potential area for future research.\n\nOverall, the paper presents a significant advancement in the field of diffusion model distillation, offering a practical and efficient method for transforming pre-trained multi-step diffusion models into one-step generators without compromising performance.", "questions": "Some of these are already mentioned under weakness. \n- Evaluating the performance of SIM on different image domains would broaden the applicability of this approach\n- Discussing scenarios where SIM would not perform well would help in understanding the boundaries and limitations for this approach and guide future improvements.\n\nAdditionally, it would be helpful if the authors can provide mode implementation details such as hyper-parameter tuning strategies and potential pitfalls during the training process. \n\nMisc\n- In the appendix, it is mentioned that a human preference study was conducted and the authors collected 30 responses in total. It would be valuable to know the results from the human preference study.\n- Grammar/Typos\n   - Line 130 - ‘... if we choose the sampling distribution to be the diffused ..’\n   - Line 220 appears to be grammatically incorrect - ‘It is on par with the CTM and the SiD’s official implementation has yet to be released’", "limitations": "The authors have acknowledged limitations of the proposed approach. \n\n- The applicability of SIM has not been explored in the space of other generative models such as flow-matching models.\n- Although the approach is data-free, the potential benefit of incorporating data is yet to be explored. \n\nI do not see potential negative societal impacts of this work.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "**Originality**\n- The paper presents a novel approach - Score Implicit Matching (SIM) to distill pre-trained diffusion models into one-step generator models. This approach stands out by being data-free, which is a significant deviation from conventional distillation methods that often require extensive training data.\n- The paper’s key technical insight, the score-gradient theorem, allows computation of gradients for score-based divergences, enabling efficient training. This is a novel contribution that enhances the feasibility of implicit minimization of these divergences.\n- Pseudo-Huber Distance Function normalizes the distance vector y_t, which in turn stabilizes training loss and results in robustness to hyper-params and faster convergence.\n\n**Quality**\n- The paper provides compelling empirical evidence of SIM's effectiveness on CIFAR10 image generation. The approach also outperforms other one-step generators on aesthetic score on COCO-2017 validation set. \n- The authors conduct thorough comparisons with existing methods, such as Diff-Instruct (DI) and Score Identity Distillation (SiD), showing that SIM not only outperforms these methods but also converges faster and is more robust to hyper-parameters.\n\n**Significance**\n- SIM makes a significant contribution to the practical deployment of diffusion models in real-world applications, particularly where computational efficiency is critical.\n- The robustness and fast convergence of SIM suggest that it can scale to more complex tasks and larger neural networks, making it a valuable tool for future research and application in various domains, including image and video generation.\n- The potential of SIM to generalize to other generative models, such as flow-matching models, opens up new avenues for research and application, further enhancing its significance in the field of generative modeling.\n\n**Clarity**\n- The paper is well-organized, with a logical flow from problem formulation to the introduction of the SIM method, followed by empirical evaluations and discussions of results.\n- Key concepts, such as the score-gradient theorem and the Pseudo-Huber distance function, are explained in detail for reproducibility. \nThe tables and figures that present empirical results and algorithmic steps help in better understanding the performance and implementation details of SIM.\n\n\nThe paper presents a substantial advancement in the field of diffusion model distillation, offering a robust and efficient method for converting multi-step diffusion models into one-step generators without compromising performance. The combination of theoretical innovation with strong empirical results underscores the potential of SIM for widespread application and further research in generative modeling.\nOverall, \"One-Step Diffusion Distillation through Score Implicit Matching\" is a well-executed study that addresses a key challenge in diffusion models and provides a promising solution with broad applicability.", "weaknesses": "- **Narrow Scope of Comparisons**: The empirical comparisons focus primarily on Diff-Instruct (DI) and Score Identity Distillation (SiD). While these are relevant baselines, doing a broader comparison with other state-of-the-art distillation and generative modeling techniques would provide a more comprehensive evaluation of SIM's relative performance and applicability.\n- **Dataset Limitation**: The experiments are mainly conducted on the CIFAR-10, SAM-LLaVA-Caption10M and COCO-2017 validation dataset. Including results on more diverse datasets, such as higher resolution images, different image domains (e.g., medical imaging, satellite imagery), and more complex text-to-image datasets, would strengthen the claim of SIM's broad applicability.\n- **Addressing Failure Modes**: The paper does not thoroughly discuss the potential failure modes or limitations of the SIM method. Identifying scenarios where SIM might underperform, and providing insights or hypotheses on why these failures might occur, would be valuable for understanding the boundaries of the method’s applicability and for guiding future improvements.\n- **Exploration of Data Incorporation**: While the data-free nature of SIM is highlighted as a strength, the potential benefits of incorporating new data during distillation are mentioned but not explored. Providing preliminary experiments or a more detailed discussion on how incorporating data might enhance the quality of the distilled models would be a constructive addition.\n\nThe paper presents significant contributions to the field of diffusion models. Addressing the above weaknesses would further enhance its impact and applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that you like our work. We appreciate your valuable suggestions, and we will take them in the revision. In the following paragraphs, we will address your concerns one by one.\n\n**Q1**. Doing a broader comparison with other distillation and generative modeling techniques would provide a more comprehensive evaluation of SIM's relative performance and applicability.\n\n**A1**. In this paper, we have compared SIM with Diff-Instruct and SiD, showing its soundness on both the theoretical and empirical side. Another leading diffusion distillation method might be the consistency trajectory model (CTM) [1], which uses self-consistency to distill the model. Though CTM has shown strong performances, its performances rely on using additional LPIPS losses and GAN losses, which need ground truth data and are also tricky. Besides, the CTM model requires running multiple steps of reversed diffusion sampling, which is more computationally inefficient than SIM without running multiple-step sampling with teacher diffusion. We will compare with other leading distillation methods in the revision. \n\n**Q2**. Including results on more diverse datasets, such as medical imaging, and satellite imagery would strengthen the claim of SIM's broad applicability.  \n\n**A2**. We highly appreciate your suggestions to use SIM on broader applications. In the rebuttal period, we conduct two more experiments: (1) text-to-3D generation using text-to-2D diffusion; and (2) improving the GAN generator using diffusion models; **Please check our Global rebuttal cell for details.** On both new applications, SIM has shown strong performances, showing its broad applicability. \n\n**Q3**. Providing preliminary experiments or a more detailed discussion on how incorporating data might enhance the quality of the distilled models would be a constructive addition. \n\n**A3**. We appreciate your keen intuition. One advantage of SIM is the image-data-free property. However, we agree that incorporating training data with SIM will potentially strengthen the SIM in real-world applications. A possible way to incorporate data is to use additional GAN discriminator and GAN losses together with the SIM loss. This can inject knowledge from new data into the generator model. Another direction that is worth exploring is to extend the one-step generator to multi-step generative models using variants of SIM. This potentially will combine techniques with consistency-based models. However, both the GAN loss and the multi-step generalization are more like a combination of different helpful techniques in generative modeling. In this submission, our goal is to thoroughly study the theory and practical performance of SIM. We will leave the study that incorporates data with SIM in the future.\n\n**Q4**. Discussing scenarios where SIM would not perform well would help in understanding the boundaries and limitations of this approach and guide future improvements. \n\n**A4**. We appreciate your constructive suggestion. In our text-to-image experiment, we find our SIM-DiT-600M sometimes fails to generate high-quality tiny human faces, hands, as well as legs. However, we find that the teacher PixelArt-$\\alpha$ model also has such issues. We believe that a stronger teacher diffusion model will possibly address such failure cases. Please check **Figure 2** in our **added one-page** in global rebuttal for some failure cases. \n\n**Q5**. The human preference study.\n\n**A5**. We feel sorry for the confusion, we put the human preference comparison of SIM-DiT-600M and PixelArt-$\\alpha$ diffusion model in **Table 3** in our submission. The result shows that the teacher PixelArt-$\\alpha$ model slightly outperforms the SIM one-step model with a winning rate of 54.88\\%. We will polish the presentation in the revision. \n\n**Q6**. Implementation details. \n\n**A6**. We will release the code if the paper gets published. In this response, we add some more implementation details for the one-step text-to-image generator model. When distilling the generator, we first reformulate the PixelArt-$\\alpha$ diffusion with the so-called \"data-prediction\" formulation as proposed in the EDM paper [2]. We then initialize our generator using the \"data-prediction\" model with a fixed noise sigma of 2.5, following the same setting of Diff-Instruct and SiD. When distillation, we use a fixed classifier-free guidance scale of 4.5 for teacher diffusion. For the auxiliary diffusion model, we do not use the CFG. This setting is inspired by the DMD paper [3]. We will add more implementation details in the Appendix in the revision. \n\n**Q7**. Some typos.\n\n**A7**. We will address the typos and improve our writing in the revision. \n\n[1] Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion\n\n[2] Elucidating the Design Space of Diffusion-Based Generative Models\n\n[3] One-step Diffusion with Distribution Matching Distillation\n\n**We hope our response has resolved your concerns. If you still have any concerns, please let us know.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VUsAWnNNhm", "reviewer": "Reviewer_pChh", "summary": "This paper proposes a new distribution matching loss between the one-step generator and the pre-trained diffusion model. The reverse KL divergence proposed in Diff-Instruct is generalized through the Score-divergence gradient Theorem.", "questions": "1. Is the score-divergence theorem mentioned in the diffusion model community? It seems applicable to diffusion model training.\n\n2. Will you release the code?", "limitations": "None.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The Score-divergence theorem is well adapted to generalize Diff-Inst.\n\n2. The empirical results with generalized objective function show better with expanded hyperparameter space.\n\n3. A method like CTM utilizes GAN loss (w/ true data) to boost the performance. Besides the CTM, this paper's performance seems good.\n\n4. The distillation training cost seems cheap.", "weaknesses": "1. More analysis on $\\alpha$ seems required.\n\n2. The score network for the student model is still required. I think the existence of an auxiliary score network is expensive. For example, a recent paper in [1] does not utilize an auxiliary score network. \n\n3. Adding recall metric in cifar-10 experiments if you want to claim Diff-Instruct suffers from mode-collapse. \n\n4. Since Diff-Instruct uses reverse-KL divergence, their mode collapse is intuitive for me. Why does the objective function you propose not to suffer from mode collapse? I know why forward KL divergence can cover the mode and reverse KL can not do it. Please explain it similar sense. There may be a difference depending on the $\\alpha$.\n\n[1] Multistep Distillation of Diffusion Models via moment-matching", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We will address your concerns one by one. Before that, we first give a summary of the main contributions of our work.\n\nIn this work, we introduce score implicit matching (SIM), a novel diffusion distillation algorithm that achieves **competitive performances on CIFAR10 generation** tasks, and **lossless aesthetic performances** on the one-step distillation of the **DiT-based text-to-image diffusion model**. Besides the strong empirical performances, the theoretical foundation of SIM is also sound: **we prove that the SIM loss shares the same $\\theta$ gradient as our proposed score-based divergence**. Therefore using an SGD-type optimization algorithm to minimize SIM loss is secretly equivalent to minimizing the intractable score-based divergence. Besides, our proposed score-gradient theorem may bring **new theoretical tools for future studies on generative models**. \n\nNext, we address your concerns one by one.\n\n**Q1**. More analysis on $\\alpha$.\n\n**A1**. We are not sure what you mean by $\\alpha$. In the SIM algorithm (Algorithm 1), we do not have an $\\alpha$ hyperparameter. We guess that you mentioned $\\alpha$ to mean the SiD algorithm, which uses an empirical \n$\\alpha$ hyperparameter to fuse its loss functions. SIM is different from SiD in theory. As we have shown in Sections 3.3 and 3.4, when taking the distance function to be squared L2 distance, SiD's empirical fused loss with an $\\alpha=1.0$ is a special case of SIM. Besides, SIM supports a flexible choice of distance functions to define and instantiate different score-based loss functions. This distinguishes SIM from previous methods such as SiD and Diff-Instruct. \n\n**Q2**. The existence of an auxiliary score network is expensive. A recent paper in [1] does not utilize an auxiliary score network. \n\n**A2.** We appreciate your keen intuition. We acknowledge that SIM needs an additional online diffusion model, which brings more memory costs. However, in practice, we find that the **additional cost is acceptable** if we use Pytorch techniques. For example, in our text-to-image experiment, we use 4 Nvidia A100-80G GPUs for training the one-step generator. We use the Pytorch Distributed data-parallel (DDP) to support multi-GPU gradient syncing and use the BF16 numerical format for training. We can effectively train the model with a total batch size of 256 even with no gradient accumulation. If we use gradient accumulation techniques, we can enlarge the batch size to 1024 with some speed decline. \n\nHowever, **we highly appreciate your suggestions to refer to [1] to figure out the possibility of getting rid of the online diffusion model**. After a careful read of [1], we like the idea of moment matching to drop the online diffusion model. We will involve a discussion on SIM and [1] in the revision and leave the study that drops the online diffusion while maintaining strong one-step generation performance in our future work.  \n\n**Q3**. Why does SIM not suffer from mode collapse? \n\n**A3**. We appreciate your good question! SIM is essentially different from the Diff-Instruct algorithm in the divergence that the generator aims to minimize. If we use $p_d$ to represent data distribution and $p_g$ the generator distribution, the Diff-Instruct minimizes the integral of the **KL divergence** between the generator and the teacher diffusion model with a form $$\\mathcal{D}\\_{KL} (p\\_g, p\\_d) = \\mathbb{E}\\_{x\\sim p\\_g}\\log \\frac{p\\_g(x)}{p\\_d(x)},$$ while the SIM minimizes the score-based divergence with a form $$\\mathcal{D}\\_{SD}(p\\_g,p\\_d) = \\mathbb{E}\\_{x\\sim \\pi} d(\\nabla\\_x \\log p\\_g(x) - \\nabla\\_x \\log p\\_d(x)).$$ KL divergence is notorious for mode-collapse issues because the likelihood ratio $\\frac{p_g(x)}{p_d(x)}$ will be ill-defined if $p_g$ and $p_d$ have misaligned density support. However, the score-based divergence does not have such a \"ratio\", and therefore is safe in the case when $p_g$ and $p_d$ have misaligned support. Besides, our SIM with Phuber distance (equation 3.8)  has a self-normalization form, which helps stabilize the training loss to a constant scale which potentially addresses the mode-collapse issue. \n\n**Q4**. Is the score-divergence theorem mentioned in the diffusion model community? \n\n**A4**. As we introduced in Section 3.2, the Score-divergence gradient Theorem is proved by taking the parameter gradient to both sides of the so-called score-projection identity. The score-projection identity is well-known for proving the equivalence of denoising score matching and classical score matching in [2]. Recently, [3] also used the projection identity to prove value equality when deriving SiD loss. \n\nHowever, as far as we know, our score-divergence gradient theorem is the first time it appeared in the community, marking it as an important theoretical contribution in our paper. In this paper, we propose and use the score-divergence gradient theorem to prove the gradient equality of SIM loss and intractable score-based divergence instead of value equality. This distinguishes SIM from previous methods such as SiD that only prove value equality that is insufficient in theory. \n\n**Q5**. Will you release the code? \n\n**A5**. We will release the code if the paper gets published. \n\n**Q6**. Add a recall metric to show that Diff-Instruct suffers from mode-collapse. \n\n**Q6**. Thank you for your suggestion. In the rebuttal period, we compute the precision and recall of the DI model. It has a recall of 0.52 which is pretty low, indicating its mode-collapse issue. \n\n[1] Multistep Distillation of Diffusion Models via moment-matching\n\n[2] A Connection Between Score Matching and Denoising Autoencoders\n\n[3] Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation\n\n**We hope our answers have resolved your concerns, and if you still have any concerns, please do let us know.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4KDWkkP50Q", "reviewer": "Reviewer_wPU2", "summary": "This work proposed Score Implicit Matching (SIM) to distill diffusion models into a one-step generator. The core idea is to use the “score-gradient theorem” to transform the minimization of score-based divergences between generator and real score functions into an implicit and tractable minimization problem. The theory does not specify the underlying divergence or distance metric for score functions, and seems to generalize many distribution matching methods, such as SiD and DI. Experiments were conducted on CiFAR-10 and text-to-image generation to show the effectiveness and robustness of SIM, by distilling EDM and PixArt-$\\alpha$, respectively.", "questions": "Please see my questions in the Weaknesses section.", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Overall, the paper is well written and easy to read, although there is still a need to further polish the presentation (as I pointed in the Weaknesses section).\n- This work provides a good theoretical insight into how to transform the intractable score-based divergence into a tractable implicit minimization problem, where both share the same gradient. \n- It generalizes many previous works (SiD, DI) as its specific cases.\n- Experiments on CIFAR-10 and text-to-image generation showed convincing results to demonstrate the effectiveness of SIM.", "weaknesses": "- The reason why Pseudo-Huber distance is better than other choices (L2 Square in SiD or KL in DMD) is not quite clear to me. Can the authors provide some empirical evidence to support the hypothesis of the normalization effect? For example, we can use L2 distance (or setting c=0), does the method also work well as it also normalizes the difference between generator score and teacher score? Also, what $c$ value works the best for the Pseudo-Huber distance?\n- Can the authors show how SIM becomes a DMD objective if we set $d(\\cdot)$ to a reserve KL as in DMD? It is not quite clear to me how it works.\n- There are some other standard benchmarks for one-step distillation, including distilling EDM trained on ImageNet-64 and SD trained on text-image pairs. I wonder why not compare SIM with strong baselines on these two settings?\n- The presentation can be further improved. For example, the font size in Figure 2 is too small, and the legends in the right two figures are given without any explanation. Also, all the hyperlinks of references and citations are missing throughout the paper. Besides, there is a few typos: In Table 1, the FID of SiD ($\\alpha=1$) and SiD ($\\alpha=1.2$) can be switched. In line 24, “data synthetic” might be corrected as “data synthesis”. In line 254, “one generation steps” might be corrected as “one-step generation”. \n- This is not a major concern but there is an obvious gap between the reproduced SiD results and their reported ones. The authors admitted that it is because SiD code was not released and more importantly, the SIM results can be further improved if its hyperparameters can be tuned based on SiD’s setting. So I wonder that since SiD code has now been released: https://github.com/mingyuanzhou/SiD, maybe it is a good time to look into the reproduction issue and decide whether it can improve SIM’s performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your useful suggestions. We will address your concerns one by one. Before that, we first give a summary of the main contributions of our work.\n\nIn this work, we introduce score implicit matching (SIM), a novel distillation algorithm that achieves **competitive performances on CIFAR10 generation** tasks, and **lossless aesthetic performances** on the one-step distillation of the **DiT-based text-to-image diffusion model**. Besides the strong empirical performances, the theoretical foundation of SIM is also sound: we prove that the SIM loss has the same $\\theta$ gradient as our proposed score-based divergence. Therefore **using an SGD-type optimization algorithm to minimize SIM loss is secretly equivalent to minimizing the intractable score-based divergence**. Besides, our proposed score-gradient theorem may bring new tools for future studies on generative models. \n\nNext, we address your concerns one by one.\n\n**Q1**. Clarify the reason for using Pseudo-Huber distance.\n\n**A1**. We appreciate your keen intuitions. Though the standard L2 distance seems a fair choice, using the Pseudo-Huber distance is better. The Phuber loss (equation 3.8) has a denominator of the form $\\sqrt{\\||y_t\\||^2_2 + c^2}$. Here $y=s_g(x,t) - s_d(x,t)$ will be close to zero when the generator is converging. Therefore, if we set $c=0$ (the case of squared L2 distance), such a denominator is unstable and can lead to numerical issues. This means that Phuber with a small $c$ is better than the standard of L2 distance. \n\nAnother advantage of Phuber loss (equation 3.8) is that it has an adaptive self-normalization effect on the loss scale. This helps the training loss to be stable with a constant scale. \n\n**Q2**. Can the authors show how SIM becomes a DMD objective if we set $d(\\cdot)$ to be reversed KL? \n\n**A2**. If we consider the EDM formulation of Algorithm 2 in the Appendix, the DMD loss can be viewed as a special case of SIM loss. Let $d_{\\phi}(x_t,t)$ denotes the online EDM model, and $d_q(x_t,t)$ the teacher EDM model. In Algorithm 2, the SIM loss with Phuber distance writes: \n$$L(\\theta) = \\frac{d\\_{\\phi}(x\\_t,t) - d\\_q(x\\_t,t)}{\\sqrt{\\||d\\_{\\phi}(x\\_t,t) - d\\_q(x\\_t,t)\\||\\_2^2 + c^2}} (d\\_{\\phi}(x\\_t,t) - x\\_0).$$ If we cut off the $\\theta$ gradient of $x_t$, then $d_{\\phi}(x_t,t)$ and $d_q(x_t,t)$ will have no parameter dependence. So the SIM loss is equivalent to DMD loss with a weighting function $\\frac{1}{\\sqrt{\\||d_{\\phi}(x_t,t) - d_q(x_t,t)\\||_2^2 + c^2}}$. However, the theory behind SIM and DMD is essentially different, because SIM aims to minimize a general family of score-based divergences, instead, DMD aims to minimize the KL divergence which is notorious for mode-collapse behavior. \n\nThere might be a reason why **score-based divergence is a better choice than KL**. Recall that the definition of KL divergence needs the likelihood ratio $\\frac{p_g(x)}{p_d(x)}$, which will be ill-defined when $p_d$ and $p_d$ have misaligned density support. This can potentially lead to mode-collapse issues when using KL divergence as a target. However, the score-based divergence does not have such a \"ratio\", and therefore is safe in the case when $p_g$ and $p_d$ have misaligned support. \n\n**Q3**. The motivation to distill PixelArt-$\\alpha$ diffusion model. \n\n**A3**. In Section 4.1, we evaluate the SIM on the CIFAR10 generation benchmark thoroughly and observe strong performances without using GAN losses. After that, we would like to scale SIM to challenging text-to-image generation. We choose PixelArt-$\\alpha$ diffusion as a teacher model for three reasons. **(1)** We appreciate the great efforts of previous works on diffusion distillation of UNet-based t2i models. However, **distilling DiT-based diffusion models, such as PixelArt-$\\alpha$ diffusion, lack explorations**. **(2)** The PixelArt-$\\alpha$ model uses the T5-xxl text encoder, which is **able to understand long prompts** better than Stable Diffusion models with CLIP text encoders. We are motivated to explore high-quality one-step generator models that have such a long-prompt following ability. **(3)** We would like to explore the limits of a one-step model that is **favored by human users** in terms of Aesthetic Scores. The PixelArt-$\\alpha$ is a good model that has been trained with high-quality aesthetic images. \n\n**Q4**. Missing hyperlinks and some suggestions to improve the presentation. \n\n**A4**. We feel sorry for the confusion. The hyperlinks are missing because we have separated the main pages and the appendix. We will incorporate your suggestions to improve our presentation in the revision. \n\n**Q5**. Not a major concern, but I wonder if SIM can be further improved with the SiD codebase.\n\n**A5**. We appreciate your constructive suggestion. After reading the SiD's official code, we find the official code of SiD has multiple tricks that are different from our original implementation. With some tests, we find that these tricks, such as masking out the nan samples, the weighting function, as well as the continuous-time Karras-$\\rho$ noise sigma distribution help stabilize the training and improve the performance. \n\nDue to the limited time of the rebuttal period and limited compute, we conduct experiments with two settings to compare SIM and SiD behavior at different training stages: training from scratch, and resuming from a nearly converged generator; **Please check our global rebuttal and added one-page** for technical details. \n\nOur new experiment shows the faster convergence of SIM than SiD ($\\alpha=1.0$) using the same implementation techniques. However, training to the end will take a very long time, and we have to acknowledge that we probably can not make a final conclusion that SIM will be better than SiD without sufficient computation. Besides, we appreciate SiD's high-quality implementations and solid contributions. \n\n**We hope our response has resolved your concerns. If you still have any concerns, please let us know.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ROIfAesrfu", "reviewer": "Reviewer_Jq2C", "summary": "The authors study the distillation of score-based models in one-step generators. They propose a new objective function for score distillation coined Score-Based Divergence. This divergence measures the mean distance between the pre-trained score-based model and a score-based model learned on the distribution induced by the one-step generator. This method is then applied on image generation problems, on standard academic benchmarks and on large latent text-to-image models.", "questions": "* Have you observed instabilities with the optimization procedure that alternates between online score and generator? Have you tried to explore other hyper-parameters, e.g. more steps or different learning rates for the online score network? \n\n* In ProlificDreamer, the authors use LORA to fine-tune the online score-based model. Have you tried such approach to reduce the computational cost? \n\n* Could your method be directly used for learning 3D models, such as ProlificDreamer? From what I understand, setting $\\theta$ to an ensemble of particles and generating 2D images through NERFs, this would be practicable. \n\n* Theoretically, your method is agnostic to the type of generator that is used, right? Thus, we could imagine training a model with low-dimensional latent space and different architecture, such as a StyleGAN. Have you explored this?", "limitations": ".", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, this is a strong paper with a sound and novel method that obtains solid experimental results. \n\n* The authors study the distillation of score-based models, which is a very hot research topic since inference time is the major bottleneck of such models. \n\n* As far as I know, the idea of score-based divergence is novel and original. Moreover, the method comes with state-of-the-art results in a very competitive area (distillation of score-based models to one-step generators).\n\n* The general idea of score-based divergences is sound and could lead to new research works, from applied ones to theoretical ones.", "weaknesses": "* Not a big weakness, but we could argue that the training is demanding from a computational viewpoint, since it involves three large neural networks: generator, score-based network for data distribution, score-based network for generator's distribution. However, this is not specific to this particular method. \n\n* Minor: typo in the TL;DR. \"The submission propose an general\" -> \"The submission proposes a general\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that you like our work. We appreciate your valuable suggestions, and we will incorporate them in our revision. In this paper, we introduce score implicit matching (SIM), a theoretically sound distillation method that secretly minimizes a general family of score-based divergences when distillation. Our experiments on CIFAR10 generation as well as the one-step DiT-based text-to-image generator model show the superior performances of SIM on industry-level applications. In the following paragraphs, we will address your concerns one by one.\n\n**Q1**. Not a big weakness, but the training computation cost may be demanding. \n\n**A1**. We acknowledge that the training cost is a little bit expensive because the SIM involves an additional online diffusion model. However, in practice, we find that the **additional cost is acceptable** by using Pytorch techniques. For example, in our text-to-image experiment, we use 4 Nvidia A100-80G GPUs for training the one-step generator. We use the Pytorch Distributed data-parallel (DDP) to support multi-GPU gradient syncing and use BF16 numerical formate for training. With BF16 training, we can effectively train the model with a total batch size of 256 even without the gradient accumulation. If we use gradient accumulation techniques, we can enlarge the batch size to 1024 with some speed decline. \n\n**Q2**. Are there instabilities with the alternating optimization procedure? Would be better to explore other hyper-parameters.\n\n**A2**. We appreciate your good intuition. We have observed that **if the hyper-parameters are not set properly, the training may diverge**. For instance, we find that when the weighting functions and the log-normal noise sigma distribution for training online diffusion models are not properly set, the alternating training will lead to a poor result. However, compared with Diff-Instruct and SiD, we find that the updating of a one-step generator using SIM is pretty robust to hyper-parameters. We find that using different loss scaling, different learning rates (in a reasonable interval), as well as several kinds of weighting functions, does not affect the SIM updates for the generator too much. We appreciate your constructive suggestions as a new direction to explore. We have tried to update the online diffusion twice, however, this slows down the training process by approximately 1.5 times and we do not observe significant performance improvement, therefore, we still use one diffusion update per generator update, with two updates using the learning learning rate. However, we will explore more of these settings in the revision. \n\n**Q3**. Have you tried using Lora such as in ProlificDreamer [2] to reduce the computational cost?\n\n**A3**. We appreciate your good intuition. We like the trick proposed in ProlificDreamer to use a Lora online diffusion, which can reduce the memory cost. However, as we have shown in **A1**, when we use BF16 training, we are not bothered by memory issues, therefore we do not use the Lora online diffusion. Another reason is that the Lora model may have worse modeling ability than a full-parameter model, which can potentially harm the performance of SIM. \n\n\n**Q4**. Can SIM be used to train NeRF models using 2D diffusion? Can SIM be applied to other generator models such as GANs?\n\n**A4.** We highly appreciate your good intuition. As you can see, the SIM is a general method that can be used for a wide range of applications other than one-step diffusion distillation. In the rebuttal period, we conduct two more experiments: (1) text-to-3D generation using text-to-2D diffusion; and (2) improving the GAN generator using diffusion models; **Please check our Global rebuttal cell for details.** On both new applications, SIM has shown strong performances, demonstrating its broad applicability. \n\n**We hope our response has resolved your concerns. If you still have any concerns, please let us know.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LjOcnRFYih", "reviewer": "Reviewer_WiWu", "summary": "This paper proposes a distillation-based accelerated sampling method for various score-based diffusion models, such as EDM and Stable Diffusion. The authors have named this method Score Implicit Matching (SIM), which is designed to compress information from a diffusion-based teacher into single-step generator models. This method is somewhat based on SID, and SIM can be regarded as an extended version of SID to some extent. In my opinion, the crucial contributions of this paper are its application of SIM to Stable Diffusion and the more general design choice of the distance function.", "questions": "No", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper applies SIM to text-to-image (t2i) generators such as Stable Diffusion, an important experiment validating the effectiveness of accelerated sampling algorithms.\n\n2. This paper enhances SID by generalizing the design choice of the distance function to six different versions, as shown in Table 4 of the Appendix.\n\n3. This paper provides proof of their method, although the contribution of this point is relatively small since most of the proofs are based on previous work. To be specific, (1) the proof in SID use the data distribution as the target but SIM utilize the score function as target; (2) SIM uses the derivative form for derivation, but SID draws its conclusions through \n$\\nabla_{x_t}\\log p_\\theta(x_t)=[x_g-x_t]/\\sigma^2_t$\n (In EDM $\\alpha_t \\equiv 1$); (3) the only difference is the difference in the distance function, but this does not affect the derivation in any way.", "weaknesses": "1. The performance of SIM seems to be not effective, although the authors' statement that SIDs are reproduced because they don't have publicly available code. The performance gap may originate from a few tricks, and it is not yet known whether these tricks can further improve the effectiveness of SIM. Thus. I hope the authors can retrain several experiments related to CIFAR10 to further evaluate the effectiveness of SIM through SID's official implementation [1].\n\n[1] https://github.com/mingyuanzhou/sid\n\n2. The author spends a lot of space in the ``Score Implicit Matching\" section to introduce the principle of Algorithm 1, but I think the biggest difference between SIM and SID is that SIM adopts the matching based on the score function while SID adopts the matching based on the sample. The rest of the logic is basically the same. Therefore, I think it is a bit unreasonable to take this part as the contribution of the paper.\n\n3. The authors of SID recently released a new paper named ``Long and Short Guidance in Score identity Distillation for One-Step Text-to-Image Generation'' (LSG) which is also an extended version of SID. I wish the authors would add LSG into related work (unfortunately, different metrics were used between SIM and LSG, so SIM cannot simply compare it with LSG).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your useful feedback. We will address your concerns one by one. Before that, we first give a summary of our work.\n\nIn this work, we introduce score implicit matching (SIM), a novel distillation algorithm that achieves competitive performances on CIFAR10 generation tasks, and **lossless aesthetic performances** on the one-step distillation of the **DiT-based text-to-image diffusion model**. Besides the strong empirical performances, the theoretical foundation of SIM is also sound: we prove that the **SIM loss shares the same $\\theta$ gradient as our proposed score-based divergence**. Therefore using an SGD-type optimization algorithm to minimize SIM loss is secretly equivalent to minimizing the intractable score-based divergence. This **distinguishes SIM from SiD's theoretical loss functions** that are proved to be equal to Fisher divergence in value instead of parameter gradient when using an online learned score to replace the unknown generator score.\n\nNext, we address your concerns one by one.\n\n**Q1**. The relation between SIM and SiD\n\n**A1**. We appreciate the SiD paper and have obtained inspiration from it. The SiD shows a solid step to minimize Fisher divergence other than KL. However, SIM and SiD are essentially different in the following aspects.\n\n**1)** SIM loss has the same parameter gradient as the intractable score-based divergence. However, the SiD's theoretical losses only have equal value instead of an equal parameter gradient. \n\nIn Section 3.2, we have shown that SIM loss has the same parameter gradient (equation 3.5-3.7) as the target score-based divergence (equation 3.4) regardless of substituting the score function with an online learned score model (please check Theorem 3.1). This means that using an SGD-type optimization algorithm to minimize SIM loss is equivalent to minimizing the score-based divergence. On the contrary, the SiD theoretical loss functions (equation 13 and equation 20 in SiD paper) only have an equal value to the target Fisher divergence instead of the parameter gradient when using an online score function to replace the intractable generator score function. This means using a gradient-based optimization algorithm does not guarantee the minimization of the Fisher divergence. This might be the reason why SiD's theoretical-driven losses do not work well in practice as the authors of SiD have pointed out. After the theoretical argument, the authors of SiD empirically find that a linear combination of the two \"bad\" losses results in a strong loss function, which they call the \"fused loss\". Interestingly, we find that when taking SIM's distance function to be squared L2 distance, SIM recovers the empirical \"fused loss\" of SiD with $\\alpha=1.0$. **From this point of view, SIM, as well as our score-divergence gradient theorem brings a theoretical tool for analyzing SiD's empirical fused loss in a gradient view.**\n\n**2)** SIM is defined over a general family of score-based divergence with a flexible choice of distance function. As a comparison, the SiD only targets Fisher divergence, which can be viewed as a special case of SIM. Besides, trying to generalize SiD's theoretical loss (equation 20 in SiD) to support general distance functions is not direct. The proof of Equation 19 of the SiD paper relies on the expansion of the squared L2 difference. However, such an expansion may not hold for general distance functions such as L2 distance and the Pseudo-Huber distance.\n\n**3)** SIM empirically shows faster convergence than SiD. This makes SIM favored when training large-scale one-step text-to-image generator models. Please see **A2** for details. \n\n**4)** SIM shows novel scalability to complicated text-to-image one-step distillation, however, the SiD did not show such a possibility before the NeurIPS submission.   \n\nIn short conclusion, SIM has proven the equality of the loss function and the intractable divergence in a **parameter gradient** view, making it a solid theoretical contribution. Besides, SIM has shown success in scaling to train strong one-step text-to-image generators without performance losses, making it a solid empirical contribution. \n\n**Q2**. Will SIM be further improved with the SiD codebase? \n\n**A2**. We appreciate your constructive suggestion. After reading the SiD's official code, we find the official code of SiD has multiple tricks that are different from our original implementation. With some tests, we find that these tricks, such as masking out the nan samples, the weighting function, as well as the continuous-time Karras-$\\rho$ noise sigma distribution help stabilize the training and improve the performance. \n\nDue to the limited time of the rebuttal period and limited compute, we conduct experiments with two settings to compare SIM and SiD behavior at different training stages: **(1) training from scratch**, and **(2) resuming from a nearly converged** generator; **Please check our global rebuttal and added one-page** for technical details. \n\nThe new experiment shows that the faster convergence of SIM than SiD ($\\alpha=1.0$) using the same implementation techniques. However, training to the end will take a very long time, and we have to acknowledge that we probably can not make a final conclusion that SIM will be better than SiD without sufficient computation. Besides, we appreciate SiD's high-quality implementations and solid contributions. \n\n**Q3**. The authors of SID recently released a new paper named LSG, I wish the authors would add LSG into related work. \n\n**A3**. Thank you for your kind reminder. The LSG is publicly available after the NeurIPS submission deadline. After a careful reading, we find it technically interesting. We think the Long-and-short guidance technique proposed in LSG may further improve the SIM performance for text-to-image models. We will add the LSG to our related work in the revision. \n\n**We hope our response has resolved your concerns. If you still have any concerns, please let us know.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all reviewers for your valuable feedback. In this cell, we address some common concerns. 

As the **Reviewer G1sA** and **Reviewer Jq2C** wish, we run two additional experiments to demonstrate the wide applicability of SIM: (1) text-to-3D generation using text-to-2D diffusion; and (2) improving the GAN generator using diffusion models; On both new applications, SIM has shown strong performances, demonstrating its broad applicability. 

**(1) text-to-3D generation.** We follow the setting of ProlificDreamer which minimizes the KL divergence in order for 3D generation. We use the open-sourced ThreeStudio Pytorch implementation of ProlificDreamer.

Result: **Figure 3** of our **added one-page** in the global response shows a qualitative comparison of SIM and ProlificDreamer (VSD). We find that when using SIM for text-to-3D generations leads to decent results, which is visually comparable with ProlificDreamer. This result shows that SIM has the potential to be used for text-to-3D generation. However, since our paper aims to focus on one-step image generation, and the computation for a thorough quantitative evaluation for 3D generation is expensive, we only give the qualitative visualization results in the rebuttal period. We leave more study and tuning on using SIM for 3D generation in our future work. 

**(2) improving StyleGAN2 generator.** In this experiment, we use SIM to improve a pre-trained stylegan2 generator with pre-trained EDM diffusion models, following the same settings as the Diff-Instruct [2] paper's Section 4.2. **Table 1** and **Table 2** in our **added one-page** in the global response show the quantitative results of improving the generator experiment. As a result, we surprisingly find that SIM is able to improve the pre-trained StyleGAN2 generator better than Diff-Instruct can. This shows that SIM has a high potential to improve existing GAN generators, such as StyleGAN-T and GigaGAN. We will leave the scaling-up work to improve text-to-image GAN models in future work. 


[1] ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation

[2] Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models


As **Reviewer WiWu** and **Reviewer wPU2** wish, considering the limited time of the rebuttal period, we run two new experiments to compare SIM and SiD on **CIFAR10 unconditional generation** task behavior at different training stages: (3) training from scratch, and (4) resuming from a nearly converged generator; 

**(3) compare SIM and SiD from scratch.** We compare SIM and SiD ($\alpha=1.0$) with SiD's official code and hyper-parameters to train a one-step generator from scratch. Both training runs are on 8 Nvidia-A100-80G GPUs with a total batch size of 128. We use the default settings as SiD, with the only difference being the use of loss functions. We log the FID value along the training process as a performance metric. **Figure 1** in our **added one-page** shows the FID Comparison of SIM and SiD for training from scratch. 

**Result**: both methods can converge steadily. The SIM converges faster than SiD, with a lower FID value at the same number of training iterations. 

**Findings**: We find that the weighting function that SiD uses is important for stable training. We find that using this weighting function for SIM can improve SIM's performance compared with our original implementation, which makes us appreciate the technical contribution of SiD authors for their high-quality implementations. We also find that training a generator to a very low FID is slow for both SIM and SiD. Due to the limited time of rebuttal period, this motivates us for our second experiment. 

**(4) compare SIM and SiD by resuming from a nearly-converged generator.** In this experiment, we compare the convergence speed with SIM and SiD in the case of resuming from a nearly-converged one-step generator. Because a full training trial is very slow (it may last for up to several weeks with 16 GPUs). Therefore training until the end is impossible for the short rebuttal period. Therefore, we resume the generator from a checkpoint that we previously trained with our SiD implementation which has an FID of 2.8. We aim to inspect the training behavior of SIM and SiD near convergence with this experiment. For this experiment, we use 4 Nvidia-H800 GPUs with a batch size of 128 for both SiD and SIM. All hyperparameters of SIM are set the same as SiD's default choices except for a learning rate of 1e-6. 

**Result**: **Figure 1** in our **added one-page** shows the FID Comparison of SIM and SiD. Both methods can converge slowly near convergence. The SIM converges faster than SiD, with a lower FID value with the same compute. With a couple of days of training, the SIM is able to reach a minimum FID of 2.06 from 2.8, while the SiD's minimum FID is larger than 2.15. 

Next, we will address each reviewer's concerns in individual rebuttal cells.

Meta-review: Reviewers agreed that this work presents a robust and efficient method for distillation into one-step generators. While there were some concerns on experimental differences with prior work, the experimental results on large-scale text-to-image models are compelling and demonstrate effective one-step distillation.
Decision: Accept (poster)