Guided Score identity Distillation for Data-Free One-Step Text-to-Image Generation
Authors: Mingyuan Zhou
Zhendong Wang
Huangjie Zheng
Hai Huang
Venue: ICLR 2025 Poster (2025)
Keywords: stable diffusion
data-free distillation
single-step generation
classifier-free guidance
Primary area: generative models
TL;DR: We have developed long and short guidance strategies within Score identity Distillation to efficiently distill Stable Diffusion models into one-step text-to-image generators, all in a data-free manner.

Abstract: Diffusion-based text-to-image generation models trained on extensive text-image pairs have demonstrated the ability to produce photorealistic images aligned with textual descriptions. However, a significant limitation of these models is their slow sample generation process, which requires iterative refinement through the same network. To overcome this, we introduce a data-free guided distillation method that enables the efficient distillation of pretrained Stable Diffusion models without access to the real training data, often restricted due to legal, privacy, or cost concerns. This method enhances Score identity Distillation (SiD) with Long and Short Classifier-Free Guidance (LSG), an innovative strategy that applies Classifier-Free Guidance (CFG) not only to the evaluation of the pretrained diffusion model but also to the training and evaluation of the fake score network. We optimize a model-based explicit score matching loss using a score-identity-based approximation alongside our proposed guidance strategies for practical computation. By exclusively training with synthetic images generated by its one-step generator, our data-free distillation method rapidly improves FID and CLIP scores, achieving state-of-the-art FID performance while maintaining a competitive CLIP score. Notably, the one-step distillation of Stable Diffusion 1.5 achieves an FID of **8.15** on the COCO-2014 validation set, a record low value under the data-free setting.  Our code and checkpoints are available at https://github.com/mingyuanzhou/SiD-LSG.

Published as a conference paper at ICLR 2025
GUIDED SCORE IDENTITY DISTILLATION FOR DATA- FREE ONE-STEP TEXT-TO-IMAGE GENERATION
Mingyuan Zhou 1.2., Zhendong Wang', Huangjie Zheng', and Hai Huang
1 The University of Texas at Austin, "Google DeepMind, and Atlassian
distinguiched older gentleman sahurian landsape sunset 4k chinese red blouse, the style futuristic simple muliilayend ar- posier the collection of the chitecture lubitation caitinn in avian woman, dynamic anime the nees, drametic soft light [-.] [--], mysterious realism,, [-.]
a vintago sndyy uurounded by ultm malismm BY Anton Garlin, of dreamy and romantic compa- trending artstation, [..]] attions, floral explosions. [--]
books and dim lighting [--]
A fantasy-henndd portunit very beautifull girl I...k white stcampunk atmosphere, a stuo- (Pirate ship sailing into a biolu- thhrt design, colouraal, no book- mineseence with galaxy ground. yoda with sun glasses, the epic, ultra.
female elt with golden haar and sboot top, charismatic personali violes eyes, her attire
ning zirt with mecha musume aesthtic, adomed
professional photo. [...]
dancing festival I..]] 8k.
Figure I: Example generation results of resolution 512x512 from the one-step generator distilled from Stable Diffusion 2.1- base using the proposed method: Score identity Distillation with Long- -hort Guidance.
ABSTRACT
Diffusion-basee text-to-image generation models trained on extensive text-image pairs have demonstrated the ability to produce photorcalistic images aligned with textual descriptions. However, significant limitation of these models is their slow sample generation process, which requires iterative refinement through the network. To overcome this, introduce data-free guided distillation method that enables the efficient distillation of pretrained Stable Diffusion models without access the real training data, often restricted due to legal, privacy, cost concerns. This method enhances Score identity Distillation (SiD) with Long and Short Classifier-Fis Free Guidance (LSG), an innovative strategy that applies Classifier- Free Guidance (CFG) not only to the evaluation of the pretrained diffusion model but also to the training and evaluation of the fake score network. We optimize model-based explicit score matching loss using score-identity-basee approxi- mation alongside our proposed guidance strategies for practical computation. By exclusively training with synthetic images generated by its one-step generator, data-free distillation method rapidly improves FID and CLIP scores, achieving state-of-the-e art FID performance while maintaining competitive CLIP score. Notably. the one-step distillation of Stable Diffusion 1.5 achieves an FID 8.15 on the COCO-2014 4 validation
record low value under the data-free setting. Our code and checkpoints are available at
'The majority of the work done while the authors Google.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
The pursuit of generating high-resolution, photorealistic images that matches the textual descriptions has driven the machine learning community in developing powerful texi-to-image (T21) generative
models. T2I diffusion models (Nichol et al. 2022 Ramesh al. 2022 Saharia et al. 2022 Rombach
ct al. 2022 Podell et al. 2024) are currently leading the way in delivering unprecedentedly visual quality, diverse generation, and accurate text-image correspondences. They are renowned for their straightforward implementation and stability during optimization, and they receive substantial acclaim for the robust support from the open-source community (Rombach et al. 2022 Podell et al. 2024).
Despite these advantages, significant limitation of diffusion models, including those used for T21 tasks, is their slow sampling process, which involves iterative refinement through repeated passes of the generation network. Originally, this required thousands of stochastic sampling steps (Song & Ermon 2019; Ho et al. 2020; Dhariwal Nichol 2021 Song et al., 202 Recent advancements in
ODE-bases deterministic samplers have significantly reduced the required number of sampling steps
to just tens or hundreds (Song et al.] 202 Liu et al.] 2022a Lu et al.] | Karras et al. (022).
To further reduce the number of steps below ten, even down to one, the focus has shifted toward distilling the iterative-refinement based multi-step T21 generative progress, using a wide variety of
acceleration techniques (Zheng al. 2023b Meng al. 2023 Liu al. 2022b Luo et al. 2023b
Nguyen & Tran 2024 Sauer et al. 2223bb] Xu et al. 2023] Yin et al.) (223). However, they often result clearly reduced ability to match the original data distribution, reflected clearly worsening
FIDs. All of them, with the exception of SwiftBrush (Nguyen & Tran 2024 also require access to real images or the assistance of extra regression or adversarial losses.
It is commonly believed that student models used for distillation sacrifice performance for increased speed. However, recent findings from Score identity Distillation (SiD) (Zhou et al. 2024) present a notable discovery. The SiDD-based single-step student model, although trained in a data-free manner, not only simplifies the multi-step generation process required by the teacher diffusion model, EDM by Karras et al. (2022), but also excels in performance. It surpasses the teacher model in terms of Frechet inception distance (FID) (Heusel et al. 2017) on the CIFARIO-32M0 FFHQ-64x64, and AFHQ-v2 64x64 datasets. It only slightly underperforms in FID in comparison ImageNet 64x64.
The success of SiD in distilling EDM diffusion models for non-T2I generation in the pixel space has inspired us to adapt it to open-source T2I latent diffusion models, specifically Stable Diffusion (SD) versions 1.5 and 1-bse, aiming to significantly enhance their generation speed while maintaining performance. However, adapting SiD poses several notable challenges: first, SiD-EDM does not incorporate classifier-free guidance (CFG) (Ho & Salimans, 2022). which is integral to SD; second, SiD has primarily been applied distill pre-trained EDM models, which utilize noise scheduling and preconditioning methods markedly different from the DDPM noise scheduling employed in SD; third, both the complexities and sizes of the data and model in EDM are much smaller than those SD.
To address these challenges. we explore the integration of CFG and SiD for T2I diffusion distillation. In addition testing the conventional approach of enhancing CFG on the pretrained score network, we introduce a novel strategy of reducing CFG on the fake score network, as well as a combined approach that employs both strategies, which we refer to as long and short guidance (LSG). This new method efficiently distills SD models into one-step T2I generators without requiring training data. Surprisingly, it achieves a new benchmark on the COCO-2014 validation set with zero-shot FID score of 8.15, the lowest to date for one-step, data-free distillation, despite not relying on additional regression or adversarial losses, nor real data- key components of recent distillation techniques.
The development of our SiD-LSG approach builds significantly prior work in generative models, CFG, and acceleration methods. A comprehensive review of related work is provided in Appendix B
2 DATA-FREE GUIDED SCORE IDENTITY DISTILLATION
We explore the use of SiD and CFG to distill SD, the leading open-source platform for T2I diffusion models that operate on the latent space of an image encoder-decoder (Rombach et al. 2022). with specific focus on the data-free setting. Our focus is specifically on SD-vI-3 (SD1.5) and SD- --asee ISDs.1-bbae), which are two versions extensively benchmarked for diffusion distillation.
2

---

Published as a conference paper at ICLR 2025
Figure 2: Rapid advancements in distilling Stable Diffusion 1.5 are showcased by the proposed SiD method that incorporates long-shi short guidance (LSG). Key parameters include batch size of 512, learning rate of 1c-6, and LSG scale of 2. This data-free approach achieves zero-shot FID of 9.56 the COCO-2014 validation set, along with a competitive CLIIP score of 0.313. By reducing the LSG scale to 1.5, the FID can be further lowered to a record 8.15 among data-free diffusion distillation models, with a corresponding CLJP score of 0.304. The series of images, generated from the same set of random noises post-training the SiD generator with varying counts of synthesized images, illustrates progressions at O, 0.02, 0.1. 0.2, 0.5, 1, 2, 3, and 5 million images. These are equivalent to 0, 40, 200, 400, Ik, 2k, 4K, 6K. 8k, and 10K training iterations respectively, organized from the top left to the bottom right. The progression of FIDs and CLIPs is detailed in the orange solid curves in the left plot of Fig. The corresponding COCO-2014 validation text prompts are listed in Appendix F
2022), essential for T2I diffusion models to enhance their photorealism and text alignment, into the SiD loss functions. The second challenge is adapting SiD, initially tested with EDM noise scheduling, to the DDPM scheduling employed by SD. Addressing these challenges primarily requires modifying the derivation and loss functions of SiD. A third challenge arises because SD models are significantly larger and trained on bigger, more complex, and higher resolution data. Overcoming this challenge requires addressing numerous technical details, such as establishing the minimum hardware requirements and configuring the appropriate software settings to align with the constraints of the available computing platforms.
significant hurdle involves incorporating CFG (Ho & Salimans
2. PRELIMINARIES ON SCORE IDENTITY DISTILLATION
We denote as the text representation of pretrained text encoder, such CLIP (Radford al. 2021). Our objective is to distill a student model po (xg c) from a pretrained T21 diffusion model, such as SD1.5, which can generate text-guided random samples in single step as: z. Go(2. c), p(z), where Gg is a neural network parameterized by 0 that deterministically transforms noise = ~ p(z) into generated data 2g under the guidance of text c. The distribution of wy is often implicit (Mohamed & Lakshminarayan
2016; Tran et al., 2017; Yin & Zhou, 2018), lacking an analytic probability density function (PDF) but is straightforward to sample from. The marginals of the real and generated data under the forward diffusion process can be expressed as:
(
g( (mo) (mop c) datp,
po( c) g( (g) |c) dag--
This structure, characterized by explicit conditional layers but implicit marginals, exemplifies a semi-implicit distribution (Yin & Zhou 2018 Yu et al., 2023). This concept is employed by Zhou jet al. (2024) to develop SiD, a method whose single-step data-free distillation capability has so far been demonstrated only on non-T2I diffusion models based on EDM.
We define the forward diffusion transition as q(æ;, an) = Na ofI), and unlike in SiD, which adheres EDM noise scheduling where a;
allow az to vary within [0, 1] to align with the DDPM scheduling used by SD. This necessitates generalizing the equations used in SiD by permitting
a, * 1. We note that other diffusion types, such as categorical (Austin et al. 2022 Hoogeboom et
2021
Gu et al. 2022, Hu et al. 2022). Poisson (Chen & Zhou 2023), and beta diffusions (Zhou 2023), also align with the semi-implicit framework and can potentially be adapted similarly.
ct al.
Score identities. The scores S(x;) := Ve, In Pussa (x2 c) and V., In pe c) are generally unknown. However, the score of the forward conditional g( x)) N(x analytic:
If æ; ar Obee, 4 NO,II) then Va, In q(2; |x) o, 2 xx) -0, lc.
3

---

Published as a conference paper at ICLR 2025
Exploiting the semi-implicit constructions, we follow SiD to present the following three identities:
E|xo ze; c| = x Tt; c) dæg = (x, + of In Paata C c))/as,
E| æt. c] = x ( xz) dx, = (æ) + Va. Im (x) c))/ar-
ppo(ar))) [u" (ar;) V.. In Inpa c)] y æ ( le) [u" (ar)) Vx. In g(z, Fgg c)]
MESM loss. A pretrained T2I diffusion model, such SD, provides score network Sp parameter- ized by & that estimates the true data score as
o. Va, In Pdmaa (2: e) -0r c) (
(2),
It adopts fo 1, c) as the functional approximation of the conditional expectation of the real image 2o given noisy image H, and text c. expressed E[ xr, cj. adopts Eg(2,, t, c) to predict the noise inside Zt. and adopts -a, Ex(2,, c) as the functional approximation of the true score V Pdsta c). Given time step t ~ p(t) and text c, we define the modeel-based explicit score- matching (MESM) distillation loss, which form of Fisher divergence (Lyu 2009 Holmes &
Walker 2017
Yang et al. 2019 Yu & Zhang 2023),
Lo Ex (o.) [ Va, po |c) c)
Loss approximation for distillation. The MESM loss in [ is in general intractable to compute as Vz, po (x, c) is unknown. To denoise the noisy fake data 2, generated as
gg + oce, 6 I), Fg G)( there exists an optimal denoising network defined as (or (I t, B[a:g x of Va, In (a e))/aa /a. Given this optimal denoising network, the MESM loss would become
p(p))
(xx
Lo En (1la,,, (f (Ixe, t, c) (o) c)
As 1/* (0) is unknown in practice, we follow SiD to alternates between optimizing and using min (ar, = a2 (1,, 2=122 (1,, 4, <x 12.
(4)
min La ( (0,, 1,, = w() (So(ee,, ( c))² (Jo (1l 4, - xn),
(5)
0
= w(t) (t (c,( cs T) c ( c)),
where D2 is generated as in (2 and (t) are weighted coefficients that will be specified.
2.2 S1D WITH CLASSIFIER- GUIDANCE
An essential practice for enhancing photorealism and aligmment with text instructions in T2I diffusion models involves incorporating CFG into reverse diffusion. This principle also applies to distillation methods for these models, where CFG must be integrated into the appropriate terms of their distil- lation loss functions. Therefore, a key distinction in the distillation of SD, compared to previous unconditional and label-conditional diffusion models, lies in the need to introduce CFG.
SiD presents unique opportunity to apply CFG to enhance its T2I generation performance. First, we note CFG enhances text guidance by modifying the distribution of æ; given text c as
p( c, k) xx p(c] xx) (x)(((())))))))))))))))))))))( x pplza "p(x.),
which means Va, Inp(a; k) Va, Inx();) + <| V p / Va, In p(x:)) Second, the score network, when reaching its optimal, is related to the true score as f(x,,t,) = a, (xx + V., Inp(æ; (c)) = a, (x, - ore(x,, t, t)).. Therefore, with "." representing & or tb, can express the score network f. f, c) under CFG with scale k as
f,,( t, c) = f. t) + xff. (x,, c) - f.(x,,
(6)
Similarly. for noise prediction network, have c. (æ,, t, c) c. c.( t) + k(e. t, c) - (x,, t)]. With CFG, the score and noise prediction networks are related in the same way, which means that (,,,, 1, c) = -xx a.S (æ,, e)) and f.xx( 1, c) = a, (2x o.e.,, (,,, c)).
4

---

Published as a conference paper at ICLR 2025
Inspecting the two losses in 4 and 5 suggests four potential places to inject CFG. More specifically, with K1, K2; K3, K4 E R.,, where R = {or x 2 0), we modify the losses with CFGs as
La G. 1) = of pl (ott, 1, 1, - -rr ||E (2x, 1, c) cl Lo 1,0 b) w()) of Voons (, - Swi (2re, L, e))² (Frins (24,, () Fzg)
(7)
(8)
2.3 LONG AND SHORT GUIDANCE
Previous works equipped with fake score network f typically only consider adding CFG when evaluating the pretrained score network fo, such as in DMD (Yin et al. 2023). In the context of SiD, this corresponds to setting K1 R2 K3 1 and R4 > I. In this paper, discover that broad spectrum of combinations of Ki. Kig, K3, and R4 can all significantly enhance performance compared to not using any CFG all, which means setting K1 = K2 r5 K4 = 1. These different combinations are found to lead to different balances of minimizing FID, which reflects how well the generated data match the training data in distribution, and maximizing the CLIP score (Radford et al.) 2021), which reflects how well the generated images follow the textual guidance. This flexibility to accommodate various CFG combinations expands the design space for SD distillation, balancing generation quality and text adherence. However, given the vastness of the search space, we are motivated to develop strategies that constrain the scope of exploration. We present three such strategies, acknowledging that there may be more effective approaches not explored in this paper.
Long strategy: Enhancing CFG of the pretrained score network f Aligning with established practices, the most common approach involves enhancing the CFG of the pretrained score network fa- An example setting under this strategy in SiD is: K1 = R2 = K3 = and K4 = 3. The rationale is that by biasing the teacher fe to favor generations more aligned with using CFG scale greater than 1, such as K4 = 3. the student generator is compelled to follow suit. We will present experimental results to demonstrate the effectiveness of this strategy while also discussing its limitations.
Short strategy: Weakening CFG of the fake score network fu. With the availability of a fake score network føx we have developed a new strategy enhance SiD's T21 generation capabilities by reducing the CFG of f in SiD. An exemplary configuration is setting K1 = Ri4- = while allowing K2 K3 to vary within (0, 1). The underlying idea is that by diminishing fo capacity detect generations aligned with e through a reduced CFG scale (0- < K2 < 1), the student generator is incentivized produce images that better align with the textual guidance to compensate for this reduction. We will present experiments to assess the efficacy and limitations of this strategy.
Long and short CFGs. We introduce an innovative strategy termed long-shor guidance (LSG), where we enhance CFG during the training of fø by setting K1 > 1. and maintain CFGs on fy and fo during the training of Go at the same or lower scale by setting S R2 = Ki K4 S R1. The logic behind LSG is that enhancing the CFG of fa during training effectively reduces its CFG at evaluation time when used at the same or a reduced level. To our knowledge, we are the first to incorporate CFG into the training of the fake score network fø- LSG has been shown to effectively balance reducing FID scores while increasing CLIP scores, and will therefore be used as the default method.
We refer the reader to Appendix for further discussion the long and short CFG strategies.
3 EXPERIMENTS
We summarize the parameter settings, such batch size, learning rate, and optimizer configurations, in Table in Appendix [D Unless specified in ablation studies, the settings are uniformly applied across all guidance strategies. We present the details of our method in Algorithm D where generate an image in one step and generalize the setting in DMD and SiD to set ((t) as
Ky - Go(z, Selot... z, limit, c)2 N~ (0, w(t)
Te, (m. Ae)...
where C' is the total number of pixels in Zg. We initialize both &b and 0 using the pretrained ø from SD. We have tested finit (354, 550, 625, 675, 800, 900, 999} and found that the model's performance under FP32 optimization is not sensitive to these choices. To align with the setting in SiD, we set linit 625, which would result in atiec P /at... 2.5 under the DDPM schedule used to train SD1.5.
5

---

Published as a conference paper at ICLR 2025
CLIP Scores vs Iteration
FID and Scores vs Iteration
0:31
033
101
101
FD0 7DD3core
CLP Sour
FID Scare
CFG
0.27
027
0.2
10k
100k Serated number images
JM
2:54
100k Ierated number of images
1M
2.5M
Figure 3: Left (Long CFG of the truc score network): corresponding rise in CLIP scores, each influenced by different CFGs applied to the true score network. values specified in the legend displayed on secondary y-axis in corresponding line styles but with slight transparency. Together, these lines demonstrate how various CFGs impact model performance. Right (No CFG; Short CFG of the fake score network with K2 = E (0, 1); simple form of LSG that sets K1 > 1): Analogous plot to the left where the CFGs of the fake score network are not applied, weakened during evaluation, or enhanced during training.
This plot illustrates the gradual decline in FID and the
1. FID scores are plotted the primary y-axis, while CLIP scores
FID and CLIP Scores vs Iteration
FID and CUP Scores vs Iteration
100
201
0.3
RD Score
CUP Scoree
PID Score
10
101.
022
10k
102k
10M
10kk
1M
zerated number images
Ierated number of images
Figure 4: The plot with the proposed long-short guidance (LSG) demonstrates the FID and CLIP progressions of SiD in SD1.5 (left panel) and SD2.1-base (right panel).
We conduct a comprehensive study to evaluate the performance of SiD using the proposed LSG for distilling SD1.5. Additionally, we apply the same LSG scales to distill SD2. 122.--bacc, further assessing the adaptability and effectiveness of our approach across different model versions.
We consider standard setting that utilizes the Aesthetics6+ prompt (Chrrti et al. 2023) for train- ing and evaluates performance by computing zero-shot FID on the COCO-2014 validation set. We adhere to the standard protocol by generating 30k images to compare with the 40,504 im- ages in the COCO-2014 4 validation set for calculating zero-shot FID, Additionally, we employ the V
1 2b 444 2 encoder (Tharcco al. 202T Cherti et al.] (023) compute the CLIP score (Radford et al. 2021). The FID and CLIP scores presented in the figures are calculated using randomly sampled prompts from the COCO-2014 validation set. When reporting the FID and CLIP results of SiD in Table i we use the exact evaluation code codeeprroviee by GigaGAN (Kang et al. 2023). where a pre-defined list of 30k text prompts selected from the COCO-2014 validation set is used generate the 30k images, which are used for computing FID and CLIP scores with images from the validation set as the reference.
3.1 LONG AND/OR SHORT GUIDANCE STRATEGIES
No CFG. Figure demonstrates that without CFG, where Kı = N2 = K3 K4 1, results are underwhelming, highlighting the need for CFG in SiD to distill T2I diffusion models.
Long the CFG of the pretrained score network. For the long strategy, we set K1 = K3 = 1 and K₃ > 1, specifically exploring values of K4 from the set {2, 2.5, 3.5, 7.5}. As shown in the left panel of Figure 3 our experiments results indicate that this setup yields highly competitive performance, even after processing as few as 2.5 million fake images (approximately 5,000 iterations with a mini-batcl size of 512). The parameter 144 plays a crucial role in balancing FID and CLIP scores. For achieving lower FID scores, K4 2 can yield an FID close to 10, while for higher CLIP

---

Published as a conference paper at ICLR 2025
Table Comparison image generation methods across various metrics, Inference times are estimated using Nvidia 1100 GPU as reference. The numbers of the methods marked with are produced by running the publicly available model checkpoints. Other data are sourced from corresponding scientific papers with comparisons as
reported. The value followed by symbol ~ indicates it is estimated based the plots shown in the paper.
Method
Res.
Time (L)
Steps
Param.
FID (L)
CLIP (1)
Autoregressive Models
DALL-E (Ramesh et
256
12B
27.5
Viewi [ing
256
688
24.0
Parti-750M Yu et al. 2022
256
750M
10.71
Parti-3B Yu sl. 2022
256
6.4s
3B
8.10
Parti-20B
256
20B
7.23
Make-
(Gami 2022
2022
256
25.0s
11.84
Masked Models
Muse
Chang et al. 2023
256
24
7.88
0.32
Diffusion Models
GLIDE Nichol 2022
256
250++7
12:24
DALL-E Z (Kamesh 2022
256
250+27
5.5B
10.39
LDM-KL-8-D
(Kmbach al.
2022
256
3.7s
250
1,45B
12.63
Imagen Ho al 2022
256
9.1s
3B
7.27
eDiff-I Balaji et al.
256
32.0s
25+10
6.95
Generative Adversarial Networks (GANs)
LAFITE Zhou et 2022
256
0.02s
75M
26.94
StyleGAN
Sauer 2023a
512
0.10s
1
1B
13.90
~0.29
GigaGAN
Kang ei al.
2026
512
0.13s
IB
9.09
Distilled Stable Diffusion 2.1
TADD (SD-Tubb) Sauer al. 2023b
512
16.25
0.335
Distilled Stable Diffusion XL
ADD (SDXL-Turbo) Sauer
2023b
512
19.08
0.343
Stable Diffusion and accelerated distilled versions
SD1.5 (CFG=3)
Rombach al.
2022
512
2.59s
50
0.9B
8.78
(CFG5(CFG=8)
(Komiach of
A122
512
2.59g
50
0.98
13.45
0.322
74M+++4<<eep
TTi al. 20124
312
0.266
0.98
22 17.44
0.31
UniPC step)
Zhia 2023
512
0.26s
0.98
22.30
LCM-LoRA Step) [Laore
2023b
512
0.19
4
0.98
23.62
0.30
LCM-LoRA (1 step) (Tai
A1235
512
0.07s
0.98
77.90
0.24
InstaFlow-0.99 (Liu al 2025
512
0.09s
0.9B
13.10
0.28
InstaFlow-1.78 (Tu
512
UFOGen Xu
20/55
512
0.09s
0.98
12.78
DMD (CRG=3) Yin et
2023
512
0.09s
0.9B
11.49
DMD (CFG=8) Yin 2023
512
0.09%
0.98
14.93
0.32
BOOT (uu
512
0.09s
0.9B
48.20
0.26
Guided Disiiationn [Mang et al. 2023
512
0.98
37,3
0.27
SiD-LSG (x = 1.5)
512
0.09s
0.9B
8.71
0,302
SiD-LSG 1.5, double the training time)
512
0.09s
0.9B
8.15
0.304
SiD-LSG
512
0,313
SiD-LSG (x 3)
512
0.09s
0.9B
13.21
0,314
SiD-LSG (k 4.5)
512
0.09s
0.9B
16.59
0,317
Stuble Ditlusion 2. 1-bass and its distilled versions
SD2, 1-buee
RRombach et al.
2022
512
0.09s
1
0.9B
202.14
0.08
SD2.1 base
Rombach et al.
512
0.77s
25
0.9B
13.45
SwiftBrush
Nguyen
512
0.09s
0.9B
16.67
0.29
SiD-LSG (n
1.5)
512
0.09s
I
0.9B
9.52
0,308
SiD-LSG (x 2)
512
0.09s
1
0.9B
10.97
0,318
SiD-LSG
512
0.09s
0.9B
13.50
0.321
SiD-LSG 4.5)
512
0.09s
1
0.9B
16.54
0.322
scores, K4 7.5 can result CLIP score around 0.31. However, our primary objective is to devise strategy that lowers FID while minimizing CLIP degradation.
Short the CFG of the fake score network. For the short strategy, we set Ki = K4 and explore R2 #3 E {0.5, 0.125). ). As illustrated in the right panel of Figure 3 this approach delivers competitive performance, with A2 dictating the balance between FID and CLIP scores. However, compared to the long strategy, this configuration generally produces inferior results, suggested by lower CLIP scores when FIDs are controlled to similar levels.
LSG: Long and Short classifier-free Guidance. Below, explore how to effectively integrate the long and short strategies to enhance text guidance in diffusion distillation. Initially, we discovered "simplest" form of LSG strategy by amplifying the CFG during the training of fa. Specifically,
7

---

Published as a conference paper at ICLR 2025
Table 2: Comparison of HPSv2 score and Precision/Recal on the COCO-2014 validation set. The HPSv2 scores of ADD (SD-Turbo) are produced based on the publicly available model checkpoint. The HPSv2 scores of the other baselines are quoted from SwiftBrush (gguyen & Tran 2024). The Precision and Recall on COCO-2014 are obtained using the 30K images generated by the corresponding model checkpoints.
Human Preference Score v2 1
COCO-2014 Precision & Recall
Teacher
Student
Anime
Photo
(recept Alt
Phintings
Precision
Recall
SD2.1
ADD SSD Turbo) Sauer et
20236
27.46
0.65
SD1.5
LCM Lao el
A1236
22.61
22.71
22.74
22.91
SD1.5
Instaloow nw C 2023
25.98
26.32
25.79
25.93
0.53
0.45
SD1.5
B0OT Girett 2124
25.29
25.16
24.40
24.61
SD2. I-base
SwiftBrush Sggyenn Tran
26.37
0.47
SiD-LSG (K = 1.5)
26.58
26,80
26.02
26.02
0.59
0.52
SDL.5
SID-LSG 2)
SID-LGG 1.5, double the training time)
26.58
26.80
26.01
26.02
26.27
0.64
0.60
0.53
SID-LSG (x 1)
27.10
27.11
26.47
25.46
0.65
0.40
SID-L (N 4.59
27.39
27,30
26.65
26.58
0.67
0.34
SDJLSG 1.59
26.65
26.87
26.14
0.60
I-base
SID-LSG (x 2)
26.90
27.08
26.13
26:17
0.62
0.14
SID-LSG (K 3)
27.27
27.22
26.75
26.72
0.64
0.38
SID-LSG 1.5)
27.42
27.31
26.81
26.79
0.63
0.34
CUP Scores vs lteration
FID and Scores vs lberation
FID and CUP Scores Ilerat in
TD Scom
dur Scans
Eatch Lind batch 3132 H-ted
FD Score
Auttariickk Amrrrtoct Sa
387
3006
Un
300c nermtted "umber Images
314 tersted number oflmages
2000
(terabed number crimages
Figure 5: This figure illustrates the progression of FID and CLIP scores during ablation study of distilling SD1.5 using SiD-LSG, The default settings batch size 512, learning rate 1e-6, LSG scale and Prompt Aestheticsó+ are maintained unless specified otherwise. Left: The number of training fake images is doubled from 10M to 20M under LSG scales of 1.5 and 2.0. Middle: Variations in batch size and learning rate settings under LSG 1.5. Right: Comparison of training prompts Aesthetics6+, Aesthetics6.254 and 5
we set Mn to 2 or 3, while keeping K2 = K3 = R4 at 1. As illustrated in the right panel of Figure 3
setting Ki = 3 (green lines) proves effective the short strategy with x2 Kig 0.5 (red lines), and K1 = 2 (black lines) outperforms the short strategy with K2 = K33 = 0.5 (blue lines) and is comparable the long strategy with K4 (red lines in the left panel). These findings validate this "simplest" LSG as an effective guidance strategy in text-guided diffusion distillation.
Nevertheless, merely matching the performance of the best long or short strategy is insufficient to justify adopting this "simplest" LSG. The efficacy of LSG is notably enhanced when the CFG scale applied to the fake score network during its training exceeds and is maintained throughout the generator's ' training. This strategy strikes an effective balance between minimizing FID and maximizing CLIP. Specifically, for this recommended LSG configuration, we evaluated K1 = K2 K3 = K4 values within (1.5, 2, 4.5), presenting the results in Figure 4 for distilling both SD 1.5 and 2. base models. Comparisons between SD1.5 outcomes in the left panel of Figure 4 and those in Figure 3 demonstrate superior performance with this LSG setting, indicated by higher CLIP scores at controlled FID levels, and lower FID scores at controlled CLIP levels. Generally, within the range of 1.5 to 4.5, lower guidance scale correlates with better FID but worse CLIP, and vice versa, as evidenced by the curves for both SD 1.5 and 2. -base in Figure 4
3.2 ABLATION STUDY
We investigate the impact of extended training durations under two different guidance scales, varia- tions in batch size, and the selection of training prompts. Initially, we doubled the number of fake images used to train the generator from 10M to 20M and monitored the evolution of the FID and CLIP scores. From the left panel of Figure we observe continuous improvements in both FID and CLIP scores for an LSG of 1.5 when training is extended beyond 10M fake images, and sustained enhancements in CLIP scores for LSG of 2.0. Notably, by doubling the training input from 10M to 20M fake images, the FID under LSG 1.5 decreased from 8.71 1 to a record low of 8.15 among diffusion distillation methods in the data-free setting.
8

---

Published as a conference paper at ICLR 2025
We assessed the effects of batch size by considering two additional settings: batch size of 8192 with a learning rate of 4c-6, a configuration used in SiD to distill the EDM model pretrained on ImageNet 64x64 (Zhou al. 2024), and a batch size of 64 with a learning rate of 1e-6. The middle panel of Figure 5 deemonstrates that while there are initial differences in the convergence speed in terms of the number of fake images processed, the performances eventually converge similar levels. We note that while smaller batch sizes may seem to converge faster, they require more time to process the same number of images. This is due more frequent model parameter updates, higher communication costs between GPUs, and additional overheads typically associated with smaller batch sizes.
Lastly, we explored the effect of changing training prompts, shifting from the Aestheticsó+ prompts to Aesthetics6.25 25+ and Aesthetics6.55 5+ prompts (Cherti et al., (023). The performance, as shown in the right panel of Figure appears comparable across these variations. Specifically, under an LSG of 2.0, switching from Aesthetics6+ to Aesthetics6.25 25++ enabled us to further reduce the FII from the 9.56 reported in Table 9.21, although the CLIP score decreased slightly from 0.313 to 0.311, indicating no significant performance disparity between them.
The results of the ablation study show that SiD-LSG has low sensitivity to variations batch size and training prompts and its performance could potentially be further enhanced with extended training.
3.3 QUANTITATIVE AND QUALITATIVE EVALUATIONS
We present comprehensive results from prior studies across various experimental settings, including both one-step and multi-step generation methods. When evaluation results are available in existing literature (Yin et al. 2023 Liu et al. 2023 Kang et al. 2023 Nguyen & Tran 2024), we directly cite them; otherwise, if model checkpoints are accessible, either publicly or provided upon request by the authors, we utilize the evaluation code from GigaGAN to produce the reported results. For our SiD-LSG, we select K1 K2 = K3 A4 E {1.5, 2.0, 3.0, 4.5].
For comparisons of FID and CLIP scores, the results are detailed in Table [ Among all one-step distillation methods, our approach notably excels in zero-shot text-conditioned image generation on the COCO-201 dataset, as reflected by both FID-30K and CLIP scores. Specifically, with the guidance scale set as 2, our method attains FID scores as low as 9.56 and 10.97, and a CLIP score above 0.31 I and around 0.32, using SD 1.5 and 2. -bse as the pretrained backbones, respectively. Notably, by setting the guidance scale to 1.5 and doubling the training time, our method achieves a record-low data-free FID of 8.15, along with a CLIP score of 0.304, when distilling SD1.5. These results remain highly competitive when compared to other generative approaches, such as autoregressive models and GANS, and are even comparable to previous multi-step diffusion-basee sampling methods. Analyzing different is values, we observe trade-off between FID and CLIP scores: smaller x values generally yield better FID metrics, while larger values enhance CLIP scores, aligning with past findings on the impact of guidance scale.
Beyond FID and CLIP scores, we also assess Precision and Recall (Kynkäänniemi et al. 2019) as well as Human Preference Score (HPSv2) (Wu et al. 2023), which are presented in Table 2 We reuse the same 30k images from previous evaluations for the Precision and Recall calculations. For HPSv2, we follow their established protocol, generating images from 800 text prompts per category. Except for ADD trained with real data and adversarial loss, our SiD-L-SG models outperform other baselines in HPSv2 scores across all categories, as well as in Precision and Recall metrics. Notably, with K = 4.5, SiD-LSG reaches peak performance in distilling -base. Regarding Precision and Recall, higher x values lead to improved Precision, while lower values result in better Recall.
For qualitative verification, we utilize our models with K = 4.5, selecting six prompts from across all HPSv2 categories to generate images. To ensure a fair comparison, we maintain the same random seed for image generation across all methods. The visual results are illustrated in Figure 6 where SiD consistently shows superior text-image alignment and visual fidelity.
To contextualize the numerical differences in metries such FID, CLIP, and HPSv2, we present Figures to illustrate their implications for visual perception: the model with the best FID excels in diversity, while the model with the best CLIP stands out in text alignment and aesthetic quality.
For broader impact, please refer to the discussion in Appendix A For limitations and computational requirements, please refer to detailed discussion in Appendix

---

Published as a conference paper at ICLR 2025
perto hoftin geodgee goodoss necion heod boudiid oita.
some aelle the recamali tarleponntion der righit
phorowdeet nakio
dhamatic ligting ausoosedieed sestatee
Mlem foran stlnng anvernhol
daplokkng m9mm
Figure 6: Qualitative comparison of one-step distillation methods using identical text prompts and random seeds.
4 CONCLUSION AND FUTURE WORK
This paper introduces novel data-free method combining Classifier-Free Guidance (CFG) with Score identity Distillation (SiD) to efficiently distill Stable Diffusion models into effective one-step generators. By leveraging innovative Long and Short CFG strategies (LSG), distilled these models using only synthetic images generated by the one-step generator. This approach not only validates the practical potential of SiD but also sets new benchmarks for data-free one-step diffusion distillation, achieving remarkable zero-shot FID scores on the COCO-20 validation set. Our method enhances efficiency while maintaining generation performance, allowing learning from the teacher model without the need for real images or the inclusion of additional regression or adversarial losses. We will make our code and distilled models publicly available to facilitate further research.
In the data-free setting, we are exploring the use of SiD-LSG for privacy and security-sensitive tasks where access to actual training data is not feasible. While we have advanced the capabilities of data-free diffusion distillation, our baseline methods such as ADD and DMD, typically require the use of real or teacher-synthesized images. Moving forward, we plan lift the data-free constraint and integrate SiD-LSG with Diffusion GAN-based adversarial training, which has successfully transformed pretrained unconditional and label-conditional diffusion models into one-step generators, achieving state-of-the-art generation performance without CFG (Zhou et al. 2025). This transition will involve the use of real images to not only further enhance photo-realine
and improve text
alignment but also adapt SiD-LSG distilled one-step generators to domains that differ from those used to train the teacher. This approach aims to broaden the applicability and effectiveness of our SiD-L-G distilled diffusion models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
M. Zhou, Z. Wang, and H. Zheng acknowledge the support of NSF-IIS 2212418 and NIH-R37 CA271186.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Aluman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.0877 2023.
Michael S Albergo, Nicholas M Boffi, and Eric Canden-Fijnden Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.0879) 2023.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981- 17993, 2021.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming- Yu Liu. eDiff-I: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01322 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901, 2020.
Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang. Kevin Patrick Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-image generation via masked generative transformers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th In- ternational Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 4055 4075, PMLR, 23 29 Jul 2023. URL htletps:/prroceeeeeee html
mir prwss /222 h2t0/chaagg223322
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-ter -ext pre-training to recognize long-tail tail visual concepts. In CVPR, 2021.
Tianqi Chen and Mingyuan Zhou. Learning to Jump: Thinning and thickening latent counts for generative
modeling. In ICML 2023: International Conference on Machine Learning, July 2023. URL pttps 7/arxiv. abs/ / 2305. 18375
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman. Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-2829, 2023.
Wang Chong, David Blei, and Fei-Fei Li. Simultaneous image classification and annotation. In 2009 IEEE Conference on computer vision and pattern recognition, pp. 1903-1910. IEEE, 2009.
Tri Dao. Flashattention-2: Faster attention with hetter parallelism and work partitioning. arXiv preprint arXiv:2307.0880 2023.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16359, 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805. 2018.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8799 87944, 2021.
Ming Ding, Zhuoyi Yang. Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 14:19822-19885, 2021.

---

Published as a conference paper at ICLR 2025
Patrick Esser, Robin Rombach, and Bjorn Ommer: Taming transformers for tigh-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873- 12283, 2021.
Li Fei-Fei and Pietro Perona. Bayesian hierarchical model for learning natural scene categories. In 2005 JEEE computer society conference on computer vision and pattern recognition (CVPR'05), 005), volume 2. pp. 524-531. IEEE, 2005.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene' Scene-based text-to-image generation with human priors. In European Conference on Computer Vision, pp- 89-106. Springer, 2022.
lan Goodfellow, Jean Pouget-Abees Abadee, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances Neural Information Processing Systems, pp- 2672-2680, 2014.
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra. DRAW: A recurrent neural network for image generation. In International conference on machine learning, pp. 1462 1471. PMLR, 2015.
Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Lingjic Liu, and Joshua M Susskind. BOOT: Data-free distillation of denoising diffusion models with bootstrapping. ICML 2023 Workshop on Structured Probabilistic Inference and Generative Modeling, 2023.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In CVPR, 2022.
Pengeheng He, Xiandong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding-enhanced bert with disentangled attention. arXiv preprint (xXxx2006.03655. 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
G. Hinton, Osindero, and W. Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18 (7):1527-18) 1554,, 2006.
Jonathan Ho and Tim Salimans. Classificr-free diffusion guidance. arXiv preprint arXiv:220.112598, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. Advances in Neural Information Processing Systems, 33, 2020.
Jonathan Ho. William Chan, Chitwan Saharia, Jay Whang. Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02300, 2022.
Chris Holmes and Stephen G Walker. Assigning a value to a power likelihood in a general Bayesian model. Biometrika, 104(2):497-500 2017.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multino- mial diffusion: Learning categorical distributions. Advances in Neural Information Processing Syxtems, 34: 12454 12465, 2021.
Minghui Hu. Yujie Wang, Tat-Jen Cham, Jianfei Yang, and Ponnuthurai N Suganthan. Global context with discrete difflusion in vector quantised modelling for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11502- 11511, 2022.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman. Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. July 2021. URLhhttps:///oo wrg (tt.:.52811222ee. 5143773 If you use this software, please cite it below.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B. arXiv preprint arXiv:23 /0. 00.0625. 2023.
Minguk Kang, Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up GANs for text-to-image synthesis. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10124-10134, 2023.
12

---

Published as a conference paper at ICLR 2025
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401 -4100, 2019.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of liffusion-based generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.). Advances in Neural Information Processing Systems, 2022. URL nttps://openzev// /FFTTONMDe
net/ horum? id=
Dongjun Kim, Chich-Hsin Lai, Wei- Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arX1:233/0.0227, 2023.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014,
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019.
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xFormers: A modular and hackable Transformer modelling library, https
//github. com wttp:///cccccooecccrcccccccc httpchhh/ooomrrereeeeooooooo 2022.
Yaron Lipman, Ricky TQ Chen, Heli Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.0274 2022.
Luping Liu, Ren, Zhijie Lin, and Zhou Zhao. Pscudo numerical methods for difflusion models on manifolds. In International Conference on Learning Representations, 2022a. URL httl:ttps::/opeeceelleeeee n
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint urXiv:2209.0300 2022b.
Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, and Qiang Liu. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. arXiv preprint arXiv:2309.06380 2023.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen. Chongxuan Li, and Jun Zhu, DPM-Solver++: Fast solver for guided sampling of diffusion probabilistic models. In arXiv preprint arXiv:2211.01095, 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022b. URI. https: //cppsr///ppereeeee eeeee. net. www.:eet.ooomrmm.o forum? dd- 3-2uAaGw1l
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02:5 2021.
Simian Luo, Yiqin Tan, Longho Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. ArXiv, 1b./233,,04378, 2023a.
Simian Luo, Yigin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao. Lem-lora: A universal 2023b.
acceleration module. arXiv preprint 11X0v22310004378
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff-Instruct: universal approach for transferring knowledge from pre-trained diffuuioo models. In Seinty Conference on Neural Information Processing Systems, 2023c. URL
ML FILI51RRgh 51g4w
Siwei Lyu. Interpretation and generalization of score matching. In Proceedings the the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, pp. 359-366, 2009.
Zhaoyang Lyu, Xudong Xu, Ceyuan Yang. Dahua Lin, and Bo Dai. Accelerating diffuusio models via early stop of the diffusion process. arXiv preprint arXiv: 220.2224,, 2022.
Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Generating images from captions with attention. arXiv preprint 151 2015.
13

---

Published as a conference paper at ICLR 2025
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma. Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14396, 2023.
Shakir Mohamed Balaji Lakssminarayanan. Learning in implicit generative models, arXiv preprint arXiv: 161000431 2016.
Thuan Hoang Nguyen and Anh Tran. SwiftBrush: One-step lext-to-image diffussin model with variational score distillation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference Machine Learning, pp. 16784-16804, PMLR, 2022.
Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar. Diffuse Efficient, controllable and high-fidelity generation from low-dimensional latents. arXiv preprint arXiv:2201 003008, 2022.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanam. Normalizing flows for probabilistic modeling and inference. arXiv preprint arXiv: 1912.0272, 2019.
William Peebles and Saining Xie. Scalable diffusion models with transformers. Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024. URL htttps://openteeeee (ttffrrmmmm--11122288k
G. Polatkan, M. Zhou, L. Carin, D. Blei, and I. Daubechies. A Bayesian nonparametric approach image super-resolution. PAMI, 37633366-338, 2015.
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D diffusion. ArXiv, abs/2209. 14988, 2022.
Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingho Zhou, Huan Wang. Juan Carlos Niehles, Caiming Xiong, Silvio Savarese, et al. Unicontrol: A unified diffusion model for controllable visual generation in the wild, arXiv preprint orXiv:2305. // /147, 2023.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre- vre-training. 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yangi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer lcarning with a unified text-to-text transformer. Journal of machine learning research, 21 111140):1667, 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Iyya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp- 9821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents, arXiv preprint arXiv:2204.06125, 2022.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International conference on machine learning, pp. 1060-1069. PMLR, 2016.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st Intermational Conference Machine Learning, pp. 1278 1286, 2014.
Jason Tyler Rolfe Discrete variational autoencoders. arXiv preprint 16Xiv:1109900000.. 2016.
14

---

Published as a conference paper at ICLR 2025
Robin Rombach, Andreas Blatmmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-1060 10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Chasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 55:36479-36499 2022.
Ruslan Salakhutdinov and Geoffrey Hinton. Deep Boltzmann machines. In Artificial intelligence and statistics. pp. 448 455. PMLR, 2009.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In Interna- tional Conference on Learning Representations. 2022. URL https::/ normEorum? id
IIdIXIpzhoI
Axel Sauer, Katja Schwarz, and Andreas Geiger. StyleGAN-XL: Sealing StyleGAN large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022.
Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. StyleGAN-T: Unlocking the power of GANs for fast large-scale text-to-image synthesis. arXiv preprint 23ti::2300.095155 2023a.
Axel Sauer, Dominik Lorenz, A. Blattmann, and Robin Rombach. Adversarial diffusion distillation. ArXiv, abs/2311. 170022 2023b.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION 5B: open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25578-25294, 2022.
Jascha Schl-Dickster Eric Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning, pp. 2266-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. International
Conference on Learning Representations, 2021a. URL http:p/////peeeeeeeeeeeeee////oe
StigiarCHLP
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint arXiv:2310.14100 2023.
Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. In Advances Neural Information Processing Systems, pp. 11918-1930, 2019.
Yang Song and Stefano Ermon. Improved Techniques for Training Score- Based Generative Models, Advances in Neural Information Processing Systems, 33, 2020.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-bascd based diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), 2021b.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2021c. URL https::p: /twpe/aee///eeeviviiieeee
22RHS
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01 230.001639 2023.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: The new data in multimedia research. Communications of the ACM, (942)-7773 2016.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasminc Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 2rXiv:2307.09288, 2023.
Dustin Tran, Rajesh Ranganath, and David Blei. Hierarchical implicit models and likelihood-free variational inference. In Advances in Neural Information Processing Systems, pp. 5523-5533, 2017.
15

---

Published as a conference paper at ICLR 2025
Aaron den Oord, Oriol Vinyals. and Koray Kayukcuoglu. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Eukasz. Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research, (1())). 2010.
Zhendong Wang, Yifan Jiang, Yadong Lu, Pengcheng He, Weizhu Chen, Zhangyang Wang. Mingyuan Zhou, et al. In-context learning unlocked for diffusion models, Advances in Neural Information Processing Systems, 36:8542-8562, 2023a.
Zhendong Wang, Huangjic Zheng, Pengeheng Hc, Weizhu Chen, and Mingyuan Zhou. Diffusion-GAN: Gan::rainnng GANs with diffusion. In The Eleventh International Conference on Learning Representations, 2023b. URL
https://cpenzevveew net/ wwwes//formmiddee ddd
Zhengyi Wang. Cheng Lu. Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. ProlificDreamer: High-fidelity and diverse text-to-3D generation with variational score distillation, 2023c.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-lmage Synthesis. arXiv preprint arXtv:2306.09944 2023.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denois- ing diffusion GANs. In International Conference on Learning Representations, 2022. URL https
//apenreview wwwt/eeetttnnnnee http://rrrunnnaa--oorro-oa--0a- goCo
Zhengming Xing, Mingyuan Zhou, Alexey Castrodad, Guillermo Sapiro, and Lawrence Carin. Dictionary learning for noisy and incomplete hyperspectral images. SIAM Journal on Imaging Sciences, 5(1):33-56. 2012.
Tao Xu, Pengehuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. AunGAN: sine-grained text image generation with attentional generative adversarial networks. CVPR, pp. 1316- 1324, 2018
Xinggian Xu, Zhangyang Wang, Eric Zhang, Kai Wang. and Humphrey Shi. Versatile diffusion: Text, images and variations all in one difffusion model. arXiv preprint arXiv:2211.08332, 2022.
Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingho Hou. UFOGen: You forward once large scale text-to-image generation via diffusion GANs. ArXiv, abs/2311.099 2023.
Yue Yang, Ryan Martin, and Howard Bondell, Variational approximations using Fisher divergence. arXiv preprint (005:v:905.00222 2019.
Mingzhang Yin and Mingyuan Zhou, Semi-implicit variational inference. In International Conference on Machine Learning, pp. 5660-5669 5669, 2018.
Tianwei Yin, Michael Gharbi. Richard Zhang, Eli Shechtman, Frédo Durand, William T. Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. ArXiv, abs/2311.18822 2023.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. Transactions on Machine Learning Research, 2022.
Longlin Yu and Cheng Zhang. Semi-implicit variational inference via score matching, In The Eleventh Interna- tional Conference on Learning Representations, 2023. URI. httpttts:/:/opeeeriiii httprrewwwwweeeeeeeeeeeeeeeeeeee fttts/frrmm?icc
sd90a2ytrt
Longlin Yu, Tianyu Xie, Yu Zhu, Tong Yang, Xiangyu Zhang, and Cheng Zhang. Hierarchical semi-implicit variational inference with application to diffusion model acceleration. Thirry-seventl Conference on Neural Information Processing Systems, 2023.
net www.moorrndddoo Forum??d-gghiBppssss
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. StackGAN: Text to reato-realistic image synthesis with stacked generative adversarial networks. In Proceed- ings of the IEFE international conference on computer vision, pp. 59077-991,, 2017.
16

---

Published as a conference paper at ICLR 2025
Han Zhang. Weichong Yin, Yewei Fang, Lanxin Li, Boqiang Duan, Zhihua Wu, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE-ViLG: Unified generative pre-training for bidirectional vision-language generation. arXiv preprint XrXiv:2112.15283, 2021.
Hao Zhang, Chen, Long Tian, Zhengjue Wang. and Mingyuan Zhou. Variational hetero-encoder randomized gans for joint image-text modeling. International Conference on Learning Representations, 2020. URL.
https::/
net forum? d--H X5WRVtVS
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In The
Eleventh International Conference on Learning Representations, 2023. URL https:://pennewiicc
net ner/forum? 2 oek77h4444
Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: A unified predictor-correctoor framework for fast sampling of diffusion models. arXiv preprint arXiv:2302. 04867, 2023.
Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator lcarning. In International Conference Machine Learning, pp. 42390-44240.. PMLR, 2023a.
Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusion probabilistic models and diffusion-based adversarial auto-encoders. In The Eleventh International Conference on Learning Representations, 2023b.
net/ forum ? HDD.umikiddD
Huangjic Zheng, Zhendong Wang, Jianbo Yuan, Guanghan Ning, Pengeheng Hc, Quanzeng You, Hongxia Yang, and Mingyuan Zhou. Learning stackable and skippable LEGO bricks for efficient, reconfiguralle, and variable-resolution difflusion modeling. In The Twelfth International Conference on Learning Representations, 2024.
net forum? d=qgXedvwr
Mingyuan Zhou, Haojun Chen, John Paisley, Lu Ren, Guillermo Sapiro, and Lawrence Carin. Non-parametric Bayesian dictionary learning for sparse image representations. Advances in neural information processing systems, 22, 2009.
Mingyuan Zhou, Tianqi Chen, Zhendong Wang, and Huangjie Zheng. Beta diffusion. In Neural Information Processing Systems, 2023. URL httts:/t/://// wwtp///aavvvvv..v///a//eae www.iv.orrrgaeooo abs 23009 07867
Mingyuan Zhou, Huangjie Zheng, Zhendong Wang. Mingzhang Yin, and Hai Huang. Score identity distilla- tion: Exponentially fast distillation of pretrained diffusion models for one-step generation. In Forty-first International Conference on Machine Learning, 2024. URL htttps://openrevieee oop.
net/forumm
QhqQJgeURg
Mingyuan Zhou, Huangjie Zheng, Yi Gu, Zhendong Wang, and Hai Huang, Adversarial score identity distillation: Rapidly surpassing the teacher in one step. In International Conference on Learning Representations, 2025.
Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui and Tong Sun. Towards language-fiee free training for text eet-to-imne -m-agee-maae generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17907- 17917, 2022.
17

---

Published as a conference paper at ICLR 2025
Guided Score Identity Distillation for Data-Free One-Step Text-to-lmage Generation: Appendix
SD1.5 distilled with SiD-LSG (N 1.5): FID 8.15, CLIP 0.304
distilled with SiD-LSG (k 4.5): FID = 16.54, CLIP 0.322
Figure 7: Visual comparison of two SiD-LSG models: preferred for FID and the other for CLIP. All images are generated from the same text prompt: "A distinguished older gentleman in a vintage study, surrounded by books and dim lighting, his face marked by wisdom and time. 8K, hyper-realistic, cinematic, post production. The model with lower guidance scale of x = 1.5, which achieves record-low mee-ste--generate a competitive CLIP score of 0.304, produces images are more diverse align less closely text details, such as "dim lighting." Conversely, the model with higher guidance scale of x = 4.5, achieving high CLIP score 0.322 noted state-of-the human preference scores (HPSv2) as shown Table presents relatively high FID of 16.54, indicating less diversity but superior text alignment and visual quality.
FID of 15 specific
SD1.5 distilled with SiD-LSG (x 1.5): FID 8.15, CLIP 0.304
SD2.1-bart distilled with SiD-LSG (N 4.5): FID 16.54, CLIP 0.322
Figure 8: Analogous Figure 7 this plot compares two of images generated using two distinct guidance scales, both conditioning on the same text prompt: "poster art collection the asian woman, in style of gloomy, dark orange and white, dynamic anime, realistic watercolors, nintencore, weathercore, mysterious realism -ar 19:128-00 -s 750 niji 5".
A BROADER IMPACT
The broader impact of our work is multifaceted. On hand, it significantly reduces the energy required to operate state-of-thee the-thh-rt diffusion models, contributing to more sustainable On the other hand, the ease of distilling and deploying models that might be trained data with ques-
Al practices.
18

---

Published as a conference paper at ICLR 2025
tionable content intentions presents ethical challenges. Therefore, it is erucial for the community to engage in discussions on how to minimize risks while enhancing the benefits of such advancements. This involves developing robust guidelines and frameworks to govern the use and deployment of distilled models, ensuring they are used responsibly and ethically.
B RELATED WORK
Generative modeling of high-dimensional data has long been a focal point in machine learning research. This area primarily concentrates on replicating various data distributions: the original data distribution, conditional distributions influenced by labels, noisy or incomplete measurements, textual descriptions, or the joint distribution of data and other modalities. This has spurred the development of a diverse range of generative models and methodologies. Initially, these models were only capable of handling simpler. low-dimensional data such as 28 28 grayscale binarized MNIST digits (Hinton et al., |0066] Salakhutdinov & Hinton| 2009| Vincent et al., (010)) vector-quantized local descriptors (Fei-Fei & Peronal 2005 Chong et al. 2009). patches of natural and hyperspectral images (Zhou et al. 2009 Xing et al.
et al. 2013). Early models often utilized neural networks with stochastic binary hidden layers or shallow hierarchical Bayesian models. which are simpler to train but have limited capacities.
Deep generative models. To tackle the generation of high-dimensional data, such images comprising millions of pixels, substantial advancements in generative models have been made over the past decade. This period has marked the emergence of diverse deep generative models, including variational auto-encoders (VAEs) (Kingma & Welling, 2014] Rezende et al., 2014), normalizing
flows (Papamakarios et al. 2201)), generative adversarial networks (GANs) (Goodfellow et al.] 20144
Reed 2016; Karras et al. 2019; Wang al. 2023b), autoregressive models (Gregor et al.
2015 Mansimov et al., 2015). and diffusion models (Sohl-Dickstein et al. 2015 Song & Ermon
2019 Ho et al. |020 Song & Ermon 20200] Song et | Dhariwal & Nichol 2021 Karras
jet al.
2022 Peebles & [ee]2023]] 2 Zheng et al. 2024). Additionally, essential resources for creating
effective T21 synthesis systems, such large language models (Vaswani et al.] [017: Devlin et al.
2018;
Radford et al. 2018: 2019 Raffel et al. 2020 He et al. 2020; Brown et al. ||200 Achiam
al.
2023| Touvron al.
2023 Jiang al. 2023). large vision-language models (Radford et al.
2021
Ilharco et al.] [021] Cherti et al. 2023). 3) ad advanced visual tokenization and compression (Rolfe
2016
van den Oord al., [017]] Esser
Esserr et al.
Rombach et al.) 2022 Podell et al.] (2024). and
extensive training datasets (Thomee et al. 201 66 Changpinyo et al., 2021 Schuhmann et al. 2022).
have transitioned from proprietary tools of major tech companies to publicly available assets.
Previously, T2I models primarily utilized GANs and focused on small-scale, object-centric domains like flowers, birds, and face images (Reed et al., 2016: Zhang et al., 2017 Xu et al.] [2088] Zhang et al. 2020). Powered by text encoders that leverage pretrained large language models or vision- language models, which offer profound language comprehension and extract semantically rich latent representations, and supported by an extensive collection of text-image pairs, three main families of
generative models- GANs (Zhou et al.] [022 Sauer et al., 2022; Kang et al. 2023). autoregressive
models (Ramesh et al. 2021; Zhhng et al. 2021: Ding et al. 2021 Gafni et al. 2022 Yu al.
2022 Chang et al. |l. (023)) and diffusion models (Nichol et al. 2022 Ramesh et al., 2022 Saharia
al. 2022 Rombach et al. 2022 Xu et al. 2022, Wang et al. [023a] Qin et al. (0023 have
effectively capitalized on these technological advancements. These developments have facilitated the creation T2I synthesis systems that demonstrate exceptional photorealism and sophisticated language understanding.
Acceleration of cultusion-based generation. Etrly-stage diffusion models were notably slow in sampling. spurring extensive research aimed at speeding up the reverse diffusion process. Researchers have approached this by interpreting diffusion models through stochastic or ordinary differential equations and using advanced numerical solvers to enhance efficiency (Song et al., 2021c Liu et al.
2022al Lu et al. 2022b Zhang & Chen 2023 Karras et al. 2022). Additional strategies include truncating the diffusion chain to initiate generation from more structured distributions (Pandey et al.
Lyu et al.) 2022). integrating these models with GANs to boost generation 2022 Wang et al. (023b)), and exploring flow matching in diffusion modeling Lipman et al. 2022 Albergo et al., 2023). More recently, research has pivoted towards distilling reverse diffusion chains refine and expedite the generation process, direction
2022 Zheng et al.
2023b
speed (Xiao al.
(Liu et al. 2022b
19

---

Published as a conference paper at ICLR 2025
that continues evolve with new methodologies and insights (uhhman & Luhman 2021
Salimans
Ho 2022 Zheng et al., 2023at Meng et al. 2023] Song et al.
2023 Song & Dhariwal 2023) Kim
al.
2023 Sauer et al. 2023b Xu et al. 2023 Yin et al. 2023 Luo et al. 2023c Zhou et al. 2024).
T21 diffusion distillation. Recent efforts aim to accelerate the sampling process from pre-trained diffusion teachers like SD (Rombach et al., 2022). Sauer et al. |l.] (2023b) focused on distilling diffusion models into generators capable of one or two-step operations through adversarial training. Xu et al.
(2023) introduced UFOGen, utilizing a time-step-dependent discriminator for generator initialization. Luo et al. (2023a) applied consistency distillation (Song et al., (023) to text-guided latent diffusion models (Ramesh et al. (022) for efficient, high-fidelity T2I generation. Building on the idea of
using a pre-trained 2D T2I diffusion model for text-to-3D synthesis (Poole et al. 2022] Wang et al.) 2023c), SwiftBrush (Nguyen & Tran 2024) showcases its effectiveness distilling pre-trained stable
diffusion models. Distribution Matching Distillation (DMD) by Yin et al. (2023) further enhances distillation quality by adding a regression loss term.
How does SiD-LSG differ from previous methods? SiD-LSG introduces several unique features that distinguish from previous T2I diffusion distillation methods. Firstly, similar to SrihtBrush (Nguyen & Tran 2024), SiD-LSG is data-free distillation method, which eliminates the need for original training datasets or synthetic data generated with SDE/ODE solvers during the distillation process. However, unlike SwiftBrush, SiD-LSG necessitates gradient backpropagation through the score networks, a critical step akin to gradient backpropagation through the discriminator in methods employing adversarial losses. Additionally, SiD-LSG applies CFG both the training and evaluation of the fake score network. This is a departure from previous methods, which typically apply CFG only during the evaluation of the pretrained score network. Finally, SiD-LSG aims minimize model-based explicit score-matching loss, a type of Fisher divergence. In contrast, previous methods often focus on minimizing losses based on KL divergence, consistency, regression, GAN-based adversarial tactics, or combination thereof.
C FURTHER DISCUSSION ON LONG, SHORT, AND LONG-SHORT GUIDANCE
Regarding the relationship between the choice of K2 = K3 and the categorization into long, short, and long-short guidance, we clarify that various combinations of K1,2,3,1 can define these strategies:
This strategy involves enhancing the teacher's CFG more than the fake score network during inference making them equal but both larger than one. It is typically represented by K1 = K2 Ki = 1, K4 > 1, but also includes configurations where K1 = f62 K3 = K4 > 1.
Long Strategy:
Short Strategy: This approach aims to reduce the fake- www.-eeereeeoorttee CFG during evaluation compared to training or maintain them equal but greater than one. This strategy is exemplified by
6i K4 1,0 K2 = Kg 1, and can also be part of K1 R2 K3
K4 >> 1.
Long-Short Strategy: The "simplest" LSG configuration K1 > 1, = K3 = K4 = can be interpreted as incorporating both long and short strategies since increasing K1 for training the fake score network effectively implies that during inference, the fake score network is guided by a weaker CFG than both the teacher and its own training setting.
SiD-LSG employs default setting where Ki = K2 = Kg
= = 1. This implies that while the teacher's training CFG is and evaluation CFG is x > 1, the fake score network's 's training and evaluation CFG are both >> 1, making the teacher's s CFG during inference stronger than that of the fake score network.
FID-CLIP Compromise: We evaluate various CFG strategies based on the best FID achieved with fewer than 2.56M fake images used distill generators on Stable Diffusion 1.5, alongside their corresponding CLIP scores. The results, whose trajectories are depicted in Figures 3 and 4 and detailed in Table and Figure indicate that the proposed LSG strategy achieves the best balance between lowering FID and enhancing CLIP scores.
D TRAINING AND EVALUATION DETAILS
The hyperparameters tailored for our study are outlined in Table 4 It's important to note that the time and memory costs reported in Table 4 not account for those incurred during periodic evaluations
20

---

Published as a conference paper at ICLR 2025
Table Comparison of different CFG strategies in terms the best FID achieved iterating with fewer
than 2.56M fake images to distill the generators on Stable Diffusion 1.5 and their corresponding CLIP scores.
CFG Strategy
R1
K2
Rig
R4
CL.IP
10.01
0.297
2.5
10.98
0.303
Long CFG
11.75
0,303
3.5
12.82
0.304
7.5
15.78
0.308
No CFG
15.49
0.269
0.5
0.5
11.09
0.299
Short CFG
0.125
0.125
15.54
0.305
11.76
0.306
"Simplest" LSG
16.38
0.304
1.5
1.5
10.66
0.295
10.44
0.304
LSG
3
3
13.88
0,310
4.5
4.5
4.5
4.5
16.69
0.310
CLIP Score FID by CFG Strategy
5.518
2328
0.220
22.2
C-B.LAM
0.932
K- 65,11
0228
devtagy
long cno
2,1.21
Shart ahd
Figure 9: Comparison of different CFG strategies in of the best FID achieved when iterating with fewer than 2.56M images to distill the generators on Stable Diffusion 1.5 their corresponding CLIPP scores. The markers represent different CFG strategies: Long CFG denoted by red squares, Short CFG by blue circles, the "simplest" LSG by black diamonds, and LSG by orange stars.
of the FID and CLIP scores of the single-step generator, nor do they include the resources used to save model checkpoints during the distillation process. These costs can vary significantly depending on the computing resources used, including the versions of CUDA and Flash Attention, as well as the storage platforms employed and the frequency of operations.
E LIMITATIONS AND COMPUTATIONAL REQUIREMENTS
Memory and speed. Table in Appendix D offers a detailed examination of the computational resources required for the SiD distillation process employing various CFG strateiie-LLog Short, "simplest" Short, and the recommended Long- Short-hhrrrree different NVIDIA GPU platforms (RTX-A5000 with 24GB, RTX- A6000 with 48GB, and H100 with 80GB). Key observations include:
SiD-LSG can be effectively operated the RTX-A5000, which has 24GB of memory, by enabling xFormers (Lefaudeux et al. 2022) and switching to FP16 for model and gradient precision. On the RTX-A6000 with 48GB of memory, it runs using FP32 with xFormers enabled. However, our available H100 with 80GB of memory, where xFormers were not supported at the time of our experiments, there noticeable increase in GPU memory consumption.
21

---

Published as a conference paper at ICLR 2025
Algorithm 1 SiD-LSG: Score identity Distillation with Long- ShortShortaaa classifier-fres Guidance
Iuput: Pretrained network fe- generator Go. fake score network fv- fini 625, friin Ki = Ri
20, fmax 979, guidance scales
= 1.5 Initialization 4- 9, @b repeat
Sample x ~ (0 1) and c, replacing with the embedding of an empty text string 10% of the time; Define 2 = (o,
(f R, fiik, e); Sample E (Amin..... fiaa) and (O, I), and let ats ardy + d16:
Update A with v 10 v Lus, where
c.
o I1u (ate, 1, c) se, [ ||Ea. (2t, r, eelli
Sample ~ (V I) and and lot my Go(z,c) fo c): Sample (fmine bmnux and (0 I).
compute ory with D. and let ae = arstg + ozer
Update Gp with 0 VV Lo. where
Co -itin?
(Soins (20, (, - fw (2i, (Soins
(2i, (eor) xx,)
of
(E mg (#t, es,n, (2,, o)) eving (2),
until processing 10M fake images or the training hudget is exhausted Output: Go
The recommended LSG strategy, which provides an improved balance between FID and CLIP scores, demands approximately 20% more computation time per iteration and exhibits about 10% more peak memory usage on the H100 compared to the Long or Short strategies. This underlines a trade-off between achieving performance enhancements and managing resource utilization.
FP16 versus FP32. SiD-LSG can be trained under FP16 mixed precision, which significantly conserves memory and enhances processing speed, as detailed in Table 4 Although this reduced precision in optimization leads to quicker improvements in both FID and CLIP scores, it also restricts the potential for achieving the lowest FID and highest CLIP scores compared results under FP32. These effects are demonstrated in the ablation study shown in Figure 10 Additionally, FP16 operation is less stable and requires the use of gradient clipping to maintain training stability. For instance, we employ torch. nn. ut ilss s. atiis.ciipp wewip_grraareeeeeeeeeeeeeeeeeeee grad_ value_ (G. (G.parate parameters (), 1) to prevent sudden model divergence precaution that is not necessary with FP32.
Further investigation is needed to optimize FP16 performance to match that of FP32. This may involve refining loss scaling techniques, updating packages like Flash Attention or XFormers, or implementing effective warmup period with FP 16 before transitioning to FP32. We plan to address these challenges in our future studics.
We note that our initial experimental platform did not provide proper support for FlashAttention (Dao et al. 2022 Dao 2023), which adversely affected our FP16 results. Now that we have established proper support for AlashAttenioon, SiD-L.SG, especially to determine if it can match FP32's performance at a lower cost. We have updated Table displayed above to reflect significant memory reductions and noticeable acceleration in iteration speed under FP16, facilitated by the availability of FlashAttention. This enhancement would enable us use larger batch sizes per GPU under FP16, further improving time efficiency.
are keen to further explore the potential of FP16 in distilling
Reaching a performance plateau. Zhou et al. (2024) demonstrate through extensive comparisons that the SiD distilled one-step generator can reduce the FID at exponential decay rate, showing log-log linear relationship between the number of iterations and FID. This approach can match or even surpass the performance of both unconditional and label-conditional teacher diffusion models trained under the EDM framework (Karras et al.) 2022), provided sufficient training.
However, results shown in Figures and 5 indicate that the SiD-LSG distilled one-step generator on SD1.5 quickly reaches a performance plateau in reducing FID and/or increasing CLIP scores, particularly when the LSG scale exceeds 2. Additionally, data from Table I demonstrate that after processing 10M images (approximately 20k iterations at batch size of 512), SiD-LSG still does not match the text-image alignment performance of the teacher model, which achieves higher CLIP scores after 50 generation steps. Notably, by reducing the LSG scale 1.5 and doubling the training duration, SiD-LSG achieves a record-low data-fire FID of 8.15, establishing a new benchmark among all data-free diffusion distillation methods. This achievement also surpasses the teacher model's FII)
22

---

Published as a conference paper at ICLR 2025
Table 4: Hyperparameter settings and comparison of distillation time memory usage between different long and short guidance strategies of SiD, Note in order to run the long-shor guidance (LSG) with Ki K2 Ki K4 > 1. we need to turn off the EMA network for some cases (indicated with "no EMA"), otherwise it will be out of memory.
Compuring platfonn
Hyperpanimcters
Long Stratagy
Short Semiegy
Long-Short
Long-Short
CFG scalls
Learning rate
1e-6
General Settings
Half-life of EMA
S0k images
Optimizer under FP32
Adam
Pg 0.000, 16-8)
Optimizer
0.999,
1c
1.0
Time parameteoss
(ueim Suii Joon
(20, 626, 079)
xFomeers available and enabled
Yos
Batch
RTX-A5000 (21G), FP16
Max memory in GB allocated
gradient accumulation round
22.9
22.9
22.6
22.0 (co EMA)
Max memocy in GB reserved
25.0
23.0
23.0
22.1 EMA)
Time seconds per images
75
Time in hours per IM images
21
22
29
xFoomers available and enabled
Yes
Batch
GPU
* GPUs
RTX-A600 (48G), FP32
of gradient accomolation round
Max memory in GB allocated
45.7
45.7
45.0
45.8 (no EMA)
Max memoxy in GB reserved
45.7
45.7
45.9
46.0 (no EMA)
Time seconds per images
365
Time in hours per IM images
102
102
xFomers ovailable and enabled
No
Batch size per GPU
of GPUs
gradient accumulation round Max, memory
H100 (80C)), FP16
allocated
55.8
55.8
47.7
Max memory in GB reserved
57.3
55.5
49.2
65.4
Time seconds per Ik
17
17
18
per immges
Flash Atlention avaiable and embbled
Yos
Batch size per GPU
of GPUs
gradient accumolation round
(800)
memory
allocated
32.3
32.2
Max memory in GB reserved
32.6
32.4
29.6
35.4
Time in seconds per Ik images
12
12
12
15
Time hours per IM images
3
available
Batch size per GPU
of GPUs
HI00 (80C)), FP32
of gradient accumulation round allocated Max memory
58.9
57.4
53.4
memory
reserved
60.0
59.0
Time in seconds per 1. images Time in hours IM images
74
74
76
90
21
21
21
25
of 8.78, which was obtained with CFG scale of and 50 generation steps. However, this reduction in the LSG scale to 1.5 also results in a significant decline in its CLIP score.
This suggests sigmificant potential for further performance enhancements, possibly by extending beyond single-step generation, increasing the model size, or incorporating real data and additional regression or adversarial losses into the distillation process. These avenues for improvement will be one of the focuses of future studies.
Additional ablation study of the guidance strategies. At the beginning of our research, we evaluated various combinations of k1,2,3,4, quickly discarding those that converged too slowly or diverged. Specifically, the setting K = 1, Ki = Ai > 1, and K4 E (12, 1} was eliminated early due to its suboptimal performance, as depicted in Figure
F PROMPTS AND ADDITIONAL EXAMPLE IMAGES
We list the prompts used in Figure I as follows:
1. A distinguished older gentleman in a vintage study, surrounded by books and dim lighting, his face marked by wisdom and time. 8K, hyper-realistic, cinematic, post-production.
23

---

Published as a conference paper at ICLR 2025
FID and CLIP Scores Iteration
0.30
10
RID Score
Score
3, fp16
3, fp32
0.20
0.15
2.5M
Iterated number of images
FID and CLIP Scores Iteration
0.30
104
FID Score
CLIP Score
0.5,
0.5. fp32
0.20
0.15
2.5M
Iterated number of images
FID and CLIP Scores Iteration
0.30
10a
FID Score
Score
2, fp32
0.20
0.15
2.5M
Iterated number of images
Figure 10: Comparison between FP16 and FP32 under three different guidance strategies. Top: Long strategy with Kı K2 Ki and K4 = 3. Middle: Short strategy with Ki R4 1 and Kig Ki 0.5. Bottom:
short guidance (LSG) with R2 Ra R4 =
2. saharian landscape at sunset, 4k ultra realism, BY Anton Gorlin, trending on artstation, sharp focus, studio photo, intricate details, highly detailed, by greg rutkowski.
3. chinese red blouse, in the style of dreamy and romantic compositions, floral explosions 24:37 -tyliie 750 -v
4. Digital 2D, Miyazaki's style, ultimate detailed, tiny finnest details, futuristic, sci-fi, magical dreamy landscape scenery, small cute girl living alone with plushified friendly big tanuki in the gigantism of wilderness, intricate round futuristic simple multilayered architecture, habitation cabin in the trees, dramatic soft lightning, rule of thirds, cinematic.
24

---

Published as a conference paper at ICLR 2025
FID and CLIP Scores vs Iteration
0.30
0.25
FID Score
10a
A.2.2.11..1...
-0200
CLIP Score
2,2.3.=1,3,333
K12.3,41,,,33
A1.23,4 3,3,3,,
0.15
0.10
0.05
10a 104
10a
Iterated number of images
Figure 11: This figure shows the progression of FID and CLIP scores during an ablation study of distilling SD1.5 using SiD-LSG, featuring four different of K1,2,3.4 values. Settings that converged too slowly were terminated carly.
5. poster for the collection of the asian woman, the style of gloomy, dark orange and white, dynamic anime, realistic watercolors, nintencore, weathercore, mysterious realism -ar 69:128 -s 750 -niji
6. A fantasy-themed portrait female elf with golden hair and violet eyes, her attire shimmering with iridescent colors, set in an enchanted forest. 8K, best quality, fine details.
7. 'very beautiful girl in bright leggings, white short top, charismatic personality, professional photo, style of jessica drossin, super realistic photo, hyper detail, great attention skin and eyes, professional photo.
8. (steampunk atmosphere, a stunning girl with mecha musume aesthetic, adorned in intricate cyber gogle.) digital art, fractal, 32k UHD high resolution, highres, professional photogra- phy. intricate details, masterpiece, perfect anatomy, cinematic angle cinematic lighting, (dynamic warrior 1)
9. (Pirate ship sailing into bioluminescence sea with galaxy in the sky), epic, 4k, ultra.
10. tshirt design, colourful, background, yoda with sun glasses, dancing festival, ink splash, 8k.
We list the prompts used Figure which are taken from the COCO-2014 validation set, as follows:
1. many cars stuck in traffic on a high way
2. old blue with surfboard top', 3. 'a sole person sits in the front pew of large church.
3. a shot of the hollywood sign at santa monica blvd.
4. a bunch of flowers, front of a forest.
5. there some sort of vegetables bowl
6. man in pink shirt stands staring against green wall.
7. small girl in green shirt holding slice of pizza her face
8. dogs sitting in the back seat of car looking out the windwo
We list the prompts used in Figure 6 follows:
Half-length head portrait of the goddess of autumn with wheat cars on her head, depicted as dreamy and beautiful, by wlop.
25

---

Published as a conference paper at ICLR 2025
2. Walter White dressed medieval-style king.
3. A serene meadow with a tree, river, bridge, and mountains in the background under slightly overcast sunrise sky.
4. A closeup portrait of gray owl with spreaded wings attacking in cinematic lighting, digital painting by Greg Rutkowski used as album cover art on Artstation.
5. A hyena fursona sitting in the grass in a savannah at sunset.
6. A puppy staring through a red sectioned window.
We list the prompts used in Figure 12 as follows:
1. A fantasy-themed portrait of female elf with golden hair and violet eyes, her attire shimmering with iridescent colors, set in an enchanted forest. 8K, best quality, fine details.
2. pumpkins, autumn sunset in the old village, cobblestone houses, streets, plants, flowers, entrance, realistic, stunningly beautiful
3. "Highly detailed mysterious egyptian (sphynx cat), skindentation: 1.2, bright eyes, ancient egypt pyramid background, photorealistic, 1typer-realistic11.2 cinematic, masterpicce:1.1, cinematic lighting"
4. "vw bus, canvas art, abstract art printing, in the style of brian mashburn, light red and light brown, theo prins, charming character illustrations, pierre pellegrini, vintage cut-and-paste rusty debris -ar 73:92 -stylize 750 -v 6"
5. painterly style, seductive female League of legends Jinx character fighting at war, raging, crazy smile, crazy eyes, rocket lancher, guns, crazy face expression, character design, body is adorned with glowing golden runes, intense green aura around her, body dynamic epic action pose, intricate, highly detailed, epic and dynamic composition, dynamic angle, intricate details, multicolor explosion, blur effect, sharp focus, uhd, hdr, colorful shot, stormy weather, tons of flying debris around her, dark city background, modifier-CamageStyyl.. color=blood red, intensity=1.6
6. A charismatic chef in a bustling kitchen, his apron dusted with flour, smiling as he presents a beautifully prepared dish. 8K, hyper-realistic, cinematic, post- post-production.
7. A young adventurer with tousled hair and bright eyes, wearing a leather jacket and a backpack, ready to explore distant lands. 8K, hyper-realistic, cinematic, post-production.
8. "A watercolor painting of a vibrant flower field in spring, with rainbow of blossoms under a bright blue sky. 8K, best quality, fine details.".
9. digital art of beautiful tiger pokemon under an apple tree, cartoon style,Matte Paint- Mng.M Realism,Bright coyprr,hyper quality.high detail,high resolution, -video -s 750-v 6.0 -ar 1:2"
10. "painterly style, Goku fighting war, raging, blue hair, character design, body is adorndd with glowing golden runes, ycllow aura around him, body dynamic epic action pose, in- tricate. highly detailed, epic and dynamic composition, dynamic angle, intricate details, multicolor explosion, blur effect, sharp focus, uhd, hdr, colorful shot, stormy weather, tons of flying debris around him, dark city background, modifer-CCamgeeStlle color=blood red, intensity=1.6"
11. A stunning steampunk city with towering skyscrapers and intricate clockwork mechanisms, gears and pistons move in a complex symphony, steam billows from chimneys, airships navigate the bustling skylanes, a vibrant metropolis
12. "Samurai looks at the enemy, stands after the battle. fear and horror on his face, tired and beaten, sand on his face mixed with sweat, an atmosphere of darkness and horror, hyper realistic photo, In post - production, enhance the details, sharpness, and contrast to achieve the hyper- realistic effect"
13. A portrait of elemental entity with strong rim lighting and intricate details, painted digitally by Alvaro Castagnet, Peter Mohrbacher, and Dan Mumford
14. "A regal female portrait with an ornate headdress decorated with colorful gemstones and feathers, her robes rich with intricate designs and bright hues. 8K, best quality, fine details.".
26

---

Published as a conference paper at ICLR 2025
15. "A detailed painting of Atlantis by multiple artists, featuring intricate detailing and vibrant colors.".
"A landscape featuring mountains, a valley, sunset light, wildlife and a gorilla, reminiscent of Bob Ross's artwork.
Fantusy themed portrail of female elf golöen eyes,
pumpkins, lago,
Highly detailed mysterious egyptian (sphynx cal), skindenation:11.22 [..]
abstract
houses, streets,
style brian [-]]
painterly style, seductive female League legends character fighting
charismatic chef in bustling kicchen upron flour, smiling
young adventurer with tonsled hair bright eyes, wearing
watereoler painting of viheant flower spring,
digital art of beautiful tiger pokemon
puinterly style, Goku Sigttng wan, A stunning steampunk city with tow blue charccter design.
Samurai looks the enemy, stands after burtle, and horoo
under apple
skyserapers and intrictte
A portrait of elemental eniity with rim lighting and intricate [..]
A regal female purtriit with an ornase heuddrees decorated with gemstones....
A decailed puinsng of Atlantis by mult- artists, (eotuning intricate [..] A landscape featuring mountains, val- wildhif and
Figure 12: More examples from the ote-step generator distilled from Stable Diffusion 2.1 2.1- using the proposed method: Score identity Distillation with Long-Short Short Guidance.
27


Reviews:
[{"review_id": "XRehdoDeLj", "reviewer": "Reviewer_i638", "summary": "This paper focuses on a novel approach to accelerating text-to-image diffusion models through Score Identity Distillation with Long-Short Guidance (SiD-LSG), specifically targeting data-free one-step generation. By combining Score Identity Distillation with Classifier-Free Guidance (CFG), the authors efficiently distill Stable Diffusion models into one-step generators without needing real data. The proposed method incorporates both long and short CFG strategies, which balance FID and CLIP scores, enabling efficient and high-quality text-to-image generation. Experimental results demonstrate the effectiveness of the proposed method on the COCO dataset, achieving a record-low data-free FID of 8.15.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper introduces two distinct strategies for CFG in SiD: the long CFG and short CFG strategies, each with unique effects on model performance.\n1. In the long CFG, it is applied to the pretrained score network (the teacher model) with a higher guidance scale (typically $\\tau$>1), which encourages the pretrained model to generate images more aligned with the textual prompt, which in turn compels the student generator to follow this alignment. The primary effect is a strong improvement in text adherence and quality, with the capability to achieve a balanced FID and CLIP score. \n2. In the short CFG, it is applied to the fake score network (the student model) with a reduced guidance scale (typically $\\tau$ values between 0 and 1). By reducing CFG, this approach lessens the fake score network’s alignment with the text prompt, which incentivizes the student generator to improve text alignment and quality. This strategy tends to result in a trade-off where lower CLIP scores (text alignment) can occur, but FID scores (distribution match) remain competitive.", "weaknesses": "1. The main contribution of this paper is introducing a variant of CFG into SiD, i.e., long and short CFGs, which aim to balance semantic alignment and the quality of generated images. However, there seems to be a contradiction, as no single $\\tau$ value optimally balances both CLIP and FID scores (see results in Tables 1 and 2). Are there intrinsic reasons for this phenomenon?\n\n2. A more intuitive ablation study would better showcase the effectiveness of the proposed long and short CFG strategies. Specifically, in Tables 1 and 2, rather than using SiD-LSG alone, could you provide the results (both CLIP and FID) for SiD with the same $\\tau$ value across different configurations: SiD (short CFG only), SiD (long CFG only), SiD (CFG only), and SiD (original)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RvHLVl3mZN", "reviewer": "Reviewer_8grk", "summary": "The paper proposes a new distillation technique (SiD-LSG) for data-free one-step text-to-image generation. First, it adapts Score identity Distillation (SiD) to work with text-to-image models. Second, it proposes a Long and Short Classifier-Free Guidance (LSG) scheme to improve the distillation effectiveness. SiD-LSG shows state-of-the-art distillation results in a data-free setting. With a small CFG, it achieves impressive FID scores (8.15 for SD1.5 and 9.52 for SD2.1). With a large CFG, it achieves high CLIP scores, with sacrification in FID ones.", "questions": "See the weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper adapts Score identity Distillation (SiD) to work with text-to-image models.\n- SiD-LSG shows state-of-the-art distillation results in a data-free setting. With a small CFG, it achieves impressive FID scores (8.15 for SD1.5 and 9.52 for SD2.1). With a large CFG, it achieves high CLIP scores, with sacrification in FID ones.\n- The paper presents a comprehensive set of experiments.", "weaknesses": "- L268-269: The claim \"we are the first to incorporate CFG into the training of the fake score network\" is incorrect. SwiftBrush uses CFG 4.5 for both teachers (see Implementation Details in Section 4.1 in their paper).\n- While the paper proposes an interesting short guidance setting ($\\kappa 2 = \\kappa 3 < 1$), it does not apply to the long and short guidance ($\\kappa 2 = \\kappa 3 > 1$). I do not see the connection between the short guidance setting and the long and short guidance setting.\n- In the end, the long and short guidance scheme is very simple and uninteresting ($\\kappa 1 = \\kappa 2 = \\kappa 3 = \\kappa 4 = 1.5). It just applies the same CFG for both teachers, which SwiftBrush already did.\n- Some recent works (SwiftBrush v2, DMD2) should be mentioned and compared with.\n- The training protocol is expensive. The batch size is large (512 by default), requiring 64 gradient accumulation rounds on H100 (80G), FP32 setting. SiD-LSG necessitates gradient backpropagation through the score networks, which requires massive memory usage. Hence, it can only use the batch size per GPU as 1 on H100 (80G), FP32 setting.\n- Fig 3, 4, 6, 9: The curves for FID scores and CLIP scores are hard to differentiate. They have the same color and with just a small transparency difference. The authors should improve these figures.\n- Fig. 6, which is discussed in Section 3.2, is in the Appendix. It is not good. The authors should move the figure to the main paper and squeeze the text to fit in the 10-page limit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uKYfvQH1y2", "reviewer": "Reviewer_F9u6", "summary": "The authors propose an enhancement to Score Identity Distillation (SiD), a recently introduced method for distilling a pre-trained diffusion network into a single-step generator. This paper’s method specifically targets pre-trained text-to-image models based on DDPM, adapting SiD to distill two epsilon-prediction networks: SD1.5 and SD2.1.base.\n\nFollowing the SiD framework, the one-step generator ($\\theta$) is optimized by training an additional fake multi-step generator ($\\psi$). Furthermore, classifier-free guidance (CFG) is integrated into the distillation process, allowing for varied CFG values that emphasize different target objectives. This added flexibility contributes to enhanced model performance.\n\nThe results achieve state-of-the-art FID scores and demonstrate strong CLIP scores on the COCO-2014 validation prompts.", "questions": "- Does this approach valid for v-prediction networks? how does this alter the theory?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well written and easy to follow\n- The novelty and relation to SiD is clear.\n- The theory and mathematical background is solid.\n- The method is novel and important.\n- The ablation is comprehensive.\n- The results of the methods are SoTA.", "weaknesses": "- Given SiD the method novelty is limited to the adaptation of the SiD theory to DDPM and t2i models.\n- The distillation is on relatively old pre-trained networks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fEBPWv1e4j", "reviewer": "Reviewer_fY7L", "summary": "This paper mainly studies how to accelerate the diffusion-based generative model to generate high-quality images. The paper presents a novel method called Guided Score Identity Distillation with Long-Short Guidance (SiD-LSG) for data-free one-step text-to-image generation, which enhances the Score Identity Distillation (SiD) technique by incorporating Long and Short Classifier-Free Guidance (LSG).", "questions": "If a model itself does not use CFG, then the proposed method will not work？ For example, FLUX does not use CFG for inference but uses word embedding instead. Just open discussion.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.  The main contribution is The proposed method enhances the Score Identity Distillation technique by incorporating Classifier-Free Guidance , although these two techniques have been proposed in other studies.\n\n2. By reducing the number of steps required for image generation to one, the method significantly improves the efficiency of the generation process, and the proposed method achieves state-of-the-art performance in terms of Fréchet Inception Distance (FID) on the COCO-2014 validation set, setting a new benchmark for data-free one-step distillation.", "weaknesses": "1.  Regarding training efficiency, SiD needs to generate false data for training, and adding CFG will further increase the model overhead, resulting in more training time and resources.\n\n2. The data-free nature of the SiD and SID-LSG means it does not leverage real-world data to further refine the generated images, which could potentially improve photorealism and text alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)