Guided Score identity Distillation for Data-Free One-Step Text-to-Image Generation

Authors: Mingyuan Zhou, Zhendong Wang, Huangjie Zheng, Hai Huang
Primary area: generative models
Keywords: stable diffusion, data-free distillation, single-step generation, classifier-free guidance

Abstract:
Diffusion-based text-to-image generation models trained on extensive text-image pairs have demonstrated the ability to produce photorealistic images aligned with textual descriptions. However, a significant limitation of these models is their slow sample generation process, which requires iterative refinement through the same network. To overcome this, we introduce a data-free guided distillation method that enables the efficient distillation of pretrained Stable Diffusion models without access to the real training data, often restricted due to legal, privacy, or cost concerns. This method enhances Score identity Distillation (SiD) with Long and Short Classifier-Free Guidance (LSG), an innovative strategy that applies Classifier-Free Guidance (CFG) not only to the evaluation of the pretrained diffusion model but also to the training and evaluation of the fake score network. We optimize a model-based explicit score matching loss using a score-identity-based approximation alongside our proposed guidance strategies for practical computation. By exclusively training with synthetic images generated by its one-step generator, our data-free distillation method rapidly improves FID and CLIP scores, achieving state-of-the-art FID performance while maintaining a competitive CLIP score. Notably, the one-step distillation of Stable Diffusion 1.5 achieves an FID of **8.15** on the COCO-2014 validation set, a record low value under the data-free setting.  Our code and checkpoints are available at https://github.com/mingyuanzhou/SiD-LSG.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730646890607

thread_type:
official_review

<thread_object>
creator: reviewer_fy7l

<content>
summary:
This paper mainly studies how to accelerate the diffusion-based generative model to generate high-quality images. The paper presents a novel method called Guided Score Identity Distillation with Long-Short Guidance (SiD-LSG) for data-free one-step text-to-image generation, which enhances the Score Identity Distillation (SiD) technique by incorporating Long and Short Classifier-Free Guidance (LSG).

strengths:
1.  The main contribution is The proposed method enhances the Score Identity Distillation technique by incorporating Classifier-Free Guidance , although these two techniques have been proposed in other studies.

2. By reducing the number of steps required for image generation to one, the method significantly improves the efficiency of the generation process, and the proposed method achieves state-of-the-art performance in terms of Fréchet Inception Distance (FID) on the COCO-2014 validation set, setting a new benchmark for data-free one-step distillation.

weaknesses:
1.  Regarding training efficiency, SiD needs to generate false data for training, and adding CFG will further increase the model overhead, resulting in more training time and resources.

2. The data-free nature of the SiD and SID-LSG means it does not leverage real-world data to further refine the generated images, which could potentially improve photorealism and text alignment.

questions:
If a model itself does not use CFG, then the proposed method will not work？ For example, FLUX does not use CFG for inference but uses word embedding instead. Just open discussion.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730651365271

thread_type:
official_review

<thread_object>
creator: reviewer_f9u6

<content>
summary:
The authors propose an enhancement to Score Identity Distillation (SiD), a recently introduced method for distilling a pre-trained diffusion network into a single-step generator. This paper’s method specifically targets pre-trained text-to-image models based on DDPM, adapting SiD to distill two epsilon-prediction networks: SD1.5 and SD2.1.base.

Following the SiD framework, the one-step generator ($\theta$) is optimized by training an additional fake multi-step generator ($\psi$). Furthermore, classifier-free guidance (CFG) is integrated into the distillation process, allowing for varied CFG values that emphasize different target objectives. This added flexibility contributes to enhanced model performance.

The results achieve state-of-the-art FID scores and demonstrate strong CLIP scores on the COCO-2014 validation prompts.

strengths:
- The paper is well written and easy to follow
- The novelty and relation to SiD is clear.
- The theory and mathematical background is solid.
- The method is novel and important.
- The ablation is comprehensive.
- The results of the methods are SoTA.

weaknesses:
- Given SiD the method novelty is limited to the adaptation of the SiD theory to DDPM and t2i models.
- The distillation is on relatively old pre-trained networks.

questions:
- Does this approach valid for v-prediction networks? how does this alter the theory?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695003793

thread_type:
official_review

<thread_object>
creator: reviewer_8grk

<content>
summary:
The paper proposes a new distillation technique (SiD-LSG) for data-free one-step text-to-image generation. First, it adapts Score identity Distillation (SiD) to work with text-to-image models. Second, it proposes a Long and Short Classifier-Free Guidance (LSG) scheme to improve the distillation effectiveness. SiD-LSG shows state-of-the-art distillation results in a data-free setting. With a small CFG, it achieves impressive FID scores (8.15 for SD1.5 and 9.52 for SD2.1). With a large CFG, it achieves high CLIP scores, with sacrification in FID ones.

strengths:
- The paper adapts Score identity Distillation (SiD) to work with text-to-image models.
- SiD-LSG shows state-of-the-art distillation results in a data-free setting. With a small CFG, it achieves impressive FID scores (8.15 for SD1.5 and 9.52 for SD2.1). With a large CFG, it achieves high CLIP scores, with sacrification in FID ones.
- The paper presents a comprehensive set of experiments.

weaknesses:
- L268-269: The claim "we are the first to incorporate CFG into the training of the fake score network" is incorrect. SwiftBrush uses CFG 4.5 for both teachers (see Implementation Details in Section 4.1 in their paper).
- While the paper proposes an interesting short guidance setting ($\kappa 2 = \kappa 3 < 1$), it does not apply to the long and short guidance ($\kappa 2 = \kappa 3 > 1$). I do not see the connection between the short guidance setting and the long and short guidance setting.
- In the end, the long and short guidance scheme is very simple and uninteresting ($\kappa 1 = \kappa 2 = \kappa 3 = \kappa 4 = 1.5). It just applies the same CFG for both teachers, which SwiftBrush already did.
- Some recent works (SwiftBrush v2, DMD2) should be mentioned and compared with.
- The training protocol is expensive. The batch size is large (512 by default), requiring 64 gradient accumulation rounds on H100 (80G), FP32 setting. SiD-LSG necessitates gradient backpropagation through the score networks, which requires massive memory usage. Hence, it can only use the batch size per GPU as 1 on H100 (80G), FP32 setting.
- Fig 3, 4, 6, 9: The curves for FID scores and CLIP scores are hard to differentiate. They have the same color and with just a small transparency difference. The authors should improve these figures.
- Fig. 6, which is discussed in Section 3.2, is in the Appendix. It is not good. The authors should move the figure to the main paper and squeeze the text to fit in the 10-page limit.

questions:
See the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708547363

thread_type:
official_review

<thread_object>
creator: reviewer_i638

<content>
summary:
This paper focuses on a novel approach to accelerating text-to-image diffusion models through Score Identity Distillation with Long-Short Guidance (SiD-LSG), specifically targeting data-free one-step generation. By combining Score Identity Distillation with Classifier-Free Guidance (CFG), the authors efficiently distill Stable Diffusion models into one-step generators without needing real data. The proposed method incorporates both long and short CFG strategies, which balance FID and CLIP scores, enabling efficient and high-quality text-to-image generation. Experimental results demonstrate the effectiveness of the proposed method on the COCO dataset, achieving a record-low data-free FID of 8.15.

strengths:
The paper introduces two distinct strategies for CFG in SiD: the long CFG and short CFG strategies, each with unique effects on model performance.
1. In the long CFG, it is applied to the pretrained score network (the teacher model) with a higher guidance scale (typically $\tau$>1), which encourages the pretrained model to generate images more aligned with the textual prompt, which in turn compels the student generator to follow this alignment. The primary effect is a strong improvement in text adherence and quality, with the capability to achieve a balanced FID and CLIP score. 
2. In the short CFG, it is applied to the fake score network (the student model) with a reduced guidance scale (typically $\tau$ values between 0 and 1). By reducing CFG, this approach lessens the fake score network’s alignment with the text prompt, which incentivizes the student generator to improve text alignment and quality. This strategy tends to result in a trade-off where lower CLIP scores (text alignment) can occur, but FID scores (distribution match) remain competitive.

weaknesses:
1. The main contribution of this paper is introducing a variant of CFG into SiD, i.e., long and short CFGs, which aim to balance semantic alignment and the quality of generated images. However, there seems to be a contradiction, as no single $\tau$ value optimally balances both CLIP and FID scores (see results in Tables 1 and 2). Are there intrinsic reasons for this phenomenon?

2. A more intuitive ablation study would better showcase the effectiveness of the proposed long and short CFG strategies. Specifically, in Tables 1 and 2, rather than using SiD-LSG alone, could you provide the results (both CLIP and FID) for SiD with the same $\tau$ value across different configurations: SiD (short CFG only), SiD (long CFG only), SiD (CFG only), and SiD (original)?

questions:
Please refer to the Weaknesses.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper focuses on accelerating diffusion-based generative models. It introduces a novel method, Guided Score Identity Distillation with Long-Short Guidance (SiD-LSG), designed for data-free one-step text-to-image generation. The approach builds upon the existing Score Identity Distillation (SiD) framework by integrating Long and Short Classifier-Free Guidance (LSG) to enhance its effectiveness.

A key strength of this work lies in its data-free nature, making it much more generalizable especially when access to real training data is limited. The experimental results also demonstrate state-of-the-art performance. The method's scalability and adaptability to various diffusion models further enhance its significance.

Given these strengths and the consistent positive feedback from the reviewers, I recommend acceptance of the paper.