Adversarial Score identity Distillation: Rapidly Surpassing the Teacher in One Step

Authors: Mingyuan Zhou, Huangjie Zheng, Yi Gu, Zhendong Wang, Hai Huang
Primary area: generative models
Keywords: Diffusion distillation, deep generative models

Abstract:
Score identity Distillation (SiD) is a data-free method that has achieved state-of-the-art performance in image generation by leveraging only a pretrained diffusion model, without requiring any training data. However, the ultimate performance of SiD is constrained by the accuracy with which the pretrained model captures the true data scores at different stages of the diffusion process. In this paper, we introduce SiDA (SiD with Adversarial Loss), which not only enhances generation quality but also improves distillation efficiency by incorporating real images and adversarial loss. SiDA utilizes the encoder from the generator's score network as a discriminator, allowing it to distinguish between real images and those generated by SiD. The adversarial loss is batch-normalized within each GPU and then combined with the original SiD loss. This integration effectively incorporates the average "fakeness" per GPU batch into the pixel-based SiD loss, enabling SiDA to distill a single-step generator. SiDA converges significantly faster than its predecessor when distilled from scratch, and swiftly improves upon the original model's performance during fine-tuning from a pre-distilled SiD generator. This one-step adversarial distillation method establishes new benchmarks in generation performance when distilling EDM diffusion models, achieving FID scores of **1.499** on CIFAR-10 unconditional, **1.396** on CIFAR-10 conditional, and **1.110** on ImageNet 64x64. When distilling EDM2 models trained on ImageNet 512x512, our SiDA method surpasses even the largest teacher model, EDM2-XXL, which achieved an FID of 1.81 using classifier-free guidance (CFG) and 63 generation steps. Specifically, SiDA achieves FID scores of **2.156** for size XS, **1.669** for S, **1.488** for M, **1.413** for L, **1.379** for XL, and **1.366** for XXL, all without CFG and in a single generation step. These results highlight substantial improvements across all model sizes. Our code and checkpoints are available at https://github.com/mingyuanzhou/SiD/tree/sida.

Decision: accept (poster)
Reviewer scores: 6, 8, 5, 6

Reviews and discussion:
<thread>
creation date:
1730549415133

thread_type:
official_review

<thread_object>
creator: reviewer_spnq

<content>
summary:
This paper proposes an intuitive method, which mainly focuses on the improvement of score identity distillation method by introducing additional real images for adversarial training. The improvement is two fold. First, additional real images can boost generation performance over data-free score identity distillation by discrimating real and synthetic images. Second, it achieves faster convergence rate relative to the baselines. The experimental results on small-scale dataset support the superority of the proposed method.

strengths:
+ This paper proposes an intuitive extension of score identity distillation and effectively improves the generation performance on small-scale datasets, such as CIFAR and ImageNet 64x64.
+ The proposed method achieves better performance than the main baseline, SiD and other diffusion methods.

weaknesses:
+ More state-of-the-art generative methods are missing, such as DiT, RDM and MAR.
+ The method only conducts experiments on small-scale datasets or low-resolution images. I'm concern about the effectiveness of this method on large-scale dataset with high-resolution images. More results on large-scale datasets with high-resolution images consolidate the effectiveness of this method.
+ OSGAN[R1] adopts an one-step adversarial training on generative task. The comparion with OSGAN is needed to distinguish their differences and highlight the contributions of this paper.
+ This paper doesn't mention any plan about open source of code and trained weights. In spite of encouraging performance, I'm concern about the reproducibility of this paper. 
+ Some motivations are not well clarified in the paper. See Questions section.
+ Some unnecessary equations, such as Eq between 149 and 151, can be removed.

[R1] Training Generative Adversarial Networks in One Stage. CVPR 2021.

questions:
+ Why can incorporating average "fakeness" within each GPU batch into each pixel's loss solve the gap between diffusion distillation and adversarial learning?
+ Why does the proposed method can achieve faster convergence than baseline when trained from scratch?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707625194

thread_type:
official_review

<thread_object>
creator: reviewer_4cak

<content>
summary:
This paper introduces an improved version of score identity distillation and proposes training SiD with an adversarial objective. 
The main motivation is that the lack of real data in the original training of SiD can potentially lead to discrepancies to the original data distribution. The authors propose to reuse the encoder part of the fake score model as the backbone of discriminator, and use its predicted realness and fakeness scores to improve the generation results further, at no cost of additional model parameters. By combining SiD with adversarial training, together with additional techniques such as initializing the student model with the SiD pretrained model, the proposed method achieved improved one-step diffusion results on multiple datasets.

strengths:
- The discussions in the experiment section are very comprehensive. The detailed analysis of the training statistics across the training steps is appreciated. 

- The paper presented the background such as SiD and adversarial loss in diffusion well.

weaknesses:
- Presentation

In the technical parts of the paper, which is mainly the Section 4. I frequently find the presentations not clear enough. My biggest confusion arises from the 'Loss Construction that Respects the Weighting of Diffusion Distillation Loss' paragraph. I understand that balancing loss terms can be tricky, and I acknowledge that this is an important topic to discuss. But from what the author presents, I didn't find how the proposed weight 'honors the existing weighting mechanisms' and, more importantly, automatically balances the loss regardless of the datasets. I read the Eq 7 and Eq 8 and found nothing particularly new. How is Eq 7 fundamentally different from PatchGAN? 

- Novelty

Although the paper targets a problem of SiD that is not previously discussed, the solution of adding an adversarial term is by no means a novelty under the context of diffusion distillation. The adversarial object has been proven very effective in many works, such as UFOGen and SD3-turbo. The proposed method here is a simple mixture of two existing methods. 

- Evaluation and performance

Most of the experiment settings used in this paper are small-scale. And the authors only compared the proposed method with very few other methods in the 512x512 ImageNet experiment. Since efficiency is one of the main advantages the authors are trying to advocate, it is necessary to include larger-scale evaluations, such as text-to-image experiments, to allow for fair comparisons with recent methods. 

I also have a slight concern about the advantage of efficiency. It is true that, according to Figure 3, the FID of the proposed method decreases faster compared to the baseline SiD in the later half of the training; I'd like to point out that this is achieved with an additional loss term, meaning each iteration probably optimizes the model with a larger gradient magnitude. Also, the adversarial loss term will inevitably introduce more computation, so the model is not necessarily able to finish training faster compared to SiD.

questions:
Please see the Weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718330100

thread_type:
official_review

<thread_object>
creator: reviewer_kyrq

<content>
summary:
The paper introduces Score Identity Distillation with Adversarial loss. The method aims to distill a teacher diffusion model to reduce the number of inference steps to only one. The primary contribution is the combination of SiD with the use of real images and adversarial objective, which not only reduces the number of required inference steps but also improves generation quality, achieving a lower FID score across Cifar10, ImageNet-64, FFHQ-64 and AFHQ-64.

strengths:
- Achieves state-of-the-art performance with only one inference step
- The methods demonstrates improvement over the teacher network thanks to the real images
- Training remains relatively stable despite the use of an adversarial loss

weaknesses:
- Requires real images for training, which is not necessary for other distillation methods
- Experiments are demonstrated only on very small images resolution
- Ablation studies are limited to the parameter $\alpha$
- A schematic of the framework would greatly improve the paper's readability.

questions:
- How does the method perform as the number of available real images changes? What happen if you use only 10% of the training real images?
- Why not apply latent diffusion for larger images? Using VAE to compress the image would help to see if the method can scale to bigger images, without increasing the training time
- Does the approach generalize to teacher models other than VP-EDM?
- The paper arbitrarily sets $\lambda_{sid}$ = 100 and $\lambda_{adv}$ = 0.01 without discussion or ablation, although these hyperparameters likely play an important role in the method. How do these values influence the performance metrics?
- In Figure 1, for $\alpha$=1.2 (the yellow line), after approximately 1000 iterations, the Inception Score (IS) drops and the FID increases sharply; additionally, SiDA with $\alpha$=1.2 is absent from Table 1. Does this imply that the model fails to converge?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731031456207

thread_type:
official_review

<thread_object>
creator: reviewer_ajww

<content>
summary:
The authors propose SiDA (Score identity Distillation with Adversarial Loss), an enhanced version of Score identity Distillation (SiD) that aims to improve diffusion model training efficiency. The key innovation is the introduction of an adversarial loss that helps distinguish between real and generated latents at different noise levels. By reusing the fake score network in the original SiD, the authors propose a discriminator design that utilizes the encoder feature map to obtain the discriminator score. The proposed method achieves SOTA results on CIFAR-10, ImageNet-64, and other datasets.

strengths:
-	The authors repurpose the fake score network as an additional discriminator with direct using the average of the encoder features as the discriminator score. This design introduces the adversarial objective to the original SiD with minimal additional parameter cost.
-	The proposed method SiDA significantly outperforms SiD and achieves superior 1-step generation performance on various datasets.

weaknesses:
-	The discriminator design is heavily based on the U-Net encoder-decoder architecture. It is not straightforward to generalize to transformer-based diffusion models, e.g., DiT, where there is no clear partition of the encoder and decoder.
-	The current submission lacks the evaluation of Diffusion models on higher resolution, e.g., ImageNet 256x256. The cost overhead from both discriminator training and maintaining an additional fake score network can be exaggerated on high-resolution data.

questions:
-	Why $L^1(\theta)$ is still needed with the additional discriminator, i.e., why SiDA still needs tuning $\alpha$?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731526866943

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Preliminary Response (Updated): Addressing Concerns on High-Resolution Image Results

comment:
Dear Reviewers, 

We appreciate your valuable feedback. While we are preparing a comprehensive revision, we would like to promptly address the shared concern regarding the absence of results for high-resolution images. To this end, we have adapted the SiDA codebase to distill EDM2 models of various sizes pretrained on ImageNet 512×512, yielding the following results:

| **Method** | **CFG** | **NFE** | **XS (125M)** | **S (280M)** | **M (498M)** | **L (777M)** | **XL (1.1B)** | **XXL (1.5B)** |
|------------|---------|---------|---------------|--------------|--------------|--------------|---------------|----------------|
| EDM2       | N       | 63      | 3.53          | 2.56         | 2.25         | 2.06         | 1.96          | 1.91           |
| EDM2       | Y       | 63x2      | 2.91          | 2.23         | 2.01         | 1.88         | 1.85          | 1.81           |
| sCT        | Y       | 1       | -             | 10.13        | 5.84         | 5.15         | 4.33          | 4.29           |
| sCT        | Y       | 2       | -             | 9.86         | 5.53         | 4.65         | 3.73          | 3.76           |
| sCD        | Y       | 1       | -             | 3.07         | 2.75         | 2.55         | 2.40          | 2.28           |
| sCD        | Y       | 2       | -             | 2.50         | 2.26         | 2.04         | 1.93          | 1.88           |
| SiD        | N       | 1       | 3.353 ± 0.041 | 2.707 ± 0.054| 2.060 ± 0.038| 1.907 ± 0.016| 1.888 ± 0.030 | 1.969 ± 0.029              |
| SiDA       | N       | 1       | 2.228 ± 0.037 | 1.756 ± 0.023| 1.546 ± 0.023| 1.501 ± 0.024| 1.428 ± 0.018 | 1.503 ± 0.020             |
| SiD²A      | N       | 1       | **2.156 ± 0.028** | **1.669 ± 0.019** | **1.488 ± 0.038** | **1.413 ± 0.022** | **1.379 ± 0.017** | **1.366** ± 0.015              |


The results for EDM2 are sourced from Karras et al. (2024), while those for sCT and sCD are obtained from Lu and Song (2024). Although Lu and Song's work is concurrent with ours, we include their results here to highlight the superior performance of SiDA.

We observe that SiD²A achieves an FID of 1.366 on EDM2-XXL, marking the lowest score reported to date. This result is notable as it is obtained without classifier-free guidance (CFG), post-hoc exponential moving average (EMA), and utilizes only a single generation step. Furthermore, both SiDA and SiD²A applied to EDM2-S (280M parameters) surpass the performance of the largest teacher model, EDM2-XXL (1.5B parameters), as well as sCD and sCT methods. We also note that SiD, the baseline model adapted by us for EDM2, achieves performance comparable to that of the teacher model. This observation aligns with previous findings on smaller datasets and EDM models, demonstrating SiD's effectiveness across varying scales. 

**References:**

- Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. "Analyzing and Improving the Training Dynamics of Diffusion Models." In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 24174–24184, 2024.

- Cheng Lu and Yang Song. "Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models." *arXiv preprint arXiv:2410.11081*, 2024.

We will provide a more detailed response and a revised paper in due course.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper improves score identity distillation by integrating real data with adversarial loss, which enhances generation quality even surpassing the teacher in one-step generation, while also improving distillation efficiency. Overall, this work receives positive scores, and the reviewers think the idea of introducing real data is interesting, the method is well designed, and the performance gain is significant. Some concerns include: 1) the method focuses only on U-Net without considering DiT, 2) most experiments are small-scale; and 3) the use of adversarial loss is not new. AC agrees this is an interesting advance on score identity distillation, and recommends acceptance as poster.