Adversarial Score identity Distillation: Rapidly Surpassing the Teacher in One Step
Authors: Mingyuan Zhou
Huangjie Zheng
Yi Gu
Zhendong Wang
Hai Huang
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion distillation
deep generative models
Primary area: generative models
TL;DR: The integration of SiD and Diffusion GAN results in a distilled generator that achieves SoTA FID, without CFG and using only a single step for generation 🚀

Abstract: Score identity Distillation (SiD) is a data-free method that has achieved state-of-the-art performance in image generation by leveraging only a pretrained diffusion model, without requiring any training data. However, the ultimate performance of SiD is constrained by the accuracy with which the pretrained model captures the true data scores at different stages of the diffusion process. In this paper, we introduce SiDA (SiD with Adversarial Loss), which not only enhances generation quality but also improves distillation efficiency by incorporating real images and adversarial loss. SiDA utilizes the encoder from the generator's score network as a discriminator, allowing it to distinguish between real images and those generated by SiD. The adversarial loss is batch-normalized within each GPU and then combined with the original SiD loss. This integration effectively incorporates the average "fakeness" per GPU batch into the pixel-based SiD loss, enabling SiDA to distill a single-step generator. SiDA converges significantly faster than its predecessor when distilled from scratch, and swiftly improves upon the original model's performance during fine-tuning from a pre-distilled SiD generator. This one-step adversarial distillation method establishes new benchmarks in generation performance when distilling EDM diffusion models, achieving FID scores of **1.499** on CIFAR-10 unconditional, **1.396** on CIFAR-10 conditional, and **1.110** on ImageNet 64x64. When distilling EDM2 models trained on ImageNet 512x512, our SiDA method surpasses even the largest teacher model, EDM2-XXL, which achieved an FID of 1.81 using classifier-free guidance (CFG) and 63 generation steps. Specifically, SiDA achieves FID scores of **2.156** for size XS, **1.669** for S, **1.488** for M, **1.413** for L, **1.379** for XL, and **1.366** for XXL, all without CFG and in a single generation step. These results highlight substantial improvements across all model sizes. Our code and checkpoints are available at https://github.com/mingyuanzhou/SiD/tree/sida.

Published as a conference paper at ICLR 2025
ADVERSARIAL SCORE IDENTITY DISTILLATION: RAPIDLY SURPASSING THE TEACHER IN ONE STEP
Mingyuan Zhou 12, Huangjie Zheng', Yi Gu', Zhendong Wang', and Hai Huang"*
1 The University of Texas at Austin, ?Google DeepMind, and AAtlassian
ABSTRACT
Score identity Distillation (SiD) is a data-free method that has achieved state-of- the-art performance in image generation by leveraging only a pretrained diffusion model, without requiring any training data. However, the ultimate performance of SiD is constrained by the accuracy with which the pretrained model captures the true data scores at different stages of the diffusion process. In this paper, we introduce SiDA (SiD with Adversarial Loss), which not only enhances generation quality but also improves distillation efficiency by incorporating real images and adversarial loss. SiDA utilizes the encoder from the generator's s score network as a discriminator, allowing it to distinguish between real images and those generated by SiD. The adversarial loss normalizmallzzed within each GPU and then combined with the original SiD loss. This integration effectively incorporates the average "fakeness" per GPU batch into the pixel- -basd SiD loss, enabling SiDA to distill a single-step generator. SiDA converges significantly faster than its predecessor when distilled from scratch, and swiftly improves upon the original model's performance during fine-tuning from a pre-distilled SiD generator. This one-step adversarial distillation method establishes new benchmarks in generation performance when distilling EDM diffusion models, achieving FID scores of 1.499 on CIFAR- 10 unconditional, 1.396 on CIFAR-10 conditional, and 1.110 on ImageNet 64x64. When distilling EDM2 models trained on ImageNet 512x512, 12, our SiDA method surpasses even the largest teacher model, EDM2-XXI, which achieved an FID of 1.81 using classifier free guidance (CFG) and 63 generation steps. Specifically, SiDA achieves FID scores of 2.156 for size XS, 1.669 for S. 1.488 for M, 1.413 for L, 1.379 for XL, and 1.366 for XXL, all without CFG and a single generation step. These results highlight substantial improvements across all model sizes. Our code and checkpoints are available at
INTRODUCTION
Modeling the distribution of nigh-dimensional data, such natural images, has been persistent challenge in machine learning (Bishop) 20066] Murphy 2012 Goodfellow et al. (016)). Before the emergence of deep generative models, research focused primarily on constructing hierarchical models constrained by parametric distributions (Blei et al., 2003 Grifliths & Ghahramani 2005 Fei-Fei &
Perona, 2005 Chong al. 20099] Zhou et al. 2009) and developing neural networks with stochastic binary hidden layers (Hinton et al. 2006 Salakbutdinov & Hinton 2009, Vincent et al.) 2010).
supported by robust inference techniques such as Gibbs sampling, maximum likelihood, variational inference (Hoffman et al. 2013 Blei & Jordan 2006), and contrastive divergence (Hinton 2002).
The last decade has witnessed significant advancements in deep generative models, includ- ing generative adversarial networks (GANs) (Goodfellow et al.] 2014), Reed et al., (016)] Kar- ras et al.) (019)), normalizing flows (Papamakarios et al. (019), and variational auto-encoders (VAEs) (Kingma & Welling 2014| Rezende et al.) 2014). Although VAEs are noted for their stability, they often produce blurred images, while GANs are acclaimed for their ability to generate photorealistic images despite challenges with training instability and generation di- versity. These dynamics have driven research towards refining statistical distances to more
"The majority of the work was done while the authors were at Google.

---

Published as a conference paper at ICLR 2025
accurately measure discrepancies between true and generated data distributions, espe- cially in scenarios where traditional metrics fail due to non-overlapping supports in high- dimensional spaces (Arjovsky et al.. 2017
ict al. 2017 Zheng & Zhou (222). Diffusion models (Sohl-Dickstein et al. 2013] Song
rod kouge PC
& Ermon 2019; Ho et al. 2020) have fur- ther revolutionized this field with their ability to produce photorealistic images, albeit with multiple iterative refinement resulting slow sampling speeds. In response, significant ef- forts have been directed towards developing efficient methods to utilize pretrained diffu- sion models for reversing the forward diffu- sion process through iterative refinement.
Figure 1: Illustration of the SiDA algorithm. The gener- ator loss, as defined defined in (8). is determined by fake images zg
-ctrr-net-denoisee fake images fa( l). sake-score-nett net-enoe-d fake images fo t), and the take image logits. Mcanwhile, the fake score network loss, as defined in (9) is determined by [ t) and the fake- score-net encoded logits of both fake and real images.
Departing from conventional methods, Diffusion GAN (Wang et al. 2023b) reframes forward diffusion cost-effective, domain-agnostic data-augmentation strategy tackle issues of non- overlapping distribution support. This approach introduces noise at various levels and utilizes statistical distances such as Jensen-Shannon divergence (JSD) for comparing distributions. The method promotes aligmment between the true and generated data distributions at any stage during the forward diffusion process. It advocates for directly guiding the learning of a one-step generator through forward diffusion, moving away from the traditional dependence on iterative refinement-basee sampling seen in reverse diffusion processes.
Expanding on this approach, recent methodologies have evolved from directly comparing empirical distributions of diffused groundtruth and generated data via JSD to distill their scores-a notable strength of diffusion models at various stages of the forward diffusion process. This evolution has catalyzed the development of cutting-edge diffusion distillation techniques such as score distillation sampling (Poole et al. 2023), variational score distillation (Wang et al. 2023c), Diff- Diff--Instuct (Luo) jet al. 2023b). SwiftBrush (Nguyen & Tran 2024), and distribution matching distillation (DMD) (Yin et al.] 2024b). These methods refine models by analyzing the diffused KL divergence between corrupted data and model distributions, effectively leveraging the gradients of this divergence. Although directly computing this KL divergence presents significant challenges, its gradients can be effectively estimated using the scores from both the pretrained model and the current generator.
In pursuit of transcending traditional challenges associated with JSD and model-based KL. divergence- which are often difficult to optimize and prone to mode-seeking behaviors, respectively- Zhou et al. 2024b) have pioneered a model-based Fisher divergence within the diffusion distillation framework. This novel approach, notably data-free, has demonstrated for the first time the potential to match or even surpass the performance of teacher models in a single generation step. However, SiD and other data-free methods are based on the assumption that the score produced by teacher networks can well represent the data score. This assumption can potentially create a performance bottleneck for distilled single-step generators, especially if the teacher diffusion model is not well-trained or has limited capacity. In this paper, we aim to further enhance SiD by integrating real-world data. This integration seeks to compensate for the teacher model's limitations in accurately representing the true score, thereby producing even more realistic generative outcomes. This enhancement broadens the practical applications of diffusion distillation in real-world scenarios.
As illustrated in Figure T, the proposed SiDA algorithm builds on the existing fake score network and generator design of SiD, incorporating Diffusion GAN-based adversarial loss that discriminates whether generated images are real or fake at various noisy stages of the diffusion process. The discriminator shares the weight with the encoder modules of the fake score network. To ensure compatibility with SiD's pixel-level loss, we compute the discriminator loss at each spatial position of the latent encoder feature map and average across the batch within each GPU. This measure is integrated into the SiD loss for joint distillation and adversarial training and introduces no additional parameters. Our findings across teacher models of various sizes and diverse datasets indicate that SiDA improves iteration efficiency by nearly an order of magnitude when training models initialized with the teacher. Additionally, it achieves remarkably low FID when initialized from the SiD-distilled checkpoints, consistently outperforming the teacher models across all tested datasets and model sizes.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Diffusion models (Song & Ermon 2019; Ho et al. 2020: Karras et al., 2022) are a family of generative models that adopts a denoising score matching objective (Vincent 2011 SShh-DDickttiin et al. 2015) to learn the score function of complex high-dimensional distributions at different noise levels. A deep neural network is trained to match the score function of noise-corrupedd data via minimizing data-based Fisher divergence (Song & Ermon 2019). This trained score network is then used to generate new samples by iteratively denoising random noise. Diffusion models have gained significant attention and success in generative modeling due to training stability and high-quality sample generation, with example applications in text-to-image generation (Nichol et al.
2022; Ramesh et al., 20222] Saharia et al. 2022 Rombach et al. 2022 Podell et al., (024). However, the sampling process in diffusion models involves gradual denoising across many steps, making it significantly slower compared to one-step generation models like GANs and VAEs.
Numerous techniques have been proposed to accelerate diffusion models (Zheng et al. 2022 Lyu
al.
2022; Luhman & Luhman 202T, Salimans & Ho 2022 Zheng al.) 2023; Meng al.
2023
Kim et al. (023). Recently one-step and few-step diffusion distillation methods have gained
prominence, achieving the speed of GANs while retaining the robust capabilities of diffusion models.
These methods can be broadly classified into two categories: data-free (I.o et al. 2023b Nguyen &
Tran 2024] Zhou et al.] (2024b) and those requiring real images or teacher-synthesized noise-image
pairs (Liu et al. 2023), Song & Dhariwal 2023 Luo et al. 2023a Kim et al.) [023 [Suuer et al. 2023
Xu et al. 2023 Yin et al. 2024b). The latter frequently incorporate GAN-based adversarial learning as critical component to enhance the generation performance of baseline distillation methods, which
2023) to score distillation sampling (Poole et al. 2023 Wang al. 2023c). Without the adversarial component, performance typically declines significanlly (Kim et al. 2023] Sauer et al. 2023 Yin et al., 2024a), which raises concerns about the robustness and effectiveness of the underlying base models lacking adversarial learning support. We note that the term "One-Step" in this paper differs from its usage in OSGAN (Shen et al. (021), which refers to non-alternating GAN training technique, not a single generator pass.
vary from consistency models (Song et al.)
The collaborative integration of GANs and diffusion models, aimed at leveraging the strengths of both, is not concept that originated with adversarial distillation. This approach is rooted in earlier studies: Xiao et al. (2022) developed a method to learn the reverse process of the diffusion chain using a time-step-conee iiep-conditioned discriminator. Simultaneously, Zheng et al. (2022) proposed that GANs learn noisy marginal distribution at specific targeted timestep. which would then serve as the prior for initiating the reverse diffusion process. Furthermore, they fine-tuned an early version of the Stable Diffusion framework to produce this noisy marginal using discriminator equipped with an MLP head atop the encoder module of the U-Net in Stable Diffusion, specifically designed to accelerate text-to-image generation in just few steps while maintaining high quality. A key insight from Zheng ee al. (2022) is that the diffusion U-Net backbone can simultaneously be used for reverse diffusion and function as both a Diffusion GAN's generator and discriminator. Building on this, we aim to generalize and simplify the designs of Zheng et al. (2022h by repurposing the score networks in SiD to perform both distillation and adversarial learning, without introducing any additional parameters.
Our work also builds on previous efforts to enhance distribution matching by injecting noise. Mca- suring differences between distributions in high dimensions, where supports often do not overlap, is challenging and has spurred the development of advanced statistical distances (Murphy 2012 Arjovsky et al., 2017 Li et al. 2017, Zheng& & Zhou 202T). Adding noise has been shown to be effective in overlapping the supports of the true and fake data distributions, which typically reside in separate high-dimensional density regions (Wang et al. 20235). This overlap mitigates the limitations of many existing statistical distances in measuring differences between such distributions accurately.
Diffusion GAN introduced a novel strategy that matches the distribution between true and generated data at various stages of the forward diffusion process. This corrupts the observed data at different signal-to-noise ratios and aligns the distributions at multiple points throughout the diffusion pathway, using JSD for comparison to ensure a close alignment of the model and data distributions.
This approach to noise injection mnjeetion-based distribution matching has been further advanced by integrating pretrained diffusion models, which excel at estimating the true data score throughout the forward diffusion process. Building on this, while the KL divergence from diffused true to fake data distri- butions is intractable to estimate directly, its gradient can be computed using the scores of the true
3

---

Published as a conference paper at ICLR 2025
and fake data various noise levels (Poole et al. 2023 Wang al. 2023c). This advancement has
driven several recent innovations in one-step diffusion distillation methods for image generation, exemplified by studies such as Luo al. (2023b), Nguyen & Tran (2024), and Yin al. (2024b).
PRELIMINARIES
Below, we briefly review both SiD and Diffusion GAN, the foundations on which SiDA is built. A detailed description of SiDA follows in the next section.
Score identity Distillation (SiD): Moving beyond the challenges of computing JSD between diffused empirical data distributions, which can be unstable to optimize, and estimating the gradient of their KL divergence--noww for mode-seeking behaviors when expectations are taken with respect to the model distribution--Sin employs a model-based Fisher divergence to match their scores. More specifically, let us denote the forward transition kernel used in Gaussian diffusion models by g Øo) = No, I,I), where a E (0, 0₁ > 0, and the signal-to-noise ratio at o monotonically decreases towards zero as progresses. SiD explicitly represents the forward diffused data and model distributions as semi-implicit distributions (Yin & Zhou 2018 Yu et al. (023b):
Pdata J g( (a) )dzo; p0(2))) = f ((x (age]/agggl (2y))o P (mng)da...
These distributions, while lacking analytic density functions, are simple sample from. Specifically, by defining Gg as a generator converting noise into data, a diffused fake data 2; can be produced as:
c 0g + optt, 6, ~ I,, Tg - Go(z,c) c), ~~ p(z).
(1)
To distill a student generative model, the distribution pa(xx)) needs match the noisy data distribution at any time point L, ensuring that the distributions Poata (ro) and p2(xrg) are identical.
To achieve this goal, the distillation process involves adjusting the student model's parameters so that its output at various stages of diffusion aligns closely with the diffused data from the target distribution. This alignment is typically facilitated through optimization techniques that minimize specific statistical distance between these distributions across the entire diffusion trajectory (Wang let al., 2023b Luo et al.] (2023)). Following the same principle, SiD aims to minimize model-based Fisher divergence given by:
(o(o"."*'))) (0), = ER (pp())
| xe In Pdata (Ix) V æe In pa (2x)
Exe, (ax))
+ fo (xr, ( (x 0)13
where for and fo-(o) represent the optimal denoising score networks for the true and fake data distributions, respectively. These can be expressed as:
x, In In (a:Paa(())))// (x)))/a= (0),0) t) = (x, of In (2:)) E E|zr |xr]-
far (xt,, (x. + V
B6xr [xr]
GANs and Diffusion GANs: GANs (Goodfellow al. 2014) seek to match the distributions with min-max game between a discriminator D and a generator G, with the training loss formulated as:
min max (1) (log D(=)] + E==p(=)
D(G(z)))),
1).
To overcome the issues of von-overlapping probability support between two distributions, /Wang ct al. 2023b) propose Diffusion GANs to match the noisy distributions (x;, and po(x,) with discriminator conditioned on time-step i, which is sampled from an adaptive proposal distribution based on the competition between the generator and discriminator. Diffusion GANs are trained with:
min max Emo-p((a)) (;) mast) (Dz;; t)) + (== (C)))))[oo(( D( t))]
G D
From the observations above, we recognize that the core objective of both SiD and Diffusion GAN to align the noisy distributions at any time step to ensure that the generative distribution pe (x,, closely matches the clean data distribution Pdatu (.o). SiD operates under the assumption that it has access to the true data score fa- (or a pretrained estimation thereof) but not to the actual data, while Diffusion GAN operates under the reverse assumption access actual data but not far. In the following section, we will detail how these two objective functions are integrated for joint distillation and adversarial generation, leveraging their respective strengths enhance model performance.
4

---

Published as a conference paper at ICLR 2025
4 SIDA: ADVERSARIAL SCORE IDENTITY DISTILLATION
Motivations for Integrating Adversarial Loss into SiD. The model-based Fisher divergence, as shown in 2 is often intractable because neither 0" nor 2/" (0) are directly known. standard initial approximation, common across nearly all diffusion distillation methods, is to replace fo- with the denoising score network fe. which pretrained on data e; P and referred to the teacher. This approach hinges on the assumption that the teacher's 's score accurately represents the true data score, which could potentially create a performance bottleneck for distilled generators. Essentially. as the pretrained teacher S does not perfectly recover the true data score Sax, the objective function:
Co(a,, o, (0)) -En
(1fo((e;:) - (O)
represents only approximation of (2). meaning (x 0, 10 (0)) + Lo(as, 0 (0)). Optimiz- ing this function may not ensure that the distribution pa (Tg) matches Pdota (2o).
This discrepancy motivates the integration of Diffusion GAN-based adversarial loss to potentially correct deviations caused by inaccuracies in fa. aiming for a closer alignment between the generated and true data distributions. Specifically, while optimizing (4) does not guarantee that pe po(x,) will closely align with Pataa particularly if fa is not well-trained or has limited capacity, incorporating Diffusion GAN-based adversarial loss can enhance the match between the distributions of diffused true data and diffused generator-synthesizee fake data, thus improving the correspondence between pa)))) and paam (2o). Although introducing separate discriminator to implement the adversarial loss is possible, we find it unnecessary as we can effectively reuse the fake score network, thereby avoiding the introduction of any new parameters. This approach simplifies the model architecture and enhances efficiency, as further explained below.
Joint Score Estimation and Discrimination. While (0) is intractable and would introduce a complex bi-level optimization problem due to its dependence on the very parameter we aim to optimize- -SiD acknowledges the impracticality of obtaining either (0) or its gradient with respect to 0. Therefore, the same many previous works in diffusion distillation, SiD estimates zi* (0) using a denoising network vb, which is learned on the fake data, as
(5) where H, and x are drawn via reparameterization, as in (T). and xt) is a reweighting term that is typically set the same as the signal-to-noise ratio at time / (Ho et al.) 2020 Kingma et al. 2021 Hang (et al. 2023). This approximation introduces bias, which we will address and demonstrate how to mitigate effectively in later discussion.
ming (Z,, t) = (f) (x), t) 22
To avoid introducing any additional parameters or complex training pipelines, we incorporate a return-flag as an additional input to the network fe. offering options 'decoder', 'encoder', and 'encoder-decoder". When set to 'decoder', the network returns the denoised image as before. If set to 'encoder', it performs average pooling the output of the last U-Net encoder block of f along the channel dimension and returns a 2D discriminator map. For "encoder-decoder" the network outputs both the discriminator map and continues through the U-Net decoder blocks return the denoised image. The choice of option depends on whether we are updating the fake score network or the generator, and whether the input is real image or a generator-syntheset to extract 2D discriminator maps either under the 'encoder' option or jointly with the denoised images under the encoder-decoded option, we are now positioned define the adversarial loss function.
image. With the capability
The gradient of this loss function will either directly impact the encoder part of fø or the generator, contingent upon the optimization target-whether it is the joint loss involving fake-score estimation and discrimination or the combined loss of diffusion distillation and adversarial generation. It is also important to highlight that no additional parameters have been introduced to facilitate the joint estimation of fake scores and discrimination between true and fake images. This streamlined approach aids in maintaining model efficiency without compromising the effectiveness of the methodology.
Joint Diffusion Distillation and Ad Generation. While recognizing the impracticality of directly obtaining either (f) or its gradient with respect to 9, substituting 1 (0) in 4 with a denoising network & could introduce severely biased gradient that undermines the optimization of f. SiD addresses this challenge in gradient estimation by subtracting a biased gradient estimate, which has proven ineffective in practice, from less biased one that has shown standalone effectiveness.
5

---

Published as a conference paper at ICLR 2025
This strategy results in a loss function that significantly enhances performance, expressed as:
(sid) Co w(t) 2 0,0,a'
(6)
where (t) are weight coefficients, is bias correction weight that is typically set at 1 or 1.2, and
(sid) 0,t.a -a t) - (z,, 1) + (ff t) (a t))² (f (m,, x.y).
SiD employs an alternating optimization strategy, updating i according (5) and 0 according to 6 using only senerstor-yynthesizedd fake data, drawn as specified in ( for parameter updates. This approach ensures that both components are iteratively refined with outputs generated by the model itself, without relying on real data imputs.
An alternative theoretical justification for the SiD loss presented in (6) has been recently explored by Luo et al. (224)) demonstrating that this loss, when a = 1, can be derived under certain assumptions about the data generation mechanism, specifically involving stop-gre moop-gradeent conditions. However, regardless of the method intended to recover the true theoretical loss that would yield the SiD gradient used practice, it is essential to recognize that, as long as v (0) or its gradient-detached version is part of the intended theoretical loss formulation-incuudii model-based Fisher divergence- bias introduced by replacing 3 (0) with & remains generally unavoidable in practice. This highlights the importance of empirical validation to ensure that the bias correction strategy in place is effective.
Since fo has been adapted to serve dual purposes not only performing denoising but also capable of outputting 2D discriminator map for the encoder's latent space-we now ready to define an adversarial generation loss. This loss compels the generator to trick the discriminator map into believing its output originates from diffused real image each spatial location of this latent 2D discriminator map. Consequently, the generator will be updated using both the loss given by (6) and Diffusion GAN's s generator loss, effectively integrating these components bridge the gap between the pretrained score f& and the true score fū- and hence enhance the realism of the generated outputs.
Loss Construction that Respects the Weighting of Diffusion Distillation Loss. Building on the discussions above, are now prepared to synthesize all components into a coherent loss function. We face an additional challenge: the diffusion loss used by SiD operates at the pixel level, while the Diffusion GAN loss functions at the level of the latent 2D space. This discrepancy can make it challenging to balance the losses between diffusion distillation and adversarial generation. To cireumvent the need for dataset-specific adjustments and develop a universally applicable solution, we have devised a strategy that honors the existing weighting mechanisms used in training and distilling diffusion models. This approach involves performing GPU batch pooling to calculate an average "fakeness" within each GPU batch, then incorporating this average fakeness into each pixel's loss prior to applying the pixel-specific weights. We have found this Per-GPU-bate fakeness strategy to be highly effective, demonstrating robust performance across all datasets considered in this study,
More specifically, denoting (W, H, C.) as the size of the image, a batch inside a GPU, |B] the per GPU batch size, D) the encoder part of fu, D(x;) the 2D discriminator map of size (W', .H') given input æ, and (zx;))
[ī jl the (i. element (xx;), we define the adversarial loss that measures the average in GPU-batch fakeness
[adv)
B Em,eB In D(x.) B Em, D In Di
We now define SiDA's generator loss, aggregated over a GPU batch, as follows:
c/ (sida) (B) = Eares
w())a 20t (Asa + C.
(8)
where we set Asaa = 100 and Audv.0 = 0.01 by default, unless specified otherwise. We note that while L, is influenced by the discrepancy between the teacher network and its optimal value d²,
C c(ldv) interacts directly with the training data, helping to mitigate this discrepancy. For the fake score network fe. whose encoder also serves as the discriminator D, we define the SiDA's loss aggregated over GPÚ batch as
(sdda) V
En.eh (f (xt, t) xx 1122 >adv ui L (ath))
where the discriminator loss given a batch of fake data æ, and real data y, is expressed as
L (adv) 3)
2|B[' Ey, 8B E Ex= In (f + In(1 D(x;)li' j'I).
(10)

---

Published as a conference paper at ICLR 2025
Here, Audve e is hyperparameter that adjust to approximately balance the loss scales of both terms in the summation shown in (9). Specifically, we set Àadv, = 1 for distilling all EDM-pretraingg models and Audv, 100 for distilling all EDM2-pretrained models.
We provide an overview of the SiDA implementation in Algorithm which begins training from a pretrained EDM diffusion model. In Algorithm [ outline the modifications specific to EDM2, and in Algorithm ]] we present the SiDA algorithm that initiates training from both a pretrained diffusion model and a pre-distilled SiD generator, referred as SiD² A (SiD-initialized SiDA).
EXPERIMENTAL RESULTS
We evaluate SiDA through comprehensive experiments designed assess its effectiveness compared to the teacher diffusion model and other existing baseline methods for both unconditional and label-conditional image generation. We present results that measure both quantitative and qualitative performance across various datasets, aiming to demonstrate SiDA's capability to generate high-quality samples through efficient score distillation.
We focus distilling EDM (Karras et al. 22022) diffusion models pretrained on four distinct datasets, as well as six different-sized EDM2 (Karras et al. 2024), all pretrained on ImageNet 512 x 512. Notably, while the SiD technique (Zhou et al. (024b) has been evaluated using EDM, it has not yet been tested on EDM2. This paper makes a supplementary contribution by adapting the SiD codebase for compatibility with the EDM2 framework, which represents advanced evolution of EDM, and establishing it as a stong baseline for comparison. For detailed descriptions of the experimental setup. datasets, and evaluation metrics, please refer to Appendix AA] For details of the algorithm and modifications needed to adapt SiD and SiDA from EDM to EDM2, prlease refer to Appendix
As outlined in Tables and 7 SiDA introduces negligible memory overhead compared to SiD, owing to its design that introduces no additional parameters. Although SiDA experiences an approximate 10% reduction in iteration speed and requires real data, it converges significantly faster than SiD and consistently delivers superior performance relative to the teacher model, as demonstrated below.
5.1 ABLATION ON THE CHOICE OF a.
The gradient bias correction weight factor a is an important hyperparameter in SiD, typi- cally set to or 1.2. We conducted an ablation study assess its impact on the performance of SiDA. Specifically, we tested a spectrum of values [-0.25, 0.O, 0.5, 0.75, 1.0, 1.2, 1.5] during the distillation of the EDM model pre- trained on CIFAR-I0 (unconditional). Fig- ure 2 demonstrates how the model's perfor- mance varies with different a values over the first 20k iterations at batch size of 256, as- sessed using FID and IS. We determined that an value of 1.0 yielded the best performance, prompting us to adopt this setting for sub- sequent experiments of SiDA over various datasets. In contrast, lower a values such as -0.25 or 0 resulted in suboptimal perfor- mance, while high values like 1.2 or 1.5 led to training instability and divergence. Inter- estingly, the performance disparity for val- ues between 0.5 and 1.0 was significantly nar- rower than those observed in SiD, as depicted in Figure 8 of Zhou et al. (2024b). These findings imply that although the adversarial loss may introduce some instability into the distillation process, it could also compensate for the inadequate correction of gradient biases associated with smaller values.
Ablation Study of distilling EDM model pretrained on CIFAR-10 10 (unconditional): Each plot illustrates the relation between the performance, measured by FID (lefi) and Inception Score (right) vs. the number of training iterations during distillation, across varying values of a. The batch sizer is 256. The study underscores the im- pact a on both training efficiency and generative fidelity, lcading us to select a = 1.0 for subsequent experiments.
Figure 2:
Figure 3: Analogous plot for SID?A, SiD initialized SiDA modcl. We observe trajectory of improvements in SiD² A following an initial warmup period, and more robust performance relative to a choice.
7

---

Published as a conference paper at ICLR 2025
Fumily
(.)
Teacter
VP.EDM
Rans et 5r02
1.97
9:66
9:46 K1
BigGAN
2019
D953
YDM
2122
1000
4.00
2.65
gencratios
Myy::AAN24AAA
StyLCiAN
2020
4.17
GET-Bise (Gep
TDPM
100
2.63
0.34
Diff-inemet
VP TDDMTEOOOYR 203 127A7 <10 50046
1888
9:84
Mtyl:CAN24427
20230
SoleUANRAAD4
DMDD etaal
(003h)
2.54
DMD
Kt.
Vinerl
<-Tddap Soyta (1
2.1
9776
DMD 1r0d, XOX
5.58
Distillation
CTM Kim ctal
1.73
Oac Stop
CTM Doneent 20
DMD Yossl K302
SodeGAM2
2.71
3.77
1.06
GDD-1 Zining A Taag 2024
SiD 1:0
SID 1.2
1952 #: 000
1.700 +: oon
1.44
SIDA. 1.0
1.456 ore
S13
1.923
Lonn
SID
1.113 0003
SIIA o
1,808
10,335
SID"A,
1.396
SUFA, SDD
1,49 smD
1,519
Table 2: Analogous to Table | for CIFAR-IO (condi- Table I: Comparison of unconditional generation on tional). "Direct generation" and "Distillation" meth- CIFAR-10. The best one/few-step generator under the ods presented in the table requires single NFE, and the teacher requires 35 NFE.
FID or IS metric highlighted with bold.
140
SD a.
SD 2.0
1-
SD
20
20
0
10:
15
SIDA p-
1-
SI'AA 1.2
SIDA a -1.
SID'A c-
SID
SID 1-
SID"A
2.68
2.6
1.92
1.75
10
10'
10'
10
10
10
307
10'
30
10a
107
107
101
10
10
10h
Iterated kiimages
Iteratd images
Iterated
Figure 4: Evolution of FIDs for the SiD and SiDA generator during the distillation of the EDM teacher model pretrained on CIFAR- 10 unconditional (leff) conditional (middle), with a batch size 256, and on ImageNet 64x64 (right) with a batch size of 8192, using a = 1.0 or a 1.2. The performance of EDM (35 NFEs), along with DMD and Diff-Instruct, is depicted with horizontal lines in purple, green, and red, respectively.
In alternative setting, referred to as SiD (SiD-initializee SiDA), where the generator GB is initialized using the best available pre-distilled SiD generator, we observe robust and stabilized training across all choices of a. This suggests that initializing with the pre-distilled SiD generator provides a more advantageous starting point to achieve enhanced training stability and generative quality. Following these observations, and given that 1.0 and 1.2 were also utilized in SiD experiments, we have selected these same values for subsequent experiments with SiD for EDM.
5.2 BENCHMARKING EDM DISTILLATION
We begin by assessing the performance of SiDA and SiD? A, focusing on both generation quality and convergence speed across different datasets. Our comprehensive evaluation compares SiDA and SiD? A to leading deep generative models, including GANs, diffusion models, their distilled counterparts, and various variations, focusing on generation quality. Qualitatively, random images generated by SiD? A in single step displayed from Figures to i5 in the Appendix.
Quantitatively, on CIFAR-10 in both conditional and unconditional settings, SiDA and SiD² A outperform leading models, with details in Tables T and 2 Notably, SiDA achieves an FID of 1.516. outperforming all baseline models. When leveraging the one-step generators already distilled by SiD for initialization, SiD? A further advances this performance, achieving exceptionally low FID of 1.499 on CIFAR-10 unconditional with a = 1.0, and 1.396 on CIFAR-10 conditional with a = 1.2.
We further validate the effectiveness of SiDA and ? A the ImageNet 64x64 dataset. Here, SiDA demonstrated superior performance over SiD, achieving an FID of 1.353. Notably, by leveraging the advanced capabilities of the SiD one-step generator, SiD*A attained an unprece- dented low FID of 1.110 on ImageNet 64x64 with a = 1.2, surpassing the teacher model's performance by a large margin. On the FFHQ 64x64 and AFHQ-v2 64x64 datasets, known for human and animal faces, SiDA and SiD?A again prove superior. The datasets' less di- verse patterns do not diminish the models" performance, with SiDA and SiD? A maintaining
8

---

Published as a conference paper at ICLR 2025
faster convergence and surpassing the teacher model rapidly. The clear improve- ments across diverse datasets underscore their effectiveness.
Table 3: Analogous to Table I conditioning, using a pretrained EDM diffusion model as the teacher. The Precision and Recall metrics are also included.
ImageNet 64x64 with label
Fasuly
Model
NFE
FiD
Prou.(7)
Rex (T)
It's worth noting that SiDA's close competi- tor for ImageNet 64x64, GDD-I (Zheng & Yang 2024). uses a discriminator inspired by Projected GAN (Sauer et al. 202T). in- tegrating features from VGG 6-N and EfficientNet-lited Concerns arose, as dis- cussed in |Kynkäänniemi et al. (2023) and
VP-UUM
0.67
RIN
3000
1,23
DDP Cwr.iiiieee (on
250
11,00
0.67
0.58
AIM
(Ntemny 1iti Hol
359
0.83
Dillasoos
HSIVI
Xb1 19
U-VIT
Bacoc1 2014
50
4.25
DNF Bugtler 033
250
2.91
LXG0
(xtriy 11 1000ay
250
2.16
Coniitaayy
(Cl
Ster
2013
3.25
0.72
0.63
PD Saligass 5m A004
595
0.63
0.65
Song & ( about the poten- tial for encoders pretrained on ImageNet to
Galem (AteriELE Bya (xell)
DENo 2003
7.83
0.1
inadvertently enhance FID scores through feature leakage. This potential issue does not affect SiD, SiDA, SiD?A, A, ensuring our results reflect true model performance without such confounds.
CD-LIIh 131 11 2014
4.70
0.69
0.54
DMF-
23
DMD
Distilation
CTM Winrers 200m
CTM
Nissee 2019
1.75
192
0.57
0.57
GDD PEAA Yran 2014
1.15
0.75
0.50
PM
1.28
SID 1.0
2622 + 0034
0.73
0.65
SiDA also demonstrates accelerated con- vergence, evidenced in Figures 4 and 6 The logarithmic scale plots show SiDA's FID decreasing more rapidly than SiD's, an efficiency that also holds when scaled up to ImageNet 64x64. Overall, SiDA consistently outperforms SiD in terms of convergence speed by a substantial margin, marking significant acceleration, partic- ularly noteworthy since SiD already converges at an exponential rate. This rapid improvement in both speed and performance underscores SiDA's effectiveness and efficiency in generating high-quality images across different datasets. These results not only highlight SiD A's enhanced image quality and reduced iteration needs but also the overall efficiency of SiDA and SiD?A A across various datasets, confirming the advantages of these advanced distillation techniques.
SID o = 1.2
1.524
0009
0.24
0.53
SIDA, 6 (o0))
0:34
1.00)00)))
5.3 BENCUMARKING EDM2 DISTILLATION
We evaluate the effective- ness of SiD, SiDA, and
Table 4: FID scores and number of parameters for various methods on ImageNet 512x512 are presented. The results for EDM2 are sourced from Karras al. (0044). while those for sCT and sCD are obtained from Lu Song (2024).
SiD² A in distilling EDM2 models of six different sizes, all pretrained on Im- ageNet 512x512. Our find- ings and comparisons are detailed in Tables 4
Method
CPG
NEE
(125M)
((2000))
M (498M)
((77))
XL. (1.18)
XXL.
63
3,53
2.25
1.96
EDM2
63x2
291
2.25
2,01
1.88
1335
10:13
3.84
4.33
SCT
9.85
5.53
4.65
3.73
3.76
5 and
SCD
3,07
1.75
2.55
2.40
2:28
<CD
2,50
2.26
2.04
1.93
1.58
an depicted in Figure 5
SID
3,353 8544
2.707
9254
2,050
0285
1.907 0005
1,388 2:000
19969
The rapid convergence of the algorithms is visually illustrated in Figures g through 10
SIDA
2.228
825
1,756
1,516
0125
1.501
1.126
2-0015
1.503
SID-A
2.156
1.669
1,488
1.413
1,379
1.366
Random images generated by SiD A in a single step are displayed in Figures 16 through 34 Notably, SiD of Zhou et al. (2024b), the baseline single-step algorithm on which SiDA is based, already performs comparably to the EDM2 teacher, which requires 63 NFEs, and to the stable and scalable consistency distillation (sCD) method of Lu & Song (2024) that uses 2 NFEs-a a concurrent development to SiDA. Moreover, SiD convincingly outperforms sCD with 1 NFE, even without using CFG during distillation. As indicated in Table 5 SiD, operating with a single step and without CFG in a data-free setting, ranks among the top performers of all generative models reported on ImageNet 512x512, This state-of-the-art performance provides a robust foundation for both SiDA and S which will incorporate real data in their processes.
In line with these distillation results for the EDM family, both SiDA and SiD A show superior performance over SiD in distilling EDM2, achieving FIDs of 1.756 and 1.669, respectively, with the EDM2-S model, which has only 280M parameters. These scores already outperform the EDM2-XXL model and sCD at 1.5B parameters, whose FIDs exceed 1.8. Remarkably, when scaled to EDM2-XL and EDM2- XXLL, A achieves record-low FIDs of 1.379 and 1.366, respectively, significantly surpassing previous methods, whether single or multi-step.
9

---

Published as a conference paper at ICLR 2025
Table 5: Performance comparison of generative models trained on ImageNet at 512x512 resolution (Left:
without Classifier- Free Guidance (CFG); Right: with CFG). The results for EDM2 of six different sizes and their distillations using sCD, SiD, and SiDA algorithms are provided in Table
Method (CFG=N)
NFE (1)
4Params
FID (1)
Method (CPG=Y)
NFE
4Params
FID (
ADM-G
Diarisal & Nichol b021b
250
559M
23.21
ADM-G
(Diuriwal & Nichol
2021b
250x2
5559M
7.72
U-DIT-B
Tam N07410
250
204M
15.39
U-VIT-L/
250+2
287M
DIT-XI.22
Ald
250
675M
12.03
U-VT--1
BaR cral 2023
250x2
50IM
4.05
MaskDiT
Zhempp mi 217240
79
736M
10.79
ADM-U (Dianea) & Niceo
20216
250x2
2-55M
3.85
ADM-U
Diunnaa Niclm 2021
250
2-55M
9.96
LEGO-X
250x2
681M
3.99
LEGD
250
9.01
DIM-H
250+2
860M
160M
8.43
LEGO-XL-UG
2024b
250x2
681M
3.74
DiMR-XLAR
250
525M
7,93
DyDiT-XL Zhai sir N074
100x2
675M
3.61
MaskGIT (Chali eraln
2022
12
227M
7.32
DIRISSM MMMM--XLEE Tair mi
20144
250x2
660M
3.41
MAGVIE-2
12
307M
1.61
DIT-XL22 (Pooble'
250x2
675M
3.04
FUAeS
1000
320M
DRWKV
250x2
779M
2.95
DIREERI 20733
64
307M
3.07
DIMR-XI /AR Traneo mi 2U2A
250x2
525M
2.89
VDM (Kinbmir A Cimn 2012
512
28
2.99
DIS-H/2 (Fei ol 201
250x2
900M
2.88
MAR (Lraral 2074]
100
481M
2.74
DiffT (Hmmmtza
2025
250x2
561M
2.67
StylefAAN-XI:
2022
2.41
VDM++ Kinnnn EN SL0
512x2
SID" ADDD--- (OGES)
125M
2.156
VAR-d30 Timeee Pi1ZA
10x2
2.38
2.63
SIDAA-EDM2-S (ouns)
2HIM
1.669
SIT.LL Ma Marerra 2024
250x2
675M
2.62
ADDA-EDM2 M (ours)
498M
1.488
Larp--DiT-ssal
ctal 2023
250x2
38
2.52
SID" A-EDM2-L (ours)
777M
1.413
MaskDiT-G 2bent P004
79x2
736M
SIr? A-FM----XXI (aurs)
1.18
1.379
MAGVIE-2Z YOR 20230
64x2
307M
1.91
XXL (ours)
1,5B
1,366
MARE Li
100x2
181M
1.73
(4) EDM2-XX
(b) EDME-S
(c) EDME-M
(4) EDM2-1
(e) EDM2-XI
( EDM2-XXI
Figure 5: Evolution of FIDs for the SiD, SiDA, and SID A generators during the distillation of the EDM2
teacher models of six different sizes pretrained on ImageNet 512x5 12, with a batch size of 2048 and a 1.0.
The performance of EDM2 without classifier- free guidance (CFG) and EDM2 with CFG, using 63 NFEs, along with the simple, stable, and scalable consistency distillation (sCD) method Lf & Song (2024) with and 2 NFEs, depicted with horizontal lines in purple, red, green, and cyan respectively.
Mirroring findings from EDM distillation, SiDA also exhibits accelerated convergence when distilling EDM2, as shown in Figure 5 Logarithmic-scale plots show that SiDA achieves an order of magnitude faster decrease in FID compared to SiD, with efficiency that scales effectively to ImageNet 512x512 12 and across generators of varying sizes. For instance, while SiD required training the generator with approximately 100M generator-synthesiiie SiDA needed only 10M such images to exceed the teacher's 's performance. Overall, while SiD proves to be strong baseline, rivaling the performance of the teacher and methods published subsequently, SiDA consistently surpasses both the teacher and SiD by large margins in terms of final metrics and convergence speed. This rapid improvement underscores SiDA's effectiveness and efficiency in generating nighreesolution images in a single step across various model sizes.
images to match or surpass the teacher's s performance,
6 CONCLUSION
In this paper, we present Score identity Distillation with Adversarial Loss (SiDA), an innovative framework that optimizes diffusion model distillation by integrating the strengths of Score identity Distillation (SiD) and Diffusion GAN. SiDA accelcrates the generation process, significantly reducing the number of iterations required compared to conventional score distillation methods, while pro- ducing high-quality images. Extensive tests across various datasets CIFAR-10 10, ImageNet (64x64 and 51 512x512), FFHQ, AFHQ-v2-and that SiDA consistently surpasses both the base teacher diffusion models and other distilled variants. This is validated by superior Fréchet Inception Distance scores and other performance metrics. No- tably, SiDA achieves these results without utilizing classifier free guidance (Ho & Salimans 2022). suggesting potential for further enhancement with its integration. Specifically, incorporating the long-and-short guidance strategy as described by Zhou et al.] (2024a) could further refine SiDA's performance. Future work will explore expanding SiDA's methodology to additional generative tasks and refining its capabilities through innovative distillation techniques.
different scales of pretrained EDM2 models demonstrate
https://iihhahcco//A/eeee VLJ MIJ www.--AArrrryyhlllnn argo-DTT Natassaanatssssaiiille... png.
MAR also requires 256 stoesiiee steps before per parforming revens diffasion
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
M. Zhou, H. Zheng, Y. Gu, and Z. Wang acknowledge the support of NSF-IIS 2212418 and NIH-R37 CA271186.
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In
Proceedings of the 34th International Conference on Machine Learning-Voin Volume 70, pp. 214-223, 2017.
Fan Bao, Chongxuan Li, Yue Cao, and Jun Zhu. All are worth words: A ViT backhone for score-based diffusion models. arXiv preprint arXiv:2209. 12152, 2022.
Fan Bao. Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li. Hang Su, and Jun Zhu. All are worth words: A VIT backbone for diffusion models, In Proceedings of the IEEE/CVF conference computer vision and pottern recognition, pp. 22669-22677 2023.
David Berthelot, Arnaud Autef, Jierui Lin, Dian Ang Yap. Shuangfei Zhai, Siyuan Hu, Daniel Zheng, Walter Talbott, and Eric Gu. Tract: Denoising diffusion models with transitive closure time- distillation. arXiv preprint arXiv:2303.000 04444888,. 2023.
Christopher M Bishop. Patten Recognition and Machine Learning. springer, 2006.
D. M. Blei and M. Jordan. Variational inference for Dirichlet process mixtures. Bayesian analysis, 1(1): 121-143, 2006.
David Blei, Andrew Y Ng, and Michael Jordan. Latent dirichlet allocation. Journal of Machine Learning Research, 3:993-1022, 2003.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL
net/forum i d-Bl xs9m09Fm
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. MaskGIT: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315 11325, 2022.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-War Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 8188-8197, 2020.
Wang Chong, David Blei, and Fei-Fei Li. Simultaneous image classification and annotation. In 2009 IEEE Conference on computer vision and pattern recognition, pp. 1903-1910, IEEE, 2009.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794 2021a.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs image synthesis. In A. Beygelz- imer, Y. Dauphin. Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, 2021b. URL BttLttttp///ppeeeeee//
net /tt/:/ooommmdaaaacaaaea
Zhengcong Fei, Mingyuan Fan, Changqian Yu, and Junshi Huang. Scalable diffusion models with state space backbone. arXiv preprint arXiv:2402.05608 2024a.
Zhengcong Fei, Mingyuan Fan, Changqian Yu, Debang Li, and Junshi Huang. Diffusion- Scaling RWKV-like architectures for diffusion models. arXiv preprint 1.Xx::2444.0447777 2024b.
Li Fei-Fei and Pietro Perona. A Bayesian hierarchical model for learning natural scene categories. In 2005 IEEE computer society conference computer vision and puttern recognition (CVPR '05), volume 2, pp. 524-531. IEEE, 2005.
Zhengyang Geng, Ashwini Pokle, and J Zico Kolter. One-step diffusion distillation via deep equilibrium models. Advonces Neural Information Processing Systems, 36, 2023.
11

---

Published as a conference paper at ICLR 2025
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu. David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances Neural Information Processing Systems, pp. 2672-2680, 2014.
lan Goodfelllw, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://wwa
(eeplearningbook.org
T. L. Griffiths and Z. Ghahramani. Infinite latent feature models and the Indian buffet process. In NIPS, 2005,
Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, and Joshua M Susskind. BOOT: Data-free distillation of denoising diffusion models with bootstrapping. In ICML 2023 Workshop Structured Probabilistic Inference and Generative Modeling, 2023.
Tiankai Hang. Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. arXiv preprint urXiv:2303.09556 2023.
Ali Hatamizadeh, Jiaming Song, Guilin Liu, Jan Kautz, and Arash Vahdat. Diffit: Diffusion vision transformers for image generation. In European Conference on Computer Vision, pp. 37-55. Springer, 2025.
G. Hinton, S. Osindero, and Y.W.. Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18 (7):157--1554, 2006.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14 19::1771-1000 2002.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:200.122500 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. Advances in Neural Information Processing Systems, 33, 2020.
Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. The Journal of Machine Learning Research, 14(1):1303-1347 2013.
Allan Jabri, David Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. arXiv preprint arXiv:2212.J 1972, 2022.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference computer vision and pottern recognition, pp. 4401 4410, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lchtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In Proc. CVPR, 2020.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion based generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.). Advances Neural Information Processing Systems, 2022. URL htttss://openeee www.reeewwweeeeeeeeeeee. www.net/forrmmm
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics diffusion models. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 24174 241844, 2024.
Dongjun Kim, Chich-Har Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279, 2023.
Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the ELBO with simple data augmentation. In seirth-seentth Conference on Neural Information Processing Systems, 2023. URL https::/<penrett
net
Diederik P. Kingma Max Welling. Auto-cncoding variational Bayes. In International Conference on Learning Representations, 2014.
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models, arXiv preprint arXiw:2107.00630, 2021.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko I ehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2025
Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes fréchet inception distance. The Eleventh International Conference on Learning Representations. 2023. / net/ http://wwwwww.ooooooooooooooooo. 2ttee//oooummmmodooX66666
Chun-Liang Li, Wei-Cheng Chang. Yu Cheng. Yiming Yang. and Barnabás Póczos. MMD GAN: Towards deeper understanding of moment matching network. In Advances in Neural Information Processing Sysiems, pp.
2017.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406. 11800611886... 1838, 2024.
Qihao Liu, Zhanpeng Zeng, Ju He, Qihang Yu, Xiaohui Shen, Liang-Chich Chen. Alleviating distortion in image generation via multi-resolution diffusion models and time-dependent layer normalization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https
wwt.e/eww.eeeeeeeeeeeeeeeeeeeeee //apenreview net wtte://oouuuddd=3Jwtte...
Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, and Qiang Liu. Instaflow: One step is enough for
high-quality diffusion-based lext-lo-image generation. arXiv preprint arXiv:2309.06380 2023.
Cheng Lu and Yang Song. Simplifying. stabilizing and scaling contimuous-time consistency models. arXiv preprint arXiv:2410. 1081, 2024.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfci Chen, Chongxuan Li, and Jun Zhu. DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (cds.), Advances Neural Information Processing Systems, 2022. URL https://cppereviiee net. forumforrmir 2uuummm 2wAaGwlP
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2/01.02398, 2021.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li. and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. ArXiv, abs/23 10.04378, 2023a.
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff-Instruct: universal approach for transferring knowledge from pre-trained diffusion models. In shirty- Conference on Neural Information Processing Systems, 2023b. URL [tttps://openrevieet /
Forum? MatimiidddMMiss iig4
Weijian Luo. Zemin Huang, Zhengyang Geng, Zico Kolter, and Guo-jun Qi. One-step diffusion distillation through score implicit matching, arXiv preprint arXiv:241016699, 2024.
Zhaoyang Lyu, Xudong Xu, Ceyuan Yang, Dahua Lin, and Bo Dai. Accelerating diffusion models via early stop of the diffusion process. arXiv preprint arXiv:2205. 12524, 2022.
Nanye Ma, Mark Goldstein, Michael Alhergo, Nicholas M Boffi, Eric Vanden- Fijnden, and Saining Xie. SiT: Exploring flow and diffusion-based generative models with scalable interpolant transformers. arXiv preprint 21Xi::2401.087400 2024.
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma. Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14297-14306, 2023.
Kevin P Murphy. Machine Learning: Probabilistic Perspective. MIT Press, 2012.
Thuan Hoang Nguyen and Anh Tran. SwiftBrush: One-step lext-to-image diffussin model with variational score distillation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMI.R, 2021.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, pp. 16784-16804. PMLR, 2022.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision, arXiv preprint arXiv:2304.07199 2023.
13

---

Published as a conference paper at ICLR 2025
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminaraynnan. Normalizing flows for probabilistic modeling and inference. arXiv preprint arXiv:1912.02760 2019.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:22/2.09744, 2022.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 4205. 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024. URL http::://ppeeeeeeieiee////////
net / www.umriidiia..... 522R8xqf
Ben Poolc, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. DrcamFusion: Text-to-3D using 2D diffusion. In The Eleventh International Conference on Learning Representations, 2023. URL httphhss///ppeeeeii.et net fet//form?20 id=F 7Vyy
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125 2022.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International conference on machine learning, pp. 1060-1069, PMLR, 2016.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. In Proceedings of the 31st International Conference on Machine Learning, pp. 1278-122 1286, 2014.
Robin Rombach, Andreas Blattmann, Dominik Loreaz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10684-10660 10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-3366 364944, 2022.
Ruslan Salakhutdinov and Geoffrey Hinton. Deep Boltzmann machines. In Artificial intelligence and statistics. pp. 448-455 PMLR. 2009.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In Interna- tional Conference on Learning Representations, 2022. URL htt.://ppss//pppeeeeeeee.eeeeeee
TIdIXIpzhoI
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In Advances in Neural Information Processing Systems, pp. 2234-2242 2016.
Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. Advances in Neural Information Processing Systems, 34:17480-17497 2021.
Axel Sauer. Katja Sehwarz, Andreas Geiger. StyleGAN-X, Scaling StyleGAN to large diverse datasets. In ACM SIGGRAPH 2022 Conference Proceedings, pp- 1-10, 2022.
Axel Sauer, Dominik Lorenz, A. Blattmann, and Robin Rombach. Adversarial diffuusio distillation. ArXiv, his/2311.170444 2023.
Chengehao Shen, Youtan Yin, Xinchao Wang, Xubin Li, Jie Song, and Mingli Song. Training generative adversarial networks in stage. Proceedings of the TEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 3350-3360. 2021.
Jascha
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International
Conference on Learning Representations, 2020.
14

---

Published as a conference paper at ICLR 2025
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint arXiv:2310.14100 14199 2023.
Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribution, In Advonces Neural Information Processing Systems, pp. (1918-1193, 2019.
Yang Song. Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint
0:Xiv:2303.01465 2023.
Joshua Tian Jin Tee, Kang Zhang, Chanwoo Kim, Dhananjaya Nagaraja Gowda, Hee Suk Yoon, and Chang D. Yoo. Physics informed distillation for diffusion models, 2024. URL htt.ts::///poeeeeeeeetee eew,, net./
Yao Teng, Yue Wu, Han Shi, Xuefei Ning, Guohao Dai, Yu Wang, Zhenguo Li, and Xihui Liu. DiM: Diffusion mamba for efficient high-resolution image synthesis. arXiv preprint arXiv:2405. 14224, 2024.
Keyu Tian, Yi Jiang. Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.02900 2024a.
Yuchuan Tian, Zhijun Tu, Hanting Chen, Jie Hu, Chao Xu, and Yunhe Wang. U-DiTs: Downsample tokens in u-shaped diffusion transformers. arXiv preprint arXiv:2405.022 027300 2024b.
Pascal Vincent. A Connection Between Score Matching and Denoising Autoencoders, Neural Computation. 23 (7):1611-1674, 2011.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research, 1 (1())), 2010.
Zhendong Wang. Yifan Jiang, Huangjie Zheng, Peihao Wang. Pengcheng He, Zhangyang Wang. Weizhu Chen, and Mingyuan Zhou. Patch diffusion: Faster and data-efficient training of diffusion models. arXiv preprint arXiv:2304.12522 2023a.
Zhendong Wang, Huangjie Zheng, Pengeheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion Training GANs with diffusion. In The Eleventh International Conference on Learning Representations, 2023b. URI.
htttp:://ppereeee
net. for.mm d f7 UbbPHHAA
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. PromificDreamer: High-fidelity and diverse text-to-3D generation with variational score distillation, 2023c.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denois- ing diffusion GANs. In International Conference Learning Representations, 2022. URL attps / //openrevien pett/forum?1ddujrropp
qoCoo
Sirui Xie, Zhisheng Xiao, Diederik P Kingma, Tingbo Hou, Ying Nian Wu, Kevin Patrick Murphy, Tim Salimans, Ben Poole, and Ruiqi Gao. EM distillation for one-step diffusion models, 2024. URL https
//arxiv.orq abs/ 2405. 16852
Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. UFOGen: You forward once large scale text-to-image generation via diffusion GANs. ArXiv, abs/231 0722311.0990 1.09257, 2023.
Jing Nathan Yan, Jiatao Gu, and Alexander M Rush. Diffusion models without attention. In Proceedings of the JEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8239-8249, 2024.
Mingzhang Yin and Mingyuan Zhou. Semi-implicit variational inference. In International Conference on Machine Learning, pp. 5660-5669, 2018.
Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T Free- man. Improved distribution matching distillation for fast image synthesis. arXiv preprint arXiv:2405.i 1487. 2024a.
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6613-6623, 2024b.
Lijun Yu. José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng. Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, al. Language model beats tiffussion-tneenizrr is key to visual generation. arXiv preprint arx0v:2310.05777, 2023a.
15

---

Published as a conference paper at ICLR 2025
Longlin Yu, Tianyu Xie, Yu Zhu, Tong Yang. Xiangyu Zhang, and Cheng Zhang. Hierarchical semi-implicit vari- ational inference with application diffusion model acceleration. In sevent- Conference on Neural In- formation Processing Systems, 2023b. UtL/tttps://ppeeeeeeee http://wwww...toooooooooooooooo.
Renrui Zhang. Jiaming Han, Chris Liu. Peng Gao, Aojun Zhou, Xiangfei Hu. Shilin Yan. Lu. Hongsheng Li, and Qiao. LLaMMAAdaater: Efficient finetuning language models with zero-init attention. arXiv preprint arXiv:2303. 16199, 2023.
Wangbo Zhao, Yizeng Han, Jiasheng Tang, Kai Wang. Yibing Song. Gao Huang, Fan Wang, and Yang You. Dynamic diffusion transformer. arXiv preprint arXiv:2410.03456, 2024.
Bowen Zheng and Tianming Yang. Diffusion models are innate one-step generators. arXiv preprint arXiv:2405. 20750, 2024.
Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. In International Conference on Machine Learning, pp. PMLR, 2023.
Hongkai Zheng, Weili Nic, Arash Vahdat, and Anima Anandkumar. Fast training of diffusion models with masked transformers. Transactions Machine Learning Research, 2024a. ISSN 2835-8856. URL
https:///pperreview net fort//frrmmddd// B
Huangjic Zheng and Mingyuan Zhou. Exploiting chain rule and Bayes' theorem to compare probability distributions. Advances Neural Information Processing Systems, 34:14993-15000 2021.
Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusion probabilistic models. arXiv preprint arXiv:2202.09671 2022.
Huangjic Zheng, Zhendong Wang. Jianbo Yuan, Guanghan Ning. Pengeheng He, Quanzeng You, Hongxia Yang, and Mingyuan Zhou. Learning stackable and skippable LEGO bricks for efficient, reconfigurable, and variable-resolution diffussion modeling. In The Tweifth International Conference on Learning Representations, 2024b. URL /ttp://///eeeeeeeeeeeemmmo net / forum? ? Forummid-ggXeddeee d-ggmedwwwww"
Mingyuan Zhou, Haojun Chen, John Paisley, Lu Ren, Guillermo Sapiro, and Lawrence Carin. cara-eraectiic Bayesian dictionary lcarning for sparse image representations. Advances in neural information processing systems, 22, 2009.
Mingyuan Zhou, Zhendong Wang, Huangjie Zheng, and Hai Huang. Long and short guidance in score identity distillation for one-step text-to-image generation. arXiv preprint 24Xx::2406.01560 2024a.
Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang. Score identity distilla- tion: Exponentially fast distillation of pretrained diffusion models for one-step generation. In Forty-first International Conference on Machine Learning, 2024b. URL ftttp:://///tt. ptttps://openreviiee http:/.ee/oooooo fer/forumm id=OhaQJqeONq
16

---

Published as a conference paper at ICLR 2025
Appendix for SiDA
A EXPERIMETAL SETTINGS
Implementation Details. We implement SiDA using the SiD codebase (Zhou et al. 2024b).
incorporating essential functions adapted from both EDM (Karras et al.. 2022) and EDM2 (Karras et al. 2024). Detailed adaptations of EDM2 modifications within SiD and SiDA are provided in Appendix B.2 specifically addressing our handling of forced weight normalization and data-specific weighting for the loss terms introduced by EDM2 to enhance EDM. We did not incorporate classifier- free guidance (Ho & Salimans, 2022p. use two teacher models, or set exponential moving average (EMA) parameters post-hoc, all of which are important for EDM2's performance improvements over EDM. Even without these techniques, SiD of Zhou et al. (2024b) already performs on par with EDM2 teacher models, while SiDA and SiD A significantly outperform the teacher. These additional techniques could potentially further enhance SiD and SiDA, which we leave for future work.
The score estimation network, fw. is initialized using the same architecture and parameters as the pre-trained teacher score network, So. from either EDM or EDM2. For the initialization of the student generator, Go. we explore two settings: one where Go is initialized with the same parameters as the pre-trained EDM or EDM2 teacher score network, and another where it is initialized using the distilled generator from SiD, referred to SiD-SiDA (SiD² A). The hyperparameters customized for our study are outlined in Table 7 with all remaining settings consistent with those in the EDM code (Karras et al., 2022), EDM2 code (Karras et al. 2024), and SiD code (Zhou al. 2024b).
Our training process unfolds in three stages: 1) For the first 100k images, we exclusively train fu to stabilize the score estimation. 2) For the subsequent 100k images, both fo and G5 are trained using the SiD generator loss (6), integrating the distilled insights. 3) For all remaining images, we update fe and Go with the SiDA generator loss B8, which incorporates Diffusion GAN-based adversarial adjustments to refine the generation process. Throughout the training, the score estimation network fø is consistently trained with the SiDA fake score loss (). ensuring that the adversarial components are properly integrated.
We observe that using only the adversarial loss, without the SiD component in the combined loss, leads to rapid divergence. even when the single-step generator is initialized from SiD. This observation suggests that while the added adversarial components cannot stand alone, they effectively complement the SiD loss, which operates in a data-free manner, allowing it to leverage available training data to correct the teacher's s bias and thus enhance the distillation speed and performance.
Parameter Settings. To balance the SiD) and adversarial components in both the generator and fake-score losses, we set Asid Addv,a Aadv, ø to align the scales of the four different losses. This keeps them approximately within the range of 1 to 10,000 to prevent fp16 overflow/underflow. The settings in Tables 66 and 7 achieve this. For EDM distillation, we reused the SiD parameters. For EDM2, matched the batch size to the teacher's s and adapted A from 1 to 100 to roughly equate the adversarial loss scale with the diffusion distillation loss for the fake score network. Initially, we used learning rate of 4e-6 but switched 5e-5 for faster convergence in EDM2-XS. Our experience indicates that SiD and SiDA are not overly sensitive to these settings as long as loss scales are comparable, suggesting robustness. While fine-tuning these ayper-parameters could potentially further improve outcomes, our focus wasn't on hyperparameter optimization given the already achieved thet-e- performance across various settings.
Datasets and Teacher Models. To comprehensively evaluate the effectiveness of SiDA, we first utilize four representative datasets of varying scales and resolutions and the EDM diffusion models pretrained on them, as discussed in Karras et al. (2022). These datasets include CIFAR-10 (32 x 32; both conditional and unconditional) (Krizhevsky et al. 2009), ImageNet 64x64 (Deng et al., 2009). FFHQ (64 x 64) (Karras et al.) 2019), and AFHQ-v2 (64 x 64) (Choi et al. 2020).
Additionally, we distill EDM2 diffusion models of varying sizes that were pretrained on ImageNet 512x512 12, following details provided by Karras et al. (2024). We compare these distilled models to their corresponding teacher models and to a concurrent work of Lu & Song (004) on EDM2 distillation. The models range from 125M million to 1.5 billion parameters. By utilizing these diverse datasets and model scales, we thoroughly assess the performance of SiD, SiDA, and SiD? A across different content types and complexities, ensuring robust evaluation of their generative capabilities.
17

---

Published as a conference paper at ICLR 2025
Table 6: Hyperparameter settings SiDA comparison distillation time, memory usage between SiD and SiDA, using 8 NVIDIA V100 GPUs (16 GB each) or NVIDIA H100 GPUs (80 GB each).
Methed
Hyperpurameters
CIFAR-10 32x32
ImageNet 64x64
FFHQ 64x64
AFHQ-v2 64x64
Batch size Butch size per GPU GPUs
256
8192
512
512
32
32
64
8xV100
8xH100
8xH100
8xH100
Grudient accumulation round Learning rate of (v), A)
32
1e-5
4e-6
le-5
5e-6
Loss scaling (Aas. Autv.ts Asdr.e)
(100, 0.01, 1)
0.5
fp16
False
True 1c-6
True
Thue
Optimizer Adam (cps)
lc-8
le-6
lc-6
Optimizer Adam (3,)
Optimizer Adam (32)
0,999
1.0 for SiDA; both 1.0 and 1.2 for SIDPA
2.5
Emas
augment dropout,
The in EDM and SiD for each corresponding dataset
memory allocated per GPU
13.0
46.7
31.1
31.1
max memory in GB reserved per GPU
13.3
48.0
31.3
31.3
SiD
-ecconds per Ik images
3,3
3.1
1.1
1.1
houur per 10M (10'k) images days per 100M (10%k) images
9.2
memory allocated per GPU max memory in GB reserved per GPU
13.0
13.4
48.1
31.3
31.3
SiDA
resecons per 1k images
3.6
3.5
1.2
1.2
~ hours per 10M (10*k) images days per 100M (10%k) images
10.0
9.7
3.3
3.3
4.2
Table 7: Hyperparameter settings and comparison of distillation time, and memory usage between SiD and SiDA on distilling EDM2 pretrained on ImageNet 512x512, using 16 NVIDIA A100 GPUs (40 GB each) or 8 NVIDIA H100 GPUs (80 GB each).
Method
Hyperparameters
EDM2-XS
EDM2-S
EDM2-M
EDM2-L
EDM2-XL
EDM2-XXL
Batch Batch size per GPU GPUs
64
64
32
32
16
2
16xA100
8xH100
8xH100
8xH100
8xH100
8xH100 128
Gradient accumulation
Learning rate of (v) !)
5e-5 (100, 0.01, 100)
Loss scaling (Aads Auti.:: A
fp16
True
Optimizer Adam (8,, B2. cps)
(0, 0.999,9... Ic-6) 1.0
Onx
2.5
fmax
dropout
The same EDM2 for each corresponding model size
max memory allocated per GPU
31.5
51.3
49,4
68.8
72.2
74.6
max memory in GB reserved per GPU
52.4
50.0
70.0
75.1
SID
sescondss per images
1.5
2.5
3.4
29,9
hours 10M (10") kk) images
2.5
4.2
6.9
9.4
15.8
83,1
days per 100M (10%k) images
1.1
1.7
2.9
3.9
6.6
35
max memory allocated per GPU
31.5
51.3
49.4
68.9
72.2
74.6
max memory in GB reserved per GPU
34.2
52.4
52.0
70.1
73.4
75.1
SIDA
rsscconds per 1k images
1.0
1.6
2.7
3.7
6.1
31.2
r-hours per 10M (10%) images dayy per (10%k) images
4.4
7.5
10.3
16.9
86.7
Evaluation Protocol. We assess the quality of image generation using the Fréchet Inception
Distance (FID) and Inception Score (IS) (Salimans et al.) (2016). Following Karras al.] (2024). we also evaluate models trained on ImageNet 512x512 using FD the Frechet distance computed the feature space of DINOv2 (Oquab et al. 2023) Consistent with the methodology of Karras et al. (2019) 2022 2024). these scores are calculated using 50,000 generated samples, with the training dataset used by the EDM or EDM2 teacher mode serving as the reference. Additionally, we evaluate SiDA on ImageNet 64x64 using the Precision and Recall metrics (Kynkäänniemi et al. 2019), where
https::/ whw.ggttthhhh... com NV1 wwwss/dmmmmm
18

---

Published as a conference paper at ICLR 2025
both metrics are calculated using predefined reference batch Dhariwal & Nichol 2021a Dhariwal 2021: Song et al. 20233 Song Dhariwal 2023).
Nichol
In line with SiD, periodically evaluate the FID during distillation and retain the generators with the lowest FID. To ensure accuracy and statistically robust comparisons, we perform re-evaluations across 10 independent runs obtain more reliable metrics. We recommend this rigorous approach over reporting only the best metrics observed across individual runs or during the distillation process, the latter can result in biased metrics that consistently lower than the mean.
B ALGORITHM DETAILS
B. ALGORITHM Box
Algorithm Adversarial Score identity Distillation (SiDA)
1: Input: Pretrained score network fa. generator Go, generator network fa. 2.5, tmax 800, a 1.2, Asaa 100, A 0.01, Auai,v
1, image size (W, H, Ce), latent discriminator map size
(W", .H).
2: Initialization 0 - 0.3 + Φ. D(-) - encoder(1))
3: repeat
4: Sample N(,, I) and By Go(omnzz)
5:
Sample p(t). Ci NO, and az a,Ty acCc
6:
Update vo with 9
7:
((iid)) = (f)) (x,, xz Auni, L (adr))
8:
9: where the timestep distribution & ~ p(),, noise level as, and weighting function 7(t) are defined as in
Zhou et al. (2024b).
10:
if num imgs 100K then
Sample Umif(0, /max/ 1mmmm///1000000 /1000], compute de, wt. and a as defined Zhou 2024b Set
num imgs 5200k and otherwise
13:
Update Go with 8
w(()a c/(iib)) = (5 Auii (1- 14:
[ t) (mT,
of
w())a (folwr,))
( 22)
w(t)a Audi
C.WHC
of voch
2
17:
end if
18:
until the FID plateaus the training budget is exhausted
Output: Go
Algorithm SiDA for EDM2
Extra Input: Log Var
(True, False), ForceNormeag E (True, False}, Auti.s 100
2:
repeat
3: The same as SiDA for EDM with the following modifications:
4: if Log Var is True then
5: Modify the fake score network loss as ((ida))
(t)) coaver
(Ilfa t) x0.12 + logvar + Audv. a La (all))
6: end if
7:
if ForceNormmaag is True then
8: During training, apply forced weight normalization used EDM2 for SiD, and modified version of to prevent gradient backpropagation error SiDA
9: end if
10: until the FID plateaus or the training budget is exhausted Output:
19

---

Published as a conference paper at ICLR 2025
Algorithm 3 SiD²A: SiD initilized SiDA (applicable for both EDM and EDM2)
Pretrained SiD generator Osi 2: Initialization
1: Extra Input:
Owa, v/ <- 6. D(-) <- encoder(1))
3: repeat
4: The same as SiDA
5: until the FID plateaus the training budget is exhausted
6:
Output: Go
B.2 FORCED WEIGIIT NORMALIZATION WITH PRE-HOOK
An important training technique used by EDM2 to enhance EDM is the application of forced weight normalization, where each weight vector is explicitly normalized unit variance before every training step. This method is found to be fully compatible with the training of both the generator and fake score network in SiD and does not introduce gradient backpropagation errors. However, SiDA, where adversarial loss is incorporated, we find that this approach often leads to computational errors during gradient backpropagation.
We have traced the source of the problem the MPConv class of the EDM2 code, which mod- ifies self. weight in-place during the forward pass. A possible reason that even though torch. no po_gradd () was used, in-place operations parameters that require gradients can inter- fere with PyTorch's 's autograd system, leading to errors during backpropagation.
To address these issues, provide two options:
1. Disable Forced Weight Normalization:
Completely remove the in-place normalization step from the training process. This avoids the gradient backpropagation errors but also forgoes the potential benefits of forced weight normalization.
2. Use Pre-Forward Hook to Avoid In-Place Operations: Instead of modifying self. we weight in-place during the forward pass, we utilize a pre-forward hook to normal- ize the weights before each forward pass. Executed before the forward method of a module, a pre-hook in Py allows for consistent normalization of the weights without interfering with the autograd system. Employing this strategy enables forced weight normalization without causing gradient backpropagation problems in SiDA.
By implementing the second option, we maintain the potential advantages of forced weight normal- ization while ensuring stable training with adversarial loss SiDA.
In our experiments, we assessed two options during the distillation of EDM2-XS, EDM2-S, and EDM2-M and observed no significant performance differences. For SiD, we retained the original forced weight normalization. For SiDA, we generally opted for forced weight normalization with pre- hook (option 2), except for EDM2-S and EDM2-XXL. where we used no forced weight normalization (option 1). Similarly, SiD? defaults to option 2, but switches to option I for EDM2-S. These variations in choices are not intended to maximize performance for each setting, but rather to illustrate that the choice of normalization may significantly impact results, least not impede their ability to outperform the teacher.
Another clear difference between EDM2 and EDM is the use of one-layer MLP to produce the weights for each data-specific loss term. This MLP is trained alongside the main denoising network and discarded afterward. In our experiments, we did not observe clear difference in distillation performance with or without this MLP. Therefore, we enable it by default when forced weight normalization is applied and disable it when no forced normalization is used.
20

---

Published as a conference paper at ICLR 2025
Below, we provide the code for the original and modified versions of the MPConv class.
The Original MPConv Class of EDM2 That Includes Forced Weight Normalization
éperaiatence peraistent_ccce
class MPCOnv (toroh.nn.Moouu Modull):
def Linit_ fself, in_channels, out_channels, kernel):
super -inii)i_nii_____ self.outt_hannelle out_channeis
selt.weight
randn (out_channels, in in_channels, vkernel)
def foel,, goin=11:
# telfftraining::
with torch.ne_gradd?
hea.f..wighc.cooy.mmno (x)) foroed weight normalization
normalize (x) # tradicional weight nozmal I isation
W w Igain / ap.sqrt (w[0] (aumel magnitude preserving scaling
w.to(x.dtype)
if w.ndin -- 2:
v.t()
assart w.ndim ==
return torch.nn. funct .oov22 (x, v, paddf ngo (N.-shape (-11//2,)1
Listing 1: Original MPConv class with forced weight normalization
The Modified MPConv Class Introducing an Optional Modified Forced Weight Normalization When Adversarial Loss Is Applied
Spersistence wwwpsstent__ttiieeeee
class XBConv :
Module)
_int__ (self, in_channels, out_channels, kernel, force_norma fon, use_gan)
super (ni_tit___ll self. cct..neesccaaness out_charnels
selg.weight L.r........aaaameeeee........ wtt.foorre_ommmizaaiiii. =foree_nocmalizationn
tost_chaneess, in_channels, wkernel)
self.use_gen use_gan i Register the formard pre-hosk
self.use_gan and self. wallf.ooree_oooe ftw.eeegsssee_tooeeerrereeeeeeee (self, htrp:pppppo.ooo..ooooooooooooooo
def fopply_orced_welll www.eeeeiiiiooommllaaiiiiooooooo
module, input):
i Only apply during training
celf.treining:
with torch.no_graddl
# weit. ttw.eightttoooooc...ooea..... (1eat32]
k_nermalized normalize(w)
welf.weight.copr. (k_normalized)
def (sel,, gain=l]:
x self. to (tathh.. if self.training and not self self.use_gan:
with torch.ne_gzaddt: :
self.weight htw.ghh.cccyy_toomaleeeeeeeee
forced weight normalization
x normalize (x) rraditional weight normalization lization
(gain np. (w[0] (1)) Magnitoode-preserutng inng x (x dtype)
w.ndim 21
return w.00
assert w.ndim
torch.nn. www.tiiiaan.anccommmmmmmm conv2d paddings (v.shape[-1] // 213
Listing 2: Modified MPCony class with pre-forward hook
21

---

Published as a conference paper at ICLR 2025
C
ADDITIONAL TRAINING AND EVALUATION DETAILS
50
50
SID
30
SID 12
30
SID 1-12
SID'A a- 1.0
SID²A a 1.0
SIDFA 1.2
SID²A a- 1.2
10
SIDA o 1.0
10
SIDA == 1.0
2.39
1,98
107
10
102
10
Iterated k-images
Iterated k-images
Figure 6: Analogous plot to Figure for FFHQ-64x64 with batch size of 512 (left) and AFHQ-v2-64x64 with batch size of 512 (right).
Table 8: Analogous Table for FFHQ 64x64.
Table 9: Analogous Table for AFHQQv2 64x64.
Family
Model
NEE
FID
Family
Model
NFE
FID (J)
Trouthr
VI-FIM Karms al
3023
79
2.30
Dillasio
Putch Dillssoe Wragg
VP-EDM
Karm 6
5005
20152
2,50
3.11
Teacher
SiD, = 1.0
VP-EDM
Karros et 2022
79
1.628 1: 0017
1.96
B00T
Gueeral X05
9,00
SiD, 1.2
1.711
0.20
Distillation
SiDA, 1.0 (ours)
1.345
0.15
Distilbtrion
SIDM 6 (oun)
1.134L 0002
SiD' A, 1.0 (ours)
1.276 = 0.00
SID" A, (ours)
1,040 oz11
1.2 (ours)
SID" cr 1.2 (ours)
1,109 oz1s
SiDA 5% Data
10
SiDA Full Data
8
6
FID
4
3.53
2.91
1.4
102
10
104
105
Iterated k-images
Figure 7: Evolution of FID scores for the SiDA generator during the distillation of the EDM2-XS teacher model, pretrained on ImageNet 512x512. Training utilized either 5% or 100% of the real training images, with hatch size of 2048 and 1.0. Horizontal lines indicate the performance of EDM2 without classifier-free guidance (CFG) in purple, using 63 NFEs, and with CFG in red, using 63x2 NFEs.
Table 10: FDDINO scores and number of parameters for different methods on ImageNet 512x512.
Method
CFG
NEE
XS (125M)
S (280M)
M (498M)
L (777M)
XL (1.IB)
XXL (1.5B)
EDM2
N
63
103.39
68.64
58.44
52.25
45.96
42.84
EDM2
63x2
79.94
52.32
41.98
38.20
35.67
33.09
SiD
91.75 = ass
65.08 4.00
55,92 + 0.5
56,25 : 1:40
$2.47 0.9
56.15 = 041
93.67 4.40
68.76 1
53.40 1
48.47 0.5
44.47
1 0.39
A
86.96
62.19
49.01
46.80
43.89
44.52
22

---

Published as a conference paper at ICLR 2025
Figure 8: The SiD method highlights rapid advancements in distilling EDM2-XL pretrained on ImageNet 512x512, utilizing batch size of 2048 and learning rate of 5e-5. Illustrations are provided using random generations with class labels: O for tench and for goldfish. The series of images, generated from the same set of random noises post-training the SiDA generator with varying counts synthesized images, illustrates progressions at O, 0.2, 0.5, 1, 2. 4.10, 20, 100, and 200 million images. These are equivalent to 0, 100, 250, 500, Ik, 2K, 5K, 10k, 50k, and 100k training iterations respectively, organized from the top left to the bottom right. The blue doshed curve Panel (e) of detailss the progression of FIDs for SiD EDM2-XL.
Figure Analogous plot to Fir the the newly proposed SiDA method, with two more classes introduced for visulization: for great white shark, and 3 for tiger shark. The black dotted curve in Panel (e) Figure 3 details the progression of FIDs for SiDA on EDM2-XI..
Figure 10: Analogous plot to Figure 10 for the the newly proposed SID' A method. The series of images,
generated from the same set of random noises post-training the SiDA generator with varying counts of synthesized images, illustrates progressions at 0, 4. 10, 20, 40 million images. These are equivalent 0, 2K. 5K, 10k, 20k training iterations respectively, organized from the left to right. The orange solid curve Panel (e) of Figure details the progression of FIDs for SiD? A on EDM2-XL.
23

---

Published as a conference paper at ICLR 2025
Figure 11: Unconditional CIFAR-11 10 32X32 random images generated with SiD*A (FID: 1.499).
24

---

Published as a conference paper at ICLR 2025
Figure 12: Label conditioning CIFAR-10 32X32 random images generated with SiD*A (FID: 1.396)
25

---

Published as a conference paper at ICLR 2025
Figure 13: Label conditioning ImageNet 64x64 random images generated with SiD*A (FID: 1.110)
26

---

Published as a conference paper at ICLR 2025
Figure 14: FFHQ 64X64 random images generated with SiD*A (FID: 1.040)
27

---

Published as a conference paper at ICLR 2025
Figure 15: AFHQ-V V2 64X64 random images generated with SiD A (FID: 1.276)
Figure 16: ImageNet 512x512 images of Class 0 (tench) generated using SiD? A without classifier- free guidance, produced in single generation step.
28

---

Published as a conference paper at ICLR 2025
Figure 17: ImageNet 512x512 images of Class (goldfish) generated using SiD²A without classifier- free guidance, produced in a single generation step.
Figure 18: ImageNet 512x512 images of Class 15 (robin) generated using SiD without classifier- free guidance, produced a single generation step.
Figure 19: ImageNet 512x512 images of Class 29 (axolot) generated using SiD? A without classifier- free guidance, produced in single generation step.
29

---

Published as a conference paper at ICLR 2025
Figure 20: ImageNet 512x512 images of Class 33 (loggerhead) generated using SiD without classifier-free guidance, produced in a single generation step.
Figure 21: ImageNet 512x512 images of Class 88 (macaw) generated using SiD?A without classifier- free guidance, produced single generation step.
Figure 22: ImageNet 512x512 images of Class 89 (sulphur-crestee cockatoo) generated using SiD without fassi guidance, produced in single generation step.
30

---

Published as a conference paper at ICLR 2025
Figure 23: ImageNet 5 512x512 images of Class 127 (white stork) generated using SID without classifier-free guidance, produced in a single generation step.
Figure 24: ImageNet 512x512 images of Class 279 (arctic fox) generated using SiD? without classifier-free guidance, produced in a single generation step.
Figure 25: ImageNet 512x512 images of Class 292 (tiger) generated using SiD A without classifier-
free guidance, produced in single generation step.
31

---

Published as a conference paper at ICLR 2025
Figure 26: ImageNet 512x512 images of Class 323 (monarch) generated using SID without classifier-free guidance, produced in a single generation step.
Figure 27: ImageNet 512x512 images of Class 387 (lesser panda) generated using SiD without classifier-free guidance, produced in single generation step.
Figure 28: ImageNet 5125512 images of Class 388 (giant panda) generated using SiD without classifier-free guidance, produced in single generation step.
32

---

Published as a conference paper at ICLR 2025
2
Figure 29: ImageNet 512x512 images of Class 417 (balloon) generated using SiDºA without
classifier-free guidance, produced in a single generation step.
Figure 30: ImageNet 512x512 images of Class 425 (bar)) generated using SiD?A without classifier- free guidance, produced a single generation step.
Figure 31: ImageNet 512x512 images of Class 933 (cheeseburger) generated using SID A without classifier-free free guidance, produced in single generation step.
33

---

Published as a conference paper at ICLR 2025
Figure 32: ImageNet 512x512 images of Class 973 (coral reef) generated using SiD without classifier-free guidance, produced in a single generation step.
Figure 33: ImageNet 512x512 images of Class 975 (lakeside) generated using SiD without classifier-free guidance, produced in a single generation step.
Figure 34: ImageNet 512x512 images of Class 980 (volcano) generated using S without classifier-free guidance, produced in single generation step.
34


Reviews:
[{"review_id": "YyNOpbqyHp", "reviewer": "Reviewer_Ajww", "summary": "The authors propose SiDA (Score identity Distillation with Adversarial Loss), an enhanced version of Score identity Distillation (SiD) that aims to improve diffusion model training efficiency. The key innovation is the introduction of an adversarial loss that helps distinguish between real and generated latents at different noise levels. By reusing the fake score network in the original SiD, the authors propose a discriminator design that utilizes the encoder feature map to obtain the discriminator score. The proposed method achieves SOTA results on CIFAR-10, ImageNet-64, and other datasets.", "questions": "-\tWhy $L^1(\\theta)$ is still needed with the additional discriminator, i.e., why SiDA still needs tuning $\\alpha$?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "-\tThe authors repurpose the fake score network as an additional discriminator with direct using the average of the encoder features as the discriminator score. This design introduces the adversarial objective to the original SiD with minimal additional parameter cost.\n-\tThe proposed method SiDA significantly outperforms SiD and achieves superior 1-step generation performance on various datasets.", "weaknesses": "-\tThe discriminator design is heavily based on the U-Net encoder-decoder architecture. It is not straightforward to generalize to transformer-based diffusion models, e.g., DiT, where there is no clear partition of the encoder and decoder.\n-\tThe current submission lacks the evaluation of Diffusion models on higher resolution, e.g., ImageNet 256x256. The cost overhead from both discriminator training and maintaining an additional fake score network can be exaggerated on high-resolution data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FTzPjiNKT7", "reviewer": "Reviewer_kYRQ", "summary": "The paper introduces Score Identity Distillation with Adversarial loss. The method aims to distill a teacher diffusion model to reduce the number of inference steps to only one. The primary contribution is the combination of SiD with the use of real images and adversarial objective, which not only reduces the number of required inference steps but also improves generation quality, achieving a lower FID score across Cifar10, ImageNet-64, FFHQ-64 and AFHQ-64.", "questions": "- How does the method perform as the number of available real images changes? What happen if you use only 10% of the training real images?\n- Why not apply latent diffusion for larger images? Using VAE to compress the image would help to see if the method can scale to bigger images, without increasing the training time\n- Does the approach generalize to teacher models other than VP-EDM?\n- The paper arbitrarily sets $\\lambda_{sid}$ = 100 and $\\lambda_{adv}$ = 0.01 without discussion or ablation, although these hyperparameters likely play an important role in the method. How do these values influence the performance metrics?\n- In Figure 1, for $\\alpha$=1.2 (the yellow line), after approximately 1000 iterations, the Inception Score (IS) drops and the FID increases sharply; additionally, SiDA with $\\alpha$=1.2 is absent from Table 1. Does this imply that the model fails to converge?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Achieves state-of-the-art performance with only one inference step\n- The methods demonstrates improvement over the teacher network thanks to the real images\n- Training remains relatively stable despite the use of an adversarial loss", "weaknesses": "- Requires real images for training, which is not necessary for other distillation methods\n- Experiments are demonstrated only on very small images resolution\n- Ablation studies are limited to the parameter $\\alpha$\n- A schematic of the framework would greatly improve the paper's readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oseOTVSa97", "reviewer": "Reviewer_4cAk", "summary": "This paper introduces an improved version of score identity distillation and proposes training SiD with an adversarial objective. \nThe main motivation is that the lack of real data in the original training of SiD can potentially lead to discrepancies to the original data distribution. The authors propose to reuse the encoder part of the fake score model as the backbone of discriminator, and use its predicted realness and fakeness scores to improve the generation results further, at no cost of additional model parameters. By combining SiD with adversarial training, together with additional techniques such as initializing the student model with the SiD pretrained model, the proposed method achieved improved one-step diffusion results on multiple datasets.", "questions": "Please see the Weakness section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The discussions in the experiment section are very comprehensive. The detailed analysis of the training statistics across the training steps is appreciated. \n\n- The paper presented the background such as SiD and adversarial loss in diffusion well.", "weaknesses": "- Presentation\n\nIn the technical parts of the paper, which is mainly the Section 4. I frequently find the presentations not clear enough. My biggest confusion arises from the 'Loss Construction that Respects the Weighting of Diffusion Distillation Loss' paragraph. I understand that balancing loss terms can be tricky, and I acknowledge that this is an important topic to discuss. But from what the author presents, I didn't find how the proposed weight 'honors the existing weighting mechanisms' and, more importantly, automatically balances the loss regardless of the datasets. I read the Eq 7 and Eq 8 and found nothing particularly new. How is Eq 7 fundamentally different from PatchGAN? \n\n- Novelty\n\nAlthough the paper targets a problem of SiD that is not previously discussed, the solution of adding an adversarial term is by no means a novelty under the context of diffusion distillation. The adversarial object has been proven very effective in many works, such as UFOGen and SD3-turbo. The proposed method here is a simple mixture of two existing methods. \n\n- Evaluation and performance\n\nMost of the experiment settings used in this paper are small-scale. And the authors only compared the proposed method with very few other methods in the 512x512 ImageNet experiment. Since efficiency is one of the main advantages the authors are trying to advocate, it is necessary to include larger-scale evaluations, such as text-to-image experiments, to allow for fair comparisons with recent methods. \n\nI also have a slight concern about the advantage of efficiency. It is true that, according to Figure 3, the FID of the proposed method decreases faster compared to the baseline SiD in the later half of the training; I'd like to point out that this is achieved with an additional loss term, meaning each iteration probably optimizes the model with a larger gradient magnitude. Also, the adversarial loss term will inevitably introduce more computation, so the model is not necessarily able to finish training faster compared to SiD.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ampFHN8AbJ", "reviewer": "Reviewer_sPNq", "summary": "This paper proposes an intuitive method, which mainly focuses on the improvement of score identity distillation method by introducing additional real images for adversarial training. The improvement is two fold. First, additional real images can boost generation performance over data-free score identity distillation by discrimating real and synthetic images. Second, it achieves faster convergence rate relative to the baselines. The experimental results on small-scale dataset support the superority of the proposed method.", "questions": "+ Why can incorporating average \"fakeness\" within each GPU batch into each pixel's loss solve the gap between diffusion distillation and adversarial learning?\n+ Why does the proposed method can achieve faster convergence than baseline when trained from scratch?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This paper proposes an intuitive extension of score identity distillation and effectively improves the generation performance on small-scale datasets, such as CIFAR and ImageNet 64x64.\n+ The proposed method achieves better performance than the main baseline, SiD and other diffusion methods.", "weaknesses": "+ More state-of-the-art generative methods are missing, such as DiT, RDM and MAR.\n+ The method only conducts experiments on small-scale datasets or low-resolution images. I'm concern about the effectiveness of this method on large-scale dataset with high-resolution images. More results on large-scale datasets with high-resolution images consolidate the effectiveness of this method.\n+ OSGAN[R1] adopts an one-step adversarial training on generative task. The comparion with OSGAN is needed to distinguish their differences and highlight the contributions of this paper.\n+ This paper doesn't mention any plan about open source of code and trained weights. In spite of encouraging performance, I'm concern about the reproducibility of this paper. \n+ Some motivations are not well clarified in the paper. See Questions section.\n+ Some unnecessary equations, such as Eq between 149 and 151, can be removed.\n\n[R1] Training Generative Adversarial Networks in One Stage. CVPR 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)