Adversarial Score Identity Distillation: Rapidly Surpassing the Teacher in One Step

## Abstract

Abstract Score identity Distillation (SiD) is a data-free method that has achieved state-of-the-art performance in image generation by leveraging only a pretrained diffusion model, without requiring any training data. However, the ultimate performance of SiD is constrained by the accuracy with which the pretrained model captures the true data scores at different stages of the diffusion process. In this paper, we introduce SiDA (SiD with Adversarial Loss), which not only enhances generation quality but also improves distillation efficiency by incorporating real images and adversarial loss. SiDA utilizes the encoder from the generator’s score network as a discriminator, boosting its ability to distinguish between real images and those generated by SiD. The adversarial loss is batch-normalized within each GPU and then combined with the original SiD loss. This integration effectively incorporates the average “fakeness” per GPU batch into the pixel-based SiD loss, enabling SiDA to distill a single-step generator either from scratch or by fine-tuning an existing one. SiDA converges significantly faster than its predecessor when trained from scratch, and swiftly improves upon the original model’s performance after an initial warmup period during fine-tuning from a pre-distilled SiD generator.
This one-step adversarial distillation method establishes new benchmarks in generation performance when distilling EDM diffusion models pretrained on CIFAR-10 (32x32) and ImageNet (64x64), achieving FID scores of 1.499 on CIFAR-10 unconditional, 1.396 on CIFAR-10 conditional, and 1.110 on ImageNet 64x64. It sets record-low FID scores when distilling EDM2 models trained on ImageNet (512x512), surpassing even the largest teacher model, EDM2-XXL, which had an FID of 1.81 with classifier-free guidance. Our SiDA’s results—achieved without classifier-free guidance—record FID scores of 2.156 for EDM2-XS, 1.669 for EDM2-S, 1.488 for EDM2-M, and 1.465 for EDM2-L, demonstrating significant improvements across all model sizes.
Our open-source code will be integrated into the SiD codebase .

## 1 Introduction

Modeling the distribution of high-dimensional data, such as natural images, has been a persistent challenge in machine learning (Bishop, ; Murphy, ; Goodfellow et al., ). Before the emergence of deep generative models, research focused primarily on constructing hierarchical models constrained by parametric distributions (Blei et al., ; Griffiths & Ghahramani, ; Fei-Fei & Perona, ; Chong et al., ; Zhou et al., ) and developing neural networks with stochastic binary hidden layers (Hinton et al., ; Salakhutdinov & Hinton, ; Vincent et al., ), supported by robust inference techniques such as Gibbs sampling, maximum likelihood, variational inference (Hoffman et al., ; Blei & Jordan, ), and contrastive divergence (Hinton, ).

The last decade has witnessed significant advancements in deep generative models, including generative adversarial networks (GANs) (Goodfellow et al., ; Reed et al., ; Karras et al., ), normalizing flows (Papamakarios et al., ), and variational auto-encoders (VAEs) (Kingma & Welling, ; Rezende et al., ). Although VAEs are noted for their stability, they often produce blurred images, while GANs are acclaimed for their ability to generate photorealistic images despite challenges with training instability and generation diversity. These dynamics have driven research towards refining statistical distances to more accurately measure discrepancies between true and generated data distributions, especially in scenarios where traditional metrics fail due to non-overlapping supports in high-dimensional spaces (Arjovsky et al., ; Li et al., ; Zheng & Zhou, ).

Diffusion models (Sohl-Dickstein et al., ; Song & Ermon, ; Ho et al., ; Karras et al., ) have further revolutionized this field with their ability to produce photorealistic images, albeit with multiple iterative refinement resulting in slow sampling speeds. In response, significant efforts have been directed towards developing efficient methods to utilize pretrained diffusion models for reversing the forward diffusion process through iterative refinement.

Departing from conventional methods, Diffusion-GAN (Wang et al., ) reframes forward diffusion as a cost-effective, domain-agnostic data-augmentation strategy to tackle issues of non-overlapping distribution support. This approach introduces noise at various levels and utilizes statistical distances such as Jensen-Shannon divergence (JSD) for comparing distributions. The method promotes alignment between the true and generated data distributions at any stage during the forward diffusion process. It advocates for directly guiding the learning of a one-step generator through forward diffusion, moving away from the traditional dependence on iterative refinement-based sampling seen in reverse diffusion processes.

Expanding on this approach, recent methodologies have evolved from directly comparing empirical distributions of diffused groundtruth and generated data via JSD to distill their scores—a notable strength of diffusion models—at various stages of the forward diffusion process. This evolution has catalyzed the development of cutting-edge diffusion distillation techniques such as score distillation sampling (Poole et al., ), variational score distillation (Wang et al., ), Diff-Instruct (Luo et al., ), SwiftBrush (Nguyen & Tran, ), and distribution matching distillation (DMD) (Yin et al., ). These methods refine models by analyzing the diffused KL divergence between corrupted data and model distributions, effectively leveraging the gradients of this divergence. Although directly computing this KL divergence presents significant challenges, its gradients can be effectively estimated using the scores from both the pretrained model and the current generator.

In pursuit of transcending traditional challenges associated with JSD and model-based KL divergence—which are often difficult to optimize and prone to mode-seeking behaviors, respectively—Zhou et al. () have pioneered a model-based Fisher divergence within the diffusion distillation framework. This novel approach, notably data-free, has demonstrated for the first time the potential to match or even surpass the performance of teacher models in a single generation step. However, SiD and other data-free methods are based on the assumption that the score produced by teacher networks can well represent the data score. This assumption can potentially create a performance bottleneck for distilled single-step generators, especially if the teacher diffusion model is not well-trained or has limited capacity. In this paper, we aim to further enhance SiD by integrating real-world data. This integration seeks to compensate for the teacher model’s limitations in accurately representing the true score, thereby producing even more realistic generative outcomes. This enhancement broadens the practical applications of diffusion distillation in real-world scenarios.

Our approach builds on the existing fake score network and generator design of SiD, incorporating an adversarial loss that discriminates whether generated images are real or fake at various noisy stages of the forward diffusion process. The discriminator shares the weight with the encoder modules of the fake score network. To ensure compatibility with SiD’s pixel-level loss, we compute the discriminator loss at each spatial position of the latent encoder feature map and average across the batch samples within each GPU. This measure is integrated into the SiD loss for joint distillation and adversarial training and introduces no additional parameters.
Our findings indicate this approach significantly improves iteration efficiency by about an order of magnitude when training models from scratch, and delivers unprecedentedly low FID scores when initialized from the SiD distilled checkpoints.

## 2 Related Work

Diffusion models (Song & Ermon, ; Ho et al., ; Karras et al., ) are a family of generative models that adopts a denoising score matching objective (Vincent, ; Sohl-Dickstein et al., ) to learn the score function of complex high-dimensional distributions at different noise levels. A deep neural network is trained to match the score function of noise-corrupted data via minimizing a data-based Fisher divergence (Song & Ermon, ). This trained score network is then used to generate new samples by iteratively denoising random noise. Diffusion models have gained significant attention and success in generative modeling due to training stability and high-quality sample generation, with example applications in text-to-image generation (Nichol et al., ; Ramesh et al., ; Saharia et al., ; Rombach et al., ; Podell et al., ). However, the sampling process in diffusion models involves gradual denoising across many steps, making it significantly slower compared to one-step generation models like GANs and VAEs.

Numerous techniques have been proposed to accelerate diffusion models (Zheng et al., ; Lyu et al., ; Luhman & Luhman, ; Salimans & Ho, ; Zheng et al., ; Meng et al., ; Kim et al., ). Recently, one-step and few-step diffusion distillation methods have gained prominence, achieving the speed of GANs while retaining the robust capabilities of diffusion models. These methods can be broadly classified into two categories: data-free (Luo et al., ; Nguyen & Tran, ; Zhou et al., ) and those requiring real images or teacher-synthesized noise-image pairs (Liu et al., ; Song & Dhariwal, ; Luo et al., ; Kim et al., ; Sauer et al., ; Xu et al., ; Yin et al., ). The latter frequently incorporate GAN-based adversarial learning as a critical component to enhance the generation performance of baseline distillation methods, which vary from consistency models (Song et al., ) to score distillation sampling (Poole et al., ; Wang et al., ). Without this adversarial component, performance often degrades significantly (Kim et al., ; Sauer et al., ; Yin et al., ).

The collaborative integration of GANs and diffusion models, aimed at leveraging the strengths of both, is not a concept that originated with adversarial distillation. This approach is rooted in earlier studies: Xiao et al. () developed a method to learn the reverse process of the diffusion chain using a time-step-conditioned discriminator. Simultaneously, Zheng et al. () proposed that GANs learn a noisy marginal distribution at a specific targeted timestep, which would then serve as the prior for initiating the reverse diffusion process. Furthermore, they fine-tuned an early version of the Stable Diffusion framework to produce this noisy marginal using a discriminator equipped with an MLP head atop the encoder module of the U-Net in Stable Diffusion, specifically designed to accelerate text-to-image generation in just a few steps while maintaining high quality. A key insight from Zheng et al. () is that the diffusion U-Net backbone can simultaneously be used for reverse diffusion and function as both a GAN’s generator and discriminator. Building on this, we aim to generalize and simplify the designs of Zheng et al. () by repurposing the score networks in SiD to perform both distillation and adversarial learning, without introducing any additional parameters.

Our work also builds on previous efforts to enhance distribution matching by injecting noise. Measuring differences between distributions in high dimensions, where supports often do not overlap, is challenging and has spurred the development of advanced statistical distances (Murphy, ; Arjovsky et al., ; Li et al., ; Zheng & Zhou, ). Adding noise has been shown to be effective in overlapping the supports of the true and fake data distributions, which typically reside in separate high-dimensional density regions (Wang et al., ). This overlap mitigates the limitations of many existing statistical distances in measuring differences between such distributions accurately.

Diffusion-GAN introduced a novel strategy that matches the distribution between true and generated data at various stages of the forward diffusion process by introducing noise at different signal-to-noise ratios. This corrupts the observed data and aligns the distributions at multiple points throughout the diffusion pathway, using JSD for comparison to ensure a close alignment of the model distribution with the data distribution.

This approach to noise injection-based distribution matching has been further advanced by integrating pretrained diffusion models, which excel at estimating the true data score throughout the forward diffusion process. Building on this, while the KL divergence from diffused true to fake data distributions is intractable to estimate directly, its gradient can be computed using the scores of the true and fake data at various noise levels (Poole et al., ; Wang et al., ). This advancement has driven several recent innovations in one-step diffusion distillation methods for image generation, exemplified by studies such as Luo et al. (), Nguyen & Tran (), and Yin et al. ().

## 3 Preliminaries

Below, we briefly review both SiD and Diffusion-GAN, the foundations on which SiDA is built. A detailed description of SiDA follows in the next section.

Score identity Distillation (SiD): Moving beyond the challenges of computing JSD between diffused empirical data distributions, which can be unstable to optimize, and estimating the gradient of their KL divergence—known for mode-seeking behaviors when expectations are taken with respect to the model distribution—SiD employs a model-based Fisher divergence to match their scores. More specifically, let us denote the forward transition kernel used in Gaussian diffusion models by $q(\boldsymbol{x}_{t}\,|\,\boldsymbol{x}_{0})=\mathcal{N}(a_{t}\boldsymbol{x}_{
0},\sigma_{t}{\bf I})$, where $a_{t}\in(0,1)$, $\sigma_{t}>0$, and the signal-to-noise ratio $a_{t}^{2}/\sigma_{t}^{2}$ monotonically decreases towards zero as $t$ progresses.
SiD explicitly represents the forward diffused data and model distributions as semi-implicit distributions (Yin & Zhou, ; Yu et al., ):

$$ $\displaystyle p_{\mathrm{data}}(\boldsymbol{x}_{t})=\int q(\boldsymbol{x}_{t}| \boldsymbol{x}_{0})p_{\mathrm{data}}(\boldsymbol{x}_{0})d\boldsymbol{x}_{0}; \quad p_{\theta}(\boldsymbol{x}_{t})=\int q(\boldsymbol{x}_{t}|\boldsymbol{x}_ {g})p_{\theta}(\boldsymbol{x}_{g})d\boldsymbol{x}_{g}.$ $$

These distributions, while lacking analytic density functions, are simple to sample from. Specifically,
by defining $G_{\theta}$ as a generator converting noise into data, a diffused fake data $\boldsymbol{x}_{t}$ can be produced as:

$$ $\displaystyle\boldsymbol{x}_{t}=a_{t}\boldsymbol{x}_{g}+\sigma_{t}\boldsymbol{ \epsilon}_{t},~{}~{}\boldsymbol{\epsilon}_{t}\sim\mathcal{N}(0,\mathbf{I}),~{} ~{}\boldsymbol{x}_{g}=G_{\theta}(\boldsymbol{z},{\boldsymbol{c}}),~{}~{} \boldsymbol{z}\sim p(\boldsymbol{z}).$ (1) $$

To distill a student generative model, the distribution $p_{\theta}(\boldsymbol{x}_{t})$ needs to match the noisy data distribution at any time point $t$, ensuring that the distributions $p_{\text{data}}(\boldsymbol{x}_{0})$ and $p_{\theta}(\boldsymbol{x}_{g})$ are identical.

To achieve this goal, the distillation process involves adjusting the student model’s parameters so that its output at various stages of diffusion aligns closely with the diffused data from the target distribution. This alignment is typically facilitated through optimization techniques that minimize a specific statistical distance between these distributions across the entire diffusion trajectory (Wang et al., ; Luo et al., ).
Following the same principle,
SiD aims to minimize a model-based Fisher divergence given by:

$$ $\displaystyle\mathcal{L}_{\theta}(\phi^{*},\psi^{*}(\theta),t)$ $\displaystyle=\mathbb{E}_{\boldsymbol{x}_{t}\sim p_{\theta}(\boldsymbol{x}_{t} )}\left[\|\nabla_{\boldsymbol{x}_{t}}\ln p_{\text{data}}(\boldsymbol{x}_{t})- \nabla_{\boldsymbol{x}_{t}}\ln p_{\theta}(\boldsymbol{x}_{t})\|_{2}^{2}\right]$ $\displaystyle=\mathbb{E}_{\boldsymbol{x}_{t}\sim p_{\theta}(\boldsymbol{x}_{t} )}\left[\frac{a_{t}^{2}}{\sigma_{t}^{4}}\|f_{\phi^{*}}(\boldsymbol{x}_{t},t)-f _{\psi^{*}(\theta)}(\boldsymbol{x}_{t},t)\|_{2}^{2}\right],$ (2) $$

where $f_{\phi^{*}}$ and $f_{\psi^{*}(\theta)}$ represent the optimal denoising score networks for the true and fake data distributions, respectively. These can be expressed as:

$$ $\displaystyle f_{\phi^{*}}(\boldsymbol{x}_{t},t)=(\boldsymbol{x}_{t}+\sigma_{t }^{2}\nabla_{\boldsymbol{x}_{t}}\ln p_{\text{data}}(\boldsymbol{x}_{t}))/a_{t} =\mathbb{E}[\boldsymbol{x}_{0}\,|\,\boldsymbol{x}_{t}],$ $\displaystyle f_{\psi^{*}(\theta)}(\boldsymbol{x}_{t},t)=(\boldsymbol{x}_{t}+ \sigma_{t}^{2}\nabla_{\boldsymbol{x}_{t}}\ln p_{\theta}(\boldsymbol{x}_{t}))/a _{t}=\mathbb{E}[\boldsymbol{x}_{g}\,|\,\boldsymbol{x}_{t}].$ (3) $$

GANs and Diffusion-GANs: GANs (Goodfellow et al., ) seek to match the distributions with a min-max game between a discriminator $D$ and a generator $G$, with the training loss formulated as:

$$ $\displaystyle\min_{G}\max_{D}\mathbb{E}_{\boldsymbol{x}\sim p_{\text{data}}( \boldsymbol{x})}[\log D(\boldsymbol{x})]+\mathbb{E}_{\boldsymbol{z}\sim p( \boldsymbol{z})}[\log(1-D(G(\boldsymbol{z})))],~{}~{}\boldsymbol{z}\sim \mathcal{N}(0,I).$ $$

To overcome the issues of non-overlapping probability support between two distributions, Diffusion-GANs (Wang et al., ) propose to match the noisy distributions $p_{\mathrm{data}}(\boldsymbol{x}_{t})$ and $p_{\theta}(\boldsymbol{x}_{t})$ with a discriminator conditioned on time-step $t$:

$$ $\displaystyle\min_{G}\max_{D}\mathbb{E}_{\boldsymbol{x}_{0}\sim p(\boldsymbol{ x}_{0}),\boldsymbol{x}_{t}\sim q(\boldsymbol{x}_{t}\,|\,\boldsymbol{x}_{0};t)} [\log(D(\boldsymbol{x}_{t};t))]+\mathbb{E}_{\boldsymbol{z}\sim p(\boldsymbol{z }),\boldsymbol{x}_{t}^{g}\sim q(\boldsymbol{x}_{t}\,|\,G(\boldsymbol{z});t)}[ \log(1-D(\boldsymbol{x}_{t}^{g};t))],$ $$

where $t$ is sampled from an adaptive proposal distribution based on the competition between the generator and discriminator.

From the observations above, we recognize that the core objective of both SiD and Diffusion-GAN is to align the noisy distributions at any time step $t$ to ensure that the generative distribution $p_{\theta}(\boldsymbol{x}_{g})$ closely matches the clean data distribution $p_{\text{data}}(\boldsymbol{x}_{0})$. SiD operates under the assumption that it has access to the true data score $f_{\phi^{*}}$ (or a pretrained estimation thereof) but not to the actual data, while Diffusion-GAN operates under the reverse assumption—access to actual data but not to $f_{\phi^{*}}$. In the following section, we will detail how these two objective functions are integrated for joint distillation and adversarial generation, leveraging their respective strengths to enhance model performance.

## 4 SIDA: Adversarial Score identity Distillation

Motivations for Integrating Adversarial Loss into SiD.
The model-based Fisher divergence, as shown in (2), is often intractable because neither $\phi^{*}$ nor $\psi^{*}(\theta)$ are directly known.
A standard initial approximation, common across nearly all diffusion distillation methods, is to replace $f_{\phi^{*}}$ with the denoising score network $f_{\phi}$, which is pretrained on data $\boldsymbol{x}_{t}\sim p_{\text{data}}(\boldsymbol{x}_{t})$ and referred to as the teacher. This approach hinges on the assumption that the teacher’s score accurately represents the true data score, which could potentially create a performance bottleneck for distilled generators. Essentially, as the pretrained teacher $f_{\phi}$ does not perfectly recover the true data score $f_{\phi^{*}}$, the objective function:

$$ $\displaystyle\mathcal{L}_{\theta}(\boldsymbol{x}_{t},\phi,\psi^{*}(\theta))$ $\displaystyle=\mathbb{E}_{\boldsymbol{x}_{t}\sim p_{\theta}(\boldsymbol{x}_{t} )}\left[\frac{a_{t}^{2}}{\sigma_{t}^{4}}\|f_{\phi}(\boldsymbol{x}_{t},t)-f_{ \psi^{*}(\theta)}(\boldsymbol{x}_{t},t)\|_{2}^{2}\right],$ (4) $$

represents only an approximation of (2), meaning $\mathcal{L}_{\theta}(\boldsymbol{x}_{t},\phi,\psi^{*}(\theta))\neq\mathcal{L}_
{\theta}(\boldsymbol{x}_{t},\phi^{*},\psi^{*}(\theta))$. Optimizing this function may not ensure that the distribution $p_{\theta}(\boldsymbol{x}_{g})$ matches $p_{\text{data}}(\boldsymbol{x}_{0})$.

This discrepancy motivates the integration of adversarial loss to potentially correct deviations caused by inaccuracies in $f_{\phi}$, aiming for a closer alignment between the generated and true data distributions. Specifically, while optimizing (4) does not guarantee that $p_{\theta}(\boldsymbol{x}_{g})$ will closely align with $p_{\text{data}}(\boldsymbol{x}_{0})$, particularly if $f_{\phi}$ is not well-trained or has limited capacity, incorporating an adversarial loss can enhance the match between the distributions of diffused true data and generator-synthesized fake data, thus improving the correspondence between $p_{\theta}(\boldsymbol{x}_{g})$ and $p_{\text{data}}(\boldsymbol{x}_{0})$. Although introducing a separate discriminator to implement the adversarial loss is possible, we find it unnecessary as we can effectively reuse the fake score network, thereby avoiding the introduction of any new parameters. This approach simplifies the model architecture and enhances efficiency, as further explained below.

Joint Score Estimation and Discrimination. While $\psi^{*}(\theta)$ is intractable and would introduce a complex bi-level optimization problem due to its dependence on $\theta$—the very parameter we aim to optimize—SiD acknowledges the impracticality of obtaining either $\psi^{*}(\theta)$ or its gradient with respect to $\theta$. Therefore, the same as many previous works in diffusion distillation, SiD estimates $\psi^{*}(\theta)$ using a denoising network $\psi$, which is learned on the fake data, as

$$ $\displaystyle\min_{\psi}\hat{\mathcal{L}}_{\psi}(\boldsymbol{x}_{t},t)= \textstyle\gamma(t)\|f_{\psi}(\boldsymbol{x}_{t},t)-\boldsymbol{x}_{g}\|_{2}^{ 2},$ (5) $$

where $\boldsymbol{x}_{t}$ and $\boldsymbol{x}_{g}$ are drawn via reparameterization, as in (1), and $\gamma(t)$ is a reweighting term that is typically set the same as the signal-to-noise ratio at time $t$ (Ho et al., ; Kingma et al., ; Hang et al., ). This approximation introduces bias, which we will address and demonstrate how to mitigate effectively in later discussion.

To avoid introducing any additional parameters or complex training pipelines, we incorporate a return-flag as an additional input to the network $f_{\psi}$, offering options ‘decoder’, ‘encoder’, and ‘encoder-decoder’. When set to ‘decoder’, the network returns the denoised image as before. If set to ‘encoder’, it performs average pooling on the output of the last U-Net encoder block of $f_{\psi}$ along the channel dimension and returns a 2D discriminator map. For ‘encoder-decoder’, the network outputs both the discriminator map and continues through the U-Net decoder blocks to return the denoised image. The choice of option depends on whether we are updating the fake score network or the generator, and whether the input is a real image or a generator-synthesized image. With the capability to extract 2D discriminator maps either under the ‘encoder’ option or jointly with the denoised images under the ‘encoder-decoder’ option, we are now positioned to define the adversarial loss function.

The gradient of this loss function will either directly impact the encoder part of $f_{\psi}$ or the generator, contingent upon the optimization target—whether it is the joint loss involving fake-score estimation and discrimination or the combined loss of diffusion distillation and adversarial generation. It is also important to highlight that no additional parameters have been introduced to facilitate joint score estimation and discrimination. This streamlined approach aids in maintaining model efficiency without compromising the effectiveness of the methodology.

Joint Diffusion Distillation and Adversarial Generation. While recognizing the impracticality of directly obtaining either $\psi^{*}(\theta)$ or its gradient with respect to $\theta$, substituting $\psi^{*}(\theta)$ in (4) with a denoising network $\psi$ could introduce a severely biased gradient that undermines the optimization of $\theta$. SiD addresses this challenge in gradient estimation by subtracting a biased gradient estimate, which has proven ineffective in practice, from a less biased one that has shown standalone effectiveness. This strategy results in a loss function that significantly enhances performance, expressed as:

$$ $\displaystyle\tilde{\mathcal{L}}_{\theta,t,\alpha}=\omega(t)\frac{a_{t}^{2}}{ \sigma_{t}^{4}}\mathcal{L}_{\theta,t,\alpha}^{(\text{sid})},$ (6) $$

where $\omega(t)$ are weight coefficients, $\alpha$ is a bias correction weight that is typically set at 1 or 1.2, and

$$ $\displaystyle\mathcal{L}_{\theta,t,\alpha}^{(\text{sid})}=-\alpha\|f_{\phi}( \boldsymbol{x}_{t},t)-f_{\psi}(\boldsymbol{x}_{t},t)\|_{2}^{2}+(f_{\phi}( \boldsymbol{x}_{t},t)-f_{\psi}(\boldsymbol{x}_{t},t))^{T}(f_{\phi}(\boldsymbol {x}_{t},t)-\boldsymbol{x}_{g}).$ $$

An alternative theoretical justification for this loss has recently been explored in Luo et al. (), demonstrating that the SiD loss can be derived under certain assumptions about the data generation mechanism, specifically involving stop-gradient conditions.
SiD employs an alternating optimization strategy, updating $\psi$ according to (5) and $\theta$ according to (6), using only generator-synthesized fake data, drawn as specified in (1), for parameter updates. This approach ensures that both components are iteratively refined with outputs generated by the model itself, without relying on real data inputs.

Since $f_{\phi}$ has been adapted to serve dual purposes—not only performing denoising but also capable of outputting a 2D discriminator map for the encoder’s latent space—we are now ready to define an adversarial generation loss. This loss compels the generator to trick the discriminator map into believing its output originates from a real image at each spatial location of this latent 2D discriminator map. Consequently, the generator will be updated using both the loss specified in (6) and a GAN’s generator loss, effectively integrating these components to bridge the gap between the pretrained score $f_{\phi}$ and the true score $f_{\phi}^{*}$, and hence enhance the realism of the generated outputs.

Loss Construction that Respects the Weighting of Diffusion Distillation Loss. Building on the discussions above, we are now prepared to synthesize all components into a comprehensive loss function. We face an additional challenge: the diffusion loss used by SiD operates at the pixel level, while the GAN loss functions at the level of the latent 2D space. This discrepancy can make it challenging to balance the losses between diffusion distillation and adversarial generation. To circumvent the need for dataset-specific adjustments and to develop a universally applicable solution, we have devised a strategy that honors the existing weighting mechanisms used in training and distilling diffusion models. This approach involves performing GPU batch pooling to calculate an average “fakeness” within each GPU batch, then incorporating this average fakeness into each pixel’s loss prior to applying the pixel-specific weights. We have found this per-GPU-batch fakeness strategy to be highly effective, demonstrating robust performance across all datasets considered in this study.

More specifically, denoting $(W,H,C_{c})$ as the size of the image, $\mathcal{B}$ a batch inside a GPU, $|\mathcal{B}|$ the per GPU batch size, $D$ the encoder part of $f_{\psi}$, $D(\boldsymbol{x}_{t})$ the 2D discriminator map of size $(W^{\prime},H^{\prime})$ given input $\boldsymbol{x}_{t}$, and $D(\boldsymbol{x}_{t})[i,j]$ the $(i,j)th$ element of $D(\boldsymbol{x}_{t})$, we
define the adversarial loss that measures the average in GPU-batch fakeness as

$$ $\displaystyle\mathcal{L}_{\theta}^{(\text{adv})}=\frac{1}{|\mathcal{B}|}\sum_{ \boldsymbol{x}_{t}\in\mathcal{B}}D(\boldsymbol{x}_{t})=\frac{1}{|\mathcal{B}|W ^{\prime}H^{\prime}}\sum_{\boldsymbol{x}_{t}\in\mathcal{B}}\sum_{i=1}^{W^{ \prime}}\sum_{j=1}^{H^{\prime}}D(\boldsymbol{x}_{t})[i,j].$ (7) $$

We now define SiDA’s generator loss, aggregated over a GPU batch, as follows:

$$ $\displaystyle\mathcal{L}_{\theta}^{(\text{sida})}(\mathcal{B})=\sum_{ \boldsymbol{x}_{t}\in\mathcal{B}}\frac{\omega(t)a_{t}^{2}}{2\sigma_{t}^{4}} \left(\lambda_{\text{sid}}\mathcal{L}_{\theta,t}^{(\text{sid})}+\lambda_{\text {adv}}{C_{c}WH}\mathcal{L}_{\theta}^{(\text{adv})}\right),$ (8) $$

where we set $\lambda_{\text{sid}}=100$ and $\lambda_{\text{adv}}=0.01$ by default, unless specified otherwise.

For the fake score network $f_{\psi}$, whose encoder also serves as the discriminator $D$, we define the SiDA’s loss aggregated over a GPU batch as

$$ $\displaystyle\sum_{\boldsymbol{x}_{t}\in\mathcal{B}}{\gamma(t)}(\|f_{\psi}( \boldsymbol{x}_{t},t)-\boldsymbol{x}_{g}\|_{2}^{2}+{L}_{\psi}^{(\text{adv})}),$ (9) $$

where the discriminator loss given a batch of fake data $\boldsymbol{x}_{t}$ and real data $\boldsymbol{y}_{t}$ is expressed as

$$ $\displaystyle{L}_{\psi}^{(\text{adv})}=\frac{1}{2|\mathcal{B}|W^{\prime}H^{ \prime}}\sum_{\boldsymbol{y}_{t},\boldsymbol{x}_{t}\in\mathcal{B}}\sum_{i^{ \prime}=1}^{W^{\prime}}\sum_{j^{\prime}=1}^{H^{\prime}}\ln(D(\boldsymbol{y}_{t })[i^{\prime},j^{\prime}])+\ln(1-D(\boldsymbol{x}_{t})[i^{\prime},j^{\prime}]).$ (10) $$

We summarize in Algorithm 1 the implementation of SiDA, which starts training from a pretrained diffusion model, and in Algorithm 2 the SiDA algorithm that begins training from both a pretrained diffusion model and a pre-distilled SiD generator, referred to as SiD^2A (SiD-initialized SiDA).

## 5 Experimental Results

We evaluate SiDA through comprehensive experiments designed to assess its effectiveness compared to the teacher diffusion model and other existing baseline methods for both unconditional and label-conditional image generation. We present results that measure both quantitative and qualitative performance across various datasets, aiming to demonstrate SiDA’s capability to generate high-quality samples through efficient score distillation. Detailed descriptions of the experimental setup, datasets, and evaluation metrics are provided below.

Datasets and Teacher Models. To comprehensively evaluate the effectiveness of SiDA,
we first utilize four representative datasets of varying scales and resolutions and the EDM diffusion models pretrained on them, as discussed in Karras et al. (). These datasets include CIFAR-10 ($32\times 32$; both conditional and unconditional) (Krizhevsky et al., ), ImageNet ($64\times 64$) (Deng et al., ), FFHQ ($64\times 64$) (Karras et al., ), and AFHQ-v2 ($64\times 64$) (Choi et al., ).

Additionally, we distill EDM2 diffusion models of varying sizes that were pretrained on ImageNet ($512\times 512$), following details provided by Karras et al. (). We compare these distilled models to their corresponding teacher models and to a concurrent work of Lu & Song () on EDM2 distillation. The models range from approximately 100 million to 1 billion parameters. By utilizing these diverse datasets and model scales, we thoroughly assess SiDA’s performance across different content types and complexities, ensuring a robust evaluation of its generative capabilities.

Evaluation protocol. We assess the quality of image generation using the Fréchet Inception Distance (FID) and Inception Score (IS) (Salimans et al., ).
Following Karras et al. (), we also evaluate models trained on ImageNet ($512\times 512$) using $\text{FD}_{\text{DINOv2}}$, the Fréchet distance computed in the feature space of DINOv2 (Oquab et al., ). Consistent with the methodology of Karras et al. (; ; ), these scores are calculated using 50,000 generated samples, with the training dataset used by the EDM or EDM2 teacher model(^1^11[https://github.com/NVlabs/edm](https://github.com/NVlabs/edm)) serving as the reference. Additionally, we evaluate SiDA on ImageNet $64\times 64$ using the Precision and Recall metrics (Kynkäänniemi et al., ), where both metrics are calculated using a predefined reference batch(^2^22[https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/64/VIRTUAL_imagenet64_labeled.npz](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/64/VIRTUAL_imagenet64_labeled.npz))(^3^33[https://github.com/openai/guided-diffusion/tree/main/evaluations](https://github.com/openai/guided-diffusion/tree/main/evaluations)) (Dhariwal & Nichol, ; Nichol & Dhariwal, ; Song et al., ; Song & Dhariwal, ). In line with SiD, we periodically evaluate the FID during distillation and preserve the generators with the lowest FID. To ensure accuracy, we conduct re-evaluations across 10 independent runs to obtain reliable metrics. We recommend this more statistically rigorous approach over reporting only the best metrics observed across independent runs or within the distillation process.

Implementation details.
We implement SiDA using the SiD codebase (Zhou et al., ), incorporating necessary functions adapted from either EDM (Karras et al., ) or EDM2 (Karras et al., ). The score estimation network, $f_{\psi}$, is initialized using the same architecture and parameters as the pre-trained teacher score network, $f_{\phi}$, from either EDM or EDM2. For the initialization of the student generator, $G_{\theta}$, we explore two settings: one where $G_{\theta}$ is initialized with the same parameters as the pre-trained EDM or EDM2 teacher score network, and another where it is initialized using the distilled generator from SiD, referred to as SiD-SiDA (SiD^2A).

Our training process unfolds in three stages:
1) For the first 100k images, we exclusively train $f_{\psi}$ to stabilize the score estimation.
2) For the subsequent 100k images, both $f_{\psi}$ and $G_{\theta}$ are trained using the SiD generator loss (6), integrating the distilled insights.
3) For all remaining images, we update $f_{\psi}$ and $G_{\theta}$ with the SiDA generator loss (8), which incorporates adversarial adjustments to refine the generation process.
Throughout the training, the score estimation network $f_{\psi}$ is consistently trained with the SiDA fake score loss (9), ensuring that the adversarial components are properly integrated. Additional implementation details are provided in Appendix B.

We observe that using only the adversarial loss, without the SiD component in the combined loss, leads to rapid divergence, even when the single-step generator is initialized from SiD.

Figure: Figure 1: Ablation Study of $\alpha$ on distilling an EDM model pretrained on CIFAR-10 (unconditional): Each plot illustrates the relation between the performance, measured by FID (left) and Inception Score (right) vs. the number of training iterations during the distillation, across varying values of $\alpha$. The batch sizer is 256. The study underscores the impact of $\alpha$ on both training efficiency and generative fidelity, leading us to select $\alpha=1.0$ for the subsequent experiments.
Refer to caption: x1.png

### 5.1 Ablation on the choice of α 𝛼 \alpha italic_α .

We conducted an ablation study to evaluate the impact of the gradient bias correction weight factor $\alpha$, a pivotal hyperparameter for SiD, on the performance of SiDA. Specifically, we tested a spectrum of $\alpha$ values $[-0.25,0.0,0.5,0.75,1.0,1.2,1.5]$ during the distillation of the EDM model pretrained on CIFAR-10 in an unconditional context. Figure 2 demonstrates how the model’s performance varies with different $\alpha$ values over the first 20k iterations at a batch size of 256, assessed using FID and IS. We determined that an $\alpha$ value of 1.0 yielded the best performance, prompting us to adopt this setting for subsequent experiments of SiDA over various datasets. In contrast, lower $\alpha$ values such as $-0.25$ or 0 resulted in suboptimal performance, while high values like 1.2 or 1.5 led to training instability and divergence. Interestingly, the performance disparity for $\alpha$ values between 0.5 and 1.0 was significantly narrower than those observed in SiD, as depicted in Figure 8 of Zhou et al. (). These findings imply that although the adversarial loss may introduce some instability, it could also compensate for the inadequate correction of gradient biases associated with smaller $\alpha$ values.

In an alternative setting, referred to as SiD^2A (SiD-initialized SiDA), where the generator $G_{\theta}$ is initialized using the best available pre-distilled SiD generator, we observe robust and stabilized training across all choices of $\alpha$. This suggests that initializing with the pre-distilled SiD generator provides a more advantageous starting point to achieve enhanced training stability and generative quality.
Following these observations, and given that $\alpha=1.0$ and $\alpha=1.2$ were also utilized in SiD experiments, we have selected these same values for subsequent experiments with SiD^2A.

Table: Table 1: Comparison of unconditional generation on CIFAR-10. The best one/few-step generator under the FID or IS metric is highlighted with bold.

Figure: Figure 3: Evolution of FIDs for the SiD and SiDA generator during the distillation of the EDM teacher model pretrained on CIFAR-10 unconditional (left) and conditional (middle), with a batch size of 256, and on ImageNet 64x64 (right) with a batch size of 8192, using $\alpha=1.0$ or $\alpha=1.2$. The performance of EDM (35 NFEs), along with DMD and Diff-Instruct, is depicted with horizontal lines in purple, green, and red, respectively.
Refer to caption: x5.png

Table: Table 3: Analogous to Table 2 for ImageNet 64x64 with label conditioning. The Precision and Recall metrics are also included.

### 5.2 Benchmarking EDM Distillation

We begin by assessing the performance of SiDA and SiD^2A, focusing on both generation quality and convergence speed across the datasets previously mentioned. Our comprehensive evaluation compares SiDA and SiD^2A against top-performing deep generative models, including GANs, traditional diffusion models, and their distilled counterparts, in terms of generation quality. Qualitatively, random images generated by SiD^2A in a single step are displayed from Figures 6 to 10 in the Appendix, illustrating the high quality achieved.

Quantitatively, on CIFAR-10 in both conditional and unconditional settings, SiDA and SiD^2A outperform leading models, with details in Tables 2 and 2.
Notably, SiDA achieves an FID of 1.516, outperforming all baseline models.
When leveraging the one-step generators already distilled by SiD for initialization, SiD^2A further advances this performance, achieving an exceptionally low FID of 1.499 on CIFAR-10 unconditional with $\alpha=1.0$, and 1.396 on CIFAR-10 conditional with $\alpha=1.2$.

We further validate the effectiveness of SiDA and SiD^2A on the ImageNet 64x64 dataset. Here, SiDA demonstrated superior performance over SiD, achieving an FID of 1.353. Notably, by leveraging the advanced capabilities of the SiD one-step generator, SiD^2A attained an unprecedented low FID of 1.110 on ImageNet 64x64 with $\alpha=1.2$, surpassing the teacher model’s performance by a large margin. On the FFHQ 64x64 and AFHQ-v2 64x64 datasets, known for human and animal faces, SiDA and SiD^2A again prove superior. The datasets’ less diverse patterns do not diminish the models’ performance, with SiDA and SiD^2A maintaining faster convergence and surpassing the teacher model rapidly.
The clear improvements across diverse datasets underscore SiDA and SiD^2A’s effectiveness.

It’s worth noting that SiDA’s close competitor for ImageNet ($64\times 64$), GDD-I (Zheng & Yang, ), uses a discriminator inspired by Projected GAN (Sauer et al., ), integrating features from VGG16-BN and EfficientNet-lite0. Concerns have been raised, as discussed in Kynkäänniemi et al. () and by Song & Dhariwal (), about the potential for encoders pretrained on ImageNet to inadvertently enhance FID scores through feature leakage. This potential issue does not affect SiD, SiDA, or SiD^2A, ensuring our results reflect true model performance without such confounds.

SiDA also demonstrates accelerated convergence, as evidenced in Figures 3 and 5. The logarithmic scale plots show SiDA’s FID decreasing more rapidly than SiD’s, an efficiency that also holds when scaled up to ImageNet 64x64.
Overall, SiDA consistently outperforms SiD in terms of convergence speed by a substantial margin, marking a significant acceleration, particularly noteworthy since SiD already converges at an exponential rate. This rapid improvement in both speed and performance underscores SiDA’s effectiveness and efficiency in generating high-quality images across different datasets.
These results not only highlight SiD^2A’s enhanced image quality and reduced iteration needs but also the overall efficiency of SiDA and SiD^2A across various datasets, confirming the advantages of these advanced distillation techniques.

### 5.3 Benchmarking EDM2 Distillation

We validate the effectiveness of SiD, SiDA, and SiD^2A in distilling EDM2 models of six different sizes pretrained on ImageNet 512x512. Our findings are summarized in Tables 4 and 7 and illustrated in Figure 4. Notably, SiD, our baseline single-step generator introduced six months ago on arXiv, performs comparably to the EDM2 teacher, which requires 63 NFEs, as well as to the simple, stable, and scalable consistency distillation (sCD) method of Lu & Song () that uses 2 NFEs—a concurrent work to SiDA. SiD also convincingly outperforms sCD with 1 NFE, despite not yet employing CFG during distillation. Consistent with findings from EDM distillation, both SiDA and SiD^2A demonstrate superior performance over SiD in distilling EDM2, achieving FIDs of 1.756 and 1.669, respectively, using the EDM2-S model with only 280M parameters. These results already surpass the EDM2-XXL model and sCD at 1.5B parameters, whose FIDs exceed 1.8.

Notably, scaling SiDA to EDM2-XL achieves a record-low FID of 1.449. While results for SiD^2A on EDM2-XL and for the SiD family on EDM2-XXL are forthcoming, Figure 4 indicates the potential for further FID reductions as these results become available.

Mirroring findings from EDM distillation, SiDA also exhibits accelerated convergence when distilling EDM2, as shown in Figure 4. Logarithmic-scale plots reveal a more rapid decrease in FID for SiDA compared to SiD, an efficiency that scales effectively to ImageNet 512x512 and across generators of various sizes. Overall, SiDA consistently surpasses prior and concurrent methods in terms of both final metrics and convergence speed. This rapid improvement highlights SiDA’s effectiveness and efficiency in generating high-resolution images across various model sizes.

**Table 4: FID scores and number of parameters for different methods on ImageNet (512x512).**
| Method | CFG | NFE | XS (125M) | S (280M) | M (498M) | L (777M) | XL (1.1B) | XXL (1.5B) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| EDM2 | N | 63 | 3.53 | 2.56 | 2.25 | 2.06 | 1.96 | 1.91 |
| EDM2 | Y | 63 | 2.91 | 2.23 | 2.01 | 1.88 | 1.85 | 1.81 |
| sCT | Y | 1 | - | 10.13 | 5.84 | 5.15 | 4.33 | 4.29 |
| sCT | Y | 2 | - | 9.86 | 5.53 | 4.65 | 3.73 | 3.76 |
| sCD | Y | 1 | - | 3.07 | 2.75 | 2.55 | 2.40 | 2.28 |
| sCD | Y | 2 | - | 2.50 | 2.26 | 2.04 | 1.93 | 1.88 |
| SiD | N | 1 | 3.353 $\pm$ 0.041 | 2.707 $\pm$ 0.054 | 2.060 $\pm$ 0.038 | 1.907 $\pm$ 0.016 | 2.005 $\pm$ 0.024 | - |
| SiDA | N | 1 | 2.228 $\pm$ 0.037 | 1.756 $\pm$ 0.023 | 1.546 $\pm$ 0.023 | 1.501 $\pm$ 0.024 | 1.449 $\pm$ 0.033 | - |
| SiD^2A | N | 1 | 2.156 $\pm$ 0.028 | 1.669 $\pm$ 0.019 | 1.488 $\pm$ 0.038 | 1.465 $\pm$ 0.028 | - | - |

Figure: (a) EDM-XS
Refer to caption: x8.png

## 6 Conclusion

In this paper, we present Score identity Distillation with Adversarial Loss (SiDA), an innovative framework that optimizes diffusion model distillation by integrating the strengths of Score identity Distillation (SiD) and Diffusion-GAN. SiDA accelerates the generation process, significantly reducing the number of iterations required compared to conventional score distillation methods, while producing high-quality images. Extensive tests across various datasets—CIFAR-10, ImageNet (64x64 and 512x512), FFHQ, AFHQ-v2—and different scales of pretrained EDM2 models demonstrate that SiDA consistently surpasses both the base teacher diffusion models and other distilled variants. This is validated by superior Fréchet Inception Distance (FID) scores and other performance metrics. Notably, SiDA achieves these results without utilizing classifier-free guidance, suggesting potential for further enhancement with its integration. Specifically, incorporating the long-and-short guidance strategy as described by Zhou et al. () could further refine SiDA’s performance. Future work will explore expanding SiDA’s methodology to additional generative tasks and refining its capabilities through innovative distillation techniques.

## Appendix A Algorithm Box

Figure: Algorithm 1 Adversarial Score identity Distillation (SiDA)

Figure: Algorithm 2 SiD^2A: SiD initilized SiDA

## Appendix B Additional Training and Evaluation Details

Figure: Figure 5: Analogous plot to Figure 3 for FFHQ-64x64 with batch size of 512 (left) and AFHQ-v2-64x64 with batch size of 512 (right).
Refer to caption: x14.png

Table: Table 5: Analogous to Table 2 for FFHQ 64x64.

**Table 7: FD${}_{\text{DINOv2}}$ scores and number of parameters for different methods on ImageNet (512x512).**
| Method | CFG | NFE | XS (125M) | S (280M) | M (498M) | L (777M) | XL (1.1B) | XXL (1.5B) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| EDM2 | N | 63 | 103.39 | 68.64 | 58.44 | 52.25 | 45.96 | 42.84 |
| EDM2 | Y | 63 | 79.94 | 52.32 | 41.98 | 38.20 | 35.67 | 33.09 |
| SiD | N | 1 | 91.75 $\pm$ 0.55 | 65.08 $\pm$ 0.32 | 55.92 $\pm$ 0.25 | 56.25 $\pm$ 0.40 | 55.87<br>$\pm$ 0.21 | - |
| SiDA | N | 1 | 93.67 $\pm$ 0.40 | 68.76 $\pm$ 0.45 | 53.40 $\pm$ 0.60 | 48.47 $\pm$ 0.45 | 46.47 $\pm$ 0.27 | - |
| SiD^2A | N | 1 | 86.96 $\pm$ 0.32 | 62.19 $\pm$ 0.36 | 49.01 $\pm$ 0.54 | 45.92 $\pm$ 0.33 | - | - |

The hyperparameters customized for our study are outlined in Table 8, with all remaining settings consistent with those in the EDM code (Karras et al., ) and SiD code (Zhou et al., ).

**Table 8: Hyperparameter settings and comparison of distillation time, and memory usage between SiD and SiDA, using 8 NVIDIA V100 GPUs (16 GB each) or 8 NVIDIA H100 GPUs (80 GB each).**
| Method | Hyperparameters | CIFAR-10 32x32 | ImageNet 64x64 | FFHQ 64x64 | AFHQ-v2 64x64 |
| --- | --- | --- | --- | --- | --- |
|  | Batch size | 256 | 8192 | 512 | 512 |
|  | Batch size per GPU | 32 | 32 | 64 | 64 |
|  | GPUs | 8xV100 | 8xH100 | 8xH100 | 8xH100 |
|  | Gradient accumulation round | 1 | 32 | 1 | 1 |
|  | Learning rate of ($\psi$, $\theta$) | 1e-5 | 4e-6 | 1e-5 | 5e-6 |
|  | Loss scaling of ($\lambda_{\text{sid}},\,\lambda_{\text{adv}}$) | (100, 0.01) |  |  |  |
|  | ema | 0.5 | 2 | 0.5 | 0.5 |
|  | fp16 | False | True | True | True |
|  | Optimizer Adam (eps) | 1e-8 | 1e-6 | 1e-6 | 1e-6 |
|  | Optimizer Adam ($\beta_{1}$) | 0 |  |  |  |
|  | Optimizer Adam ($\beta_{2}$) | 0.999 |  |  |  |
|  | $\alpha$ | $1.0$ for SiDA; both $1.0$ and $1.2$ for SiD^2A |  |  |  |
|  | $\sigma_{\text{init}}$ | 2.5 |  |  |  |
|  | $t_{\max}$ | 800 |  |  |  |
|  | augment, dropout, cres | The same as in EDM and SiD for each corresponding dataset |  |  |  |
| SiD | max memory allocated per GPU | 13.0 | 46.7 | 31.1 | 31.1 |
| max memory in GB reserved per GPU | 13.3 | 48.0 | 31.3 | 31.3 |  |
| $\sim$seconds per 1k images | 3.3 | 3.1 | 1.1 | 1.1 |  |
| $\sim$hours per 10M ($10^{4}$k) images | 9.2 | 8.6 | 3.1 | 3.1 |  |
| $\sim$days per 100M ($10^{5}$k) images | 3.8 | 3.6 | 1.3 | 1.3 |  |
| SiDA | max memory allocated per GPU | 13.0 | 46.7 | 31.1 | 31.1 |
| max memory in GB reserved per GPU | 13.4 | 48.1 | 31.3 | 31.3 |  |
| $\sim$seconds per 1k images | 3.6 | 3.5 | 1.2 | 1.2 |  |
| $\sim$hours per 10M ($10^{4}$k) images | 10.0 | 9.7 | 3.3 | 3.3 |  |
| $\sim$days per 100M ($10^{5}$k) images | 4.2 | 4.1 | 1.4 | 1.4 |  |

Figure: Figure 6: Unconditional CIFAR-10 32X32 random images generated with SiD^2A (FID: 1.499).
Refer to caption: extracted/5969463/sida_images/fakes_028160.png

Figure: Figure 7: Label conditioning CIFAR-10 32X32 random images generated with SiD^2A (FID: 1.396)
Refer to caption: extracted/5969463/sida_images/fakes_053760.png

Figure: Figure 8: Label conditioning ImageNet 64x64 random images generated with SiD^2A (FID: 1.110)
Refer to caption: extracted/5969463/sida_images/fakes_021471.png

Figure: Figure 9: FFHQ 64X64 random images generated with SiD^2A (FID: 1.040)
Refer to caption: extracted/5969463/sida_images/fakes_168960.png

Figure: Figure 10: AFHQ-V2 64X64 random images generated with SiD^2A (FID: 1.276)
Refer to caption: extracted/5969463/sida_images/fakes_175104.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: -	Why $L^1(\theta)$ is still needed with the additional discriminator, i.e., why SiDA still needs tuning $\alpha$?
soundness: 3
strengths: -	The authors repurpose the fake score network as an additional discriminator with direct using the average of the encoder features as the discriminator score. This design introduces the adversarial objective to the original SiD with minimal additional parameter cost.
-	The proposed method SiDA significantly outperforms SiD and achieves superior 1-step generation performance on various datasets.
summary: The authors propose SiDA (Score identity Distillation with Adversarial Loss), an enhanced version of Score identity Distillation (SiD) that aims to improve diffusion model training efficiency. The key innovation is the introduction of an adversarial loss that helps distinguish between real and generated latents at different noise levels. By reusing the fake score network in the original SiD, the authors propose a discriminator design that utilizes the encoder feature map to obtain the discriminator score. The proposed method achieves SOTA results on CIFAR-10, ImageNet-64, and other datasets.
weaknesses: -	The discriminator design is heavily based on the U-Net encoder-decoder architecture. It is not straightforward to generalize to transformer-based diffusion models, e.g., DiT, where there is no clear partition of the encoder and decoder.
-	The current submission lacks the evaluation of Diffusion models on higher resolution, e.g., ImageNet 256x256. The cost overhead from both discriminator training and maintaining an additional fake score network can be exaggerated on high-resolution data.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: - How does the method perform as the number of available real images changes? What happen if you use only 10% of the training real images?
- Why not apply latent diffusion for larger images? Using VAE to compress the image would help to see if the method can scale to bigger images, without increasing the training time
- Does the approach generalize to teacher models other than VP-EDM?
- The paper arbitrarily sets $\lambda_{sid}$ = 100 and $\lambda_{adv}$ = 0.01 without discussion or ablation, although these hyperparameters likely play an important role in the method. How do these values influence the performance metrics?
- In Figure 1, for $\alpha$=1.2 (the yellow line), after approximately 1000 iterations, the Inception Score (IS) drops and the FID increases sharply; additionally, SiDA with $\alpha$=1.2 is absent from Table 1. Does this imply that the model fails to converge?
soundness: 3
strengths: - Achieves state-of-the-art performance with only one inference step
- The methods demonstrates improvement over the teacher network thanks to the real images
- Training remains relatively stable despite the use of an adversarial loss
summary: The paper introduces Score Identity Distillation with Adversarial loss. The method aims to distill a teacher diffusion model to reduce the number of inference steps to only one. The primary contribution is the combination of SiD with the use of real images and adversarial objective, which not only reduces the number of required inference steps but also improves generation quality, achieving a lower FID score across Cifar10, ImageNet-64, FFHQ-64 and AFHQ-64.
weaknesses: - Requires real images for training, which is not necessary for other distillation methods
- Experiments are demonstrated only on very small images resolution
- Ablation studies are limited to the parameter $\alpha$
- A schematic of the framework would greatly improve the paper's readability.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see the Weakness section.
soundness: 3
strengths: - The discussions in the experiment section are very comprehensive. The detailed analysis of the training statistics across the training steps is appreciated. 

- The paper presented the background such as SiD and adversarial loss in diffusion well.
summary: This paper introduces an improved version of score identity distillation and proposes training SiD with an adversarial objective. 
The main motivation is that the lack of real data in the original training of SiD can potentially lead to discrepancies to the original data distribution. The authors propose to reuse the encoder part of the fake score model as the backbone of discriminator, and use its predicted realness and fakeness scores to improve the generation results further, at no cost of additional model parameters. By combining SiD with adversarial training, together with additional techniques such as initializing the student model with the SiD pretrained model, the proposed method achieved improved one-step diffusion results on multiple datasets.
weaknesses: - Presentation

In the technical parts of the paper, which is mainly the Section 4. I frequently find the presentations not clear enough. My biggest confusion arises from the 'Loss Construction that Respects the Weighting of Diffusion Distillation Loss' paragraph. I understand that balancing loss terms can be tricky, and I acknowledge that this is an important topic to discuss. But from what the author presents, I didn't find how the proposed weight 'honors the existing weighting mechanisms' and, more importantly, automatically balances the loss regardless of the datasets. I read the Eq 7 and Eq 8 and found nothing particularly new. How is Eq 7 fundamentally different from PatchGAN? 

- Novelty

Although the paper targets a problem of SiD that is not previously discussed, the solution of adding an adversarial term is by no means a novelty under the context of diffusion distillation. The adversarial object has been proven very effective in many works, such as UFOGen and SD3-turbo. The proposed method here is a simple mixture of two existing methods. 

- Evaluation and performance

Most of the experiment settings used in this paper are small-scale. And the authors only compared the proposed method with very few other methods in the 512x512 ImageNet experiment. Since efficiency is one of the main advantages the authors are trying to advocate, it is necessary to include larger-scale evaluations, such as text-to-image experiments, to allow for fair comparisons with recent methods. 

I also have a slight concern about the advantage of efficiency. It is true that, according to Figure 3, the FID of the proposed method decreases faster compared to the baseline SiD in the later half of the training; I'd like to point out that this is achieved with an additional loss term, meaning each iteration probably optimizes the model with a larger gradient magnitude. Also, the adversarial loss term will inevitably introduce more computation, so the model is not necessarily able to finish training faster compared to SiD.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: + Why can incorporating average "fakeness" within each GPU batch into each pixel's loss solve the gap between diffusion distillation and adversarial learning?
+ Why does the proposed method can achieve faster convergence than baseline when trained from scratch?
soundness: 3
strengths: + This paper proposes an intuitive extension of score identity distillation and effectively improves the generation performance on small-scale datasets, such as CIFAR and ImageNet 64x64.
+ The proposed method achieves better performance than the main baseline, SiD and other diffusion methods.
summary: This paper proposes an intuitive method, which mainly focuses on the improvement of score identity distillation method by introducing additional real images for adversarial training. The improvement is two fold. First, additional real images can boost generation performance over data-free score identity distillation by discrimating real and synthetic images. Second, it achieves faster convergence rate relative to the baselines. The experimental results on small-scale dataset support the superority of the proposed method.
weaknesses: + More state-of-the-art generative methods are missing, such as DiT, RDM and MAR.
+ The method only conducts experiments on small-scale datasets or low-resolution images. I'm concern about the effectiveness of this method on large-scale dataset with high-resolution images. More results on large-scale datasets with high-resolution images consolidate the effectiveness of this method.
+ OSGAN[R1] adopts an one-step adversarial training on generative task. The comparion with OSGAN is needed to distinguish their differences and highlight the contributions of this paper.
+ This paper doesn't mention any plan about open source of code and trained weights. In spite of encouraging performance, I'm concern about the reproducibility of this paper. 
+ Some motivations are not well clarified in the paper. See Questions section.
+ Some unnecessary equations, such as Eq between 149 and 151, can be removed.

[R1] Training Generative Adversarial Networks in One Stage. CVPR 2021.

Meta-review: This paper improves score identity distillation by integrating real data with adversarial loss, which enhances generation quality even surpassing the teacher in one-step generation, while also improving distillation efficiency. Overall, this work receives positive scores, and the reviewers think the idea of introducing real data is interesting, the method is well designed, and the performance gain is significant. Some concerns include: 1) the method focuses only on U-Net without considering DiT, 2) most experiments are small-scale; and 3) the use of adversarial loss is not new. AC agrees this is an interesting advance on score identity distillation, and recommends acceptance as poster.

Decision: Accept (Poster)