One Step Diffusion-based Super-Resolution with Time-Aware Distillation

## Abstract

Abstract Diffusion-based image super-resolution (SR) methods have shown promise in reconstructing high-resolution images with fine details from low-resolution counterparts. However, these approaches typically require tens or even hundreds of iterative samplings, resulting in significant latency. Recently, techniques have been devised to enhance the sampling efficiency of diffusion-based SR models via knowledge distillation. Nonetheless, when aligning the knowledge of student and teacher models, these solutions either solely rely on pixel-level loss constraints or neglect the fact that diffusion models prioritize varying levels of information at different time steps. To accomplish effective and efficient image super-resolution, we propose a time-aware diffusion distillation method, named TAD-SR. Specifically, we introduce a novel score distillation strategy to align the data distribution between the outputs of the student and teacher models after minor noise perturbation. This distillation strategy enables the student network to concentrate more on the high-frequency details. Furthermore, to mitigate performance limitations stemming from distillation, we integrate a latent adversarial loss and devise a time-aware discriminator that leverages diffusion priors to effectively distinguish between real images and generated images. Extensive experiments conducted on synthetic and real-world datasets demonstrate that the proposed method achieves comparable or even superior performance compared to both previous state-of-the-art (SOTA) methods and the teacher model in just one sampling step. Codes are available at https://github.com/LearningHx/TAD-SR .

###### Index Terms:

## I Introduction

Image super-resolution (SR), a cornerstone task in low-level vision, involves reconstructing high-resolution (HR) images with intricate details from low-resolution (LR) counterparts. Owing to the inherent ill-posed nature of this task, as multiple high-resolution reconstructions are plausible for a given low-resolution input, presenting a persistent and perplexing challenge. Recently, the diffusion model [, ], a novel generative model, has garnered increasing attention for its capacity to model complex data distributions. It has gradually emerged as a successor to Generative Adversarial Networks (GANs) [] in various downstream tasks, including image editing [, ], image inpainting [, ] and image super-resolution [, ].

Specifically, existing diffusion-based image super-resolution methods can be broadly categorized into two streams: one involves feeding low-resolution images along with noise into the diffusion model as input [, ], while the other [, ] adapts SR tasks by modifying the sampling process on a pre-trained diffusion model. While these methods have demonstrated promising results, generating HR images typically demands tens or even hundreds of iterative samplings, significantly impeding their practical application and further advancement.

Figure: Figure 1: Qualitative comparisons on one typical real-world example of the proposed method and recent state of the arts, including BSRGAN [], RealESRGAN [], SwinIR [], DASR [], LDM [], StableSR [], ResShift [], and SinSR []. We mark the number of sampling steps of diffusion-based SR method with the format of “ Method-A” for more intuitive visualization, where “A” is the number of sampling steps. Note that LDM contains 1000 diffusion steps in training and is accelerated to “A” steps using DDIM [] during inference. Please zoom in for a better view.
Refer to caption: x1.png

To enhance the inference efficiency of diffusion models, various acceleration techniques have been proposed, such as the development of numerical samplers [, ] and the applications of knowledge distillation [, ]. However, employing numerical solvers to accelerate sampling speed often compromises performance. Moreover, the majority of methods employing knowledge distillation are tailored for text-to-image generation, where ensuring the diversity of generated images is crucial. In contrast, super-resolution tasks necessitate output images with high visual fidelity relative to LR images. Therefore, directly applying these methods to super-resolution tasks presents significant challenges. For the SR task, ResShift [] has improved the sampling efficiency of diffusion-based SR models by utilizing information from LR images to reformulate the diffusion process, thereby reducing the number of sampling steps to 15. Furthermore, SinSR [] merges distillation techniques with a cycle consistency approach to refine the ResShift model into a single inference step. Nonetheless, it still yields blurry results in the high-frequency details of the image, despite ensuring the visual fidelity of the generated image through pixel-level loss constraints. Recently, AddSR [] employs adversarial diffusion distillation (ADD) [] for SR task to enhance sampling efficiency while ensuring the perceptual quality of generated images. However, it inherits the issue of image oversaturation present in score distillation sampling. Additionally, it relies on a pre-trained DINOV2 discriminator in pixel space, which is both expensive and challenging to optimize.

To address the issues in the aforementioned methods, we introduce a novel time-aware distillation method to accelerate the sampling rate of diffusion-based SR models. It enables the generation of high-resolution images with fine details in a single sampling step. Specifically, considering that diffusion models focus on high-frequency information at small time steps, we have designed a high-frequency enhanced score distillation strategy. This strategy refines the student network by evaluating the difference in the distribution of the teacher model and student model outputs after mild noise disturbance, thereby improving the high-frequency details in the generated images of the student model. To overcome the performance limitations of teacher models, we incorporate generative adversarial learning into the distillation framework, forcing the student model to directly generate samples that lie on the manifold of real images in a single inference step. Specifically, we engineer a time-aware discriminator within the latent space that is capable of differentiating between the data distributions of authentic and synthetically generated images subjected to various perturbations. Experimental evidence confirms that the introduction of temporal information into the discriminator significantly enhances its ability to more effectively regulate the generator’s performance.

Overall, our contributions can be summarized as follows:

- •
We introduce a novel time-aware distillation method that accelerates the diffusion-based SR model into a single inference step.
- •
To improve the high-freauency details of the generated image, we introduce a novel score distillation method that refines the student model by quantifying the discrepancy in scores between the outputs of the teacher and student models at small time steps.
- •
Furthermore, to ensure the performance of student model is not solely bound by the teacher model, we incorporate generative adversarial learning into the distillation framework. A key innovation in our approach is the development of a time-aware discriminator. This discriminator is capable of distinguishing between the data distributions of real and generated images that have undergone various perturbations in latent space.
- •
Extensive experiments on both general and facial datasets have demonstrated that our method, using only single-step sampling, achieves performance that is comparable to or surpasses state-of-the-art methods and teacher models.

## II Related Work

### II-A Image Super-Resolution.

Traditional methods [, , ] for image super-resolution rely on manual design of image priors based on subjective knowledge to restore image details. With the advancement of deep learning (DL), DL-based image super-resolution has become predominant, which mainly focused on network architecture [, , , ], image priors [, ], loss functions [, ], and other aspects [, ]. Recently, diffusion-based methods for image super-resolution have garnered widespread attention. SR3 [] incorporated low-resolution images as conditions into the denoising model to guide the sampling process. Subsequently, CDPMSR [] and IDM [] respectively utilized preprocessed images and features as conditions to enhance the perceptual quality. In order to reduce the computational cost of the model, StableSR [] and PASD [] utilized the powerful generation priors of stable diffusion (SD) [] to achieve image super-resolution in latent space through diffusion processes. However, these methods typically require dozens or even hundreds of iterations to generate high-resolution images. To enhance the inference efficiency, ResShift [] redesigned the diffusion process by shifting the residuals between high-resolution and low-resolution images to construct a Markov chain, achieving performance comparable to previous state-of-the-art methods with just 15 sampling steps.

### II-B Accelerating Diffusion Models.

Although diffusion model [, ] has formidable generation capabilities, the substantial number of inference steps poses a significant obstacle to its practical implementation leading to methods reducing the inference steps. Mainstream approaches include the development of high-order samplers [, , ] and the application of knowledge distillation techniques [, , , , ]. Denoising diffusion implicit models (DDIM) [], an early contribution, introduced a deterministic sampling method that notably decreased the number of diffusion sampling steps. DPMSolver [] proposed a fast dedicated high-order ODE solver, further reducing the diffusion sampling steps to 20. However, trajectory compression through numerical solvers often results in performance degradation, necessitating over ten inference steps to generate samples. In contrast, progressive distillation [] gradually reduces the inference steps of student models through multi-stage distillation, but the accumulation of errors in each distillation stage may affect the performance of the student model. Consistency model [] eliminates the need for computation-intensive iterations by applying consistency regularization to ODE trajectories. Additionally, Adversarial diffusion distillation (ADD) [] integrates generative adversarial networks with score distillation to enhance the perceptual quality of student network-generated images. For image super-resolution tasks, AddSR [] introduces two key advancements based on adversarial distillation technology, effectively fulfilling image super-resolution objectives. Inspired by cycle consistency loss, SinSR [] proposes a single-step image super-resolution method. However, AddSR overlooks the influence of time steps on the discriminator, while SinSR primarily focuses on constraining latent codes through pixel-level loss, neglecting perceptual distribution alignment. To achieve image super-resolution more efficiently and effectively, this work propose a time-aware diffusion distillation method.

Figure: Figure 2: We inject varying degrees of noise into the outputs of the student model, the teacher model, and the corresponding real data (HR images). These noisy data are then fed into a pre-trained diffusion-based SR model to obtain their denoising scores(visualized as clean data prediction $\hat{z}_{0}$).
Refer to caption: x2.png

## III Preliminary

Diffusion models is a type of probabilistic generative model, which utilize a Markov chain transforms complex data distribution $z_{0}\sim p_{data}$ into noise distribution $z_{T}\sim\mathcal{N}\left(0,I\right)$ and recover the data by gradually removing the noise. In image supre-resolution tasks, Resshift [] changes the initial state of diffusion model and constructs a new Markov chains to generate high-resolution images. The forward process can be mathmatically expressed as follows:

$$ $q\left(z_{t}|z_{0},y\right)=\mathcal{N}\left(z_{t}|z_{0}+\eta_{t}\left(z_{y}-z _{0}\right),\kappa^{2}\eta_{t}I\right)$ (1) $$

where $z_{0}$ and $z_{y}$ represent the latent codes obtain by encoding the HR images $x$ and LR images $y$, respectively.
$\eta_{t}$ is a serial of hyper-parameters that monotonocally increases with timestep $t$ and satifies $\eta_{0}\rightarrow 0$ and $\eta_{T}\rightarrow 1$, $\kappa$ is a hyper-parameter controlling the noise variance. Baed on this forward process, the reverse process will commence from the initial sate with rich information in low-resolution images to perform denoising. The formula is as follows:

$$ $q\left(z_{t-1}|z_{t},z_{0},y\right)=\mathcal{N}\left(z_{t-1}\big{|}\frac{\eta_ {t-1}}{\eta_{t}}z_{t}+\frac{\alpha_{t}}{\eta_{t}}z_{0},\kappa^{2}\frac{\eta_{t -1}}{\eta_{t}}\alpha_{t}\bm{I}\right)$ (2) $$

where $\alpha_{t}=\eta_{t}-\eta_{t-1}$, To mitigate the influence of randomness on distillation [], we reformulate Eq. 2 to employ deterministic sampling, as follows:

$$ $q\left(z_{t-1}|z_{t},z_{0},y\right)=\delta\left(k_{t}z_{0}+m_{t}z_{t}+j_{t}z_{ y}\right)$ (3) $$

where $\delta$ is the unit impulse, $m_{t}=\sqrt{\frac{\eta_{t-1}}{\eta_{t}}}$, $j_{t}=\eta_{t-1}-\sqrt{\eta_{t-1}\eta_{t}}$ and $k_{t}=1-j_{t}-m_{t}$. The details of the derivation can be found in SinSR []. In the backward process Eq. 3, $x_{0}$ is usually predicted by a trainable deep neural network $f_{\theta}$ with parameter $\theta$. The training objective function of $f_{\theta}$ as follows:

$$ $\min_{\bm{\theta}}\sum\nolimits_{t}w_{t}\|f_{\bm{\theta}}(\bm{z}_{t},\bm{y},t) -\bm{z}_{0}\|_{2}^{2},$ (4) $$

where $w_{t}=\frac{\alpha_{t}}{2\kappa^{2}\eta_{t}\eta_{t-1}}$. In practice omitting this weight often leads to performance improvement [].

Score Distillation Sampling (SDS) is a distillation technique for pre-trained diffusion models, effectively applied in generating 3D assets [, ] and accelerating diffusion models [, ]. It leverages the rich generative prior of diffusion models to optimize the generated images or the geenrator, which can be expressed as follows:

$$ $\nabla_{\theta}\mathcal{L}_{SDS}(\bm{z},y,\epsilon,t)=(\epsilon_{\phi}(\bm{z}_ {t},y,t)-\epsilon)\frac{\partial{\bm{z}_{t}}}{\partial{\theta}}$ (5) $$

where $z_{t}$ refers to the noised version of $z$. According to [], the U-Net jacobian term $\frac{\partial{\epsilon_{\phi}(\bm{z},y,t)}}{\partial{\bm{z}_{t}}}$ is omitted in Eq. 5 to lead an effective gradient.

Figure: Figure 3: Method overview. We train our student model to map a noisy latent $z_{T}$ to clean latent $z_{0}^{stu}$ through single step sampling. To match the student model’s output with the multi-step sampling outputs of the teacher model, we optimize the student model using both vanilla distillation loss and our proposed high-frequency enhanced score distillation. Additionally, to further improve the performance of the student model, we propose a time-aware discriminator that provides effective supervision through generative adversarial training.
Refer to caption: x3.png

## IV Methodology

### IV-A Motivation

To explore distillation methods more applicable to accelerating diffusion-based SR models, we conducted an experiment to visualize the score differences among the single-step output of the student model, the multi-step output of the teacher model, and the high-resolution images following noise perturbation in latent space. (Note that the student model has the same weight as the teacher model.) Specifically, we re-noise the outputs of the student model, the teacher model, and the real data (HR image) to varying degrees, then feed the noise data into a pre-trained diffusion-based SR model to view the denoising scores (visualized as clean data prediction $\hat{z}_{0}$ in Fig. 2) and compare their differences of scores.

From the first row of Fig. 2, it can be seen that even when real data are fed into a well-trained diffusion-based SR model, there is still a gap between the predicted scores $\hat{z}_{0}$ and the real scores $z_{0}$. This indicates that even in ideal situations, score-based distillation (SDS) itself has biases, consistent with the conclusions of previous related work [, ]. Therefore, utilizing SDS to accelerate diffusion-based SR models is not optimal. To identify the shortcomings of the student model and design a suitable distillation strategy, we further analyzed the score differences $|\hat{z}_{0}^{tch}-\hat{z}_{0}^{stu}|$ between the outputs of the teacher and student models after noise disturbance, as illustrated in the second row of Fig. 2. From the figure, it is evident that there is a notable difference in denoising scores between the output of the teacher model and that of the student model when subjected to minor noise disturbances. Additionally, diffusion models primarily focus on high-frequency information in images at small time steps. Based on these observations, it can be concluded that the student model’s output mainly lacks high-frequency details compared to the teacher model. Therefore, we believe that computing the score differences between the outputs of the teacher and student models under mild noise disturbances can provide an effective gradient direction to guide the optimization of the student model.

In addition, the third row in Fig. 2 visualizes the score difference $|\hat{z}_{0}-\hat{z}_{0}^{stu}|$ between the output of the student model and the real data after noise disturbance. It can be seen that this score difference is significant, and this difference changes with time steps. However, existing methods do not consider this factor when distinguishing the distribution of real data and generated data after noise disturbance. Instead, they directly use a discriminator to distinguish the differences between the data distributions of real and generated images subjected to various perturbations, which poses a challenge to the optimization of the discriminator.

### IV-B TAD-SR

The overview framework of our proposed TAD-SR is illustrated in Fig. 3, consisting of a teacher model $F_{\phi}$ parameterized by $\phi$, a student network $f_{\theta}$ initialized from the teacher model with weights $\theta$, and a trainable time-aware discriminator $D_{\psi}$ parameterized by $\psi$. During training, the student model generates samples from noisy data and computes the mean squared loss against the samples generated iteratively by the teacher model. Subsequently, we introduce slight noise to the samples produced by both the student and teacher models, predict the score function via the teacher model, and refine the student network by leveraging the discrepancy between the two score functions. Furthermore, to mitigate the performance constraints of the teacher model on the student model, we design a time-aware discriminator built upon the encoder of the pre-trained teacher model, enhancing the perceptual quality of the generated samples through adversarial training processes.

Vanilla distillation. We utilize the multi-step output results of the teacher model as the learning objective for the student model. It guides the student model to establish a mapping between low-resolution and high-resolution images through single-step inference. The distillation loss is formulated as follows:

$$ $\mathcal{L}_{distill}=\|F_{\phi}\left(z_{T},T,y\right)-f_{\theta}\left(z_{T},T ,y\right)\|_{2}^{2}$ (6) $$

where $z_{T}$ is obtained through the forward process Eq. (1). Specifically, when time step $t=T$, noisy latent $z_{T}\sim\mathcal{N}\left(x_{t};y,\kappa^{2}\eta_{t}\bm{I}\right)$.

Figure: (a) SDS []
Refer to caption: x4.png

High-frequency enhanced score distillation. As analyzed in Section IV-A, employing SDS [] to accelerate diffusion-based SR models is not an optimal solution. Its inherent bias may introduce meaningless gradient directions to the student model, leading to output blurring, oversaturation, and other issues [, ]. To eliminate this bias, DMD [] trains a new diffusion model to learn the score function of samples generated by the student model and updates the generator based on the difference between the score functions predicted by the new model and the teacher model. However, this approach involves a complex training process that requires alternating training between the student model and the new diffusion model.

By contrast, based on the observations presented in Fig. 2, we develop an effective and efficient score distillation method. Specifically, we analyze the score differences between the outputs of the teacher model and the student model under various noise disturbances. Experiments indicate that these differences are primarily significant under mild noise disturbances (i.e., small time steps). Given that diffusion models typically predict high-frequency information in images at small time steps, this suggests that images generated by student models are predominantly deficient in high-frequency details compared to those produced by teacher models. Consequently, we propose a high-frequency enhanced score distillation method. This approach optimizes the student model by computing the score differences between the outputs of the teacher model and the student model under slight noise disturbances, thereby enhancing the student model’s ability to accurately capture and reproduce high-frequency details. Specifically, we apply slight Gaussian noise to perturb the outputs of both the student and teacher models and model their data distributions through the teacher model. Subsequently, we optimize the student model by calculating the differences between these distributions. According to score distillation, the following formula is derived:

$$ $\mathcal{L}_{hsd}=\mathbb{E}_{z_{t^{{}^{\prime}}}^{tch},z_{t^{{}^{\prime}}}^{ stu},y}\left[\omega\left(\epsilon_{\phi}\left(z_{t^{{}^{\prime}}}^{stu},t,y \right)-\epsilon_{\phi}\left(z_{t^{{}^{\prime}}}^{tch},t,y\right)\right)\right],$ (7) $$

where $\omega=1/CS$ is a weighting function, $C$ is the number of channels and $S$ is the number of spatial pixels. $z_{t^{{}^{\prime}}}^{tch}$ and $z_{t^{{}^{\prime}}}^{stu}$ are the noise data obtained by adding noise to the outputs of the teacher model $z_{0}^{tch}$ and the output of student model $z_{0}^{stu}$, respectively, through Eq. 1. Since we primarily supplement the high-frequency details of the generated images of the student model, time step $t^{{}^{\prime}}$ in Eq. 1 are uniformly sampled in $[1,3]$. Note that during the loss backpropagation in Eq. 7, similar to SDS [], we omit the U-Net Jacobian matrix term. According to Eq. 1, we can simplify Eq. 7 as the following formula:

$$ $\mathcal{L}_{hsd}=\omega_{2}\left(z_{0}^{stu}-z_{0}^{tch}+F_{\phi}\left(z_{t^{ {}^{\prime}}}^{tch},t,y\right)-F_{\phi}\left(z_{t^{{}^{\prime}}}^{stu},t,y \right)\right).$ (8) $$

where $\omega_{2}=\frac{\omega\left(1-\eta_{t^{{}^{\prime}}}\right)}{\sqrt{\eta_{t^{{
}^{\prime}}}}\kappa}$. The details of the derivation can be found in the appendix.

From the above equation, it can be seen that when the output of the student model is the same as that of the teacher model, the loss is zero, and there is no additional bias. Compared to SDS [, ], our proposed score distillation provides more meaningful gradient guidance for student models.

Time-aware adversarial loss. ADD [] has demonstrated that combining diffusion models with generative adversarial networks can significantly improve the perceptual quality of generated images. However, this approach relies on pre-training the DINOv2 discriminator in pixel space, which is both costly and complex. To reduce training costs and enhance model performance, LADD [] employed a pre-trained diffusion model for generative adversarial training in latent space. Although LADD reduces the training costs associated with adversarial diffusion distillation, it overlooks the significant correlation between the features extracted by the diffusion model and the time steps. By employing only a single discriminator, which lacks any temporal information, to differentiate between the distribution differences of real and synthetic data across various noise disturbances, LADD introduces a challenge to the optimization of the discriminator. To address this issue, we propose a time-aware discriminator, which is capable of distinguishing between the data distributions of real and generated images that have undergone various perturbations in latent space. Specifically, we first utilize the encoder part of the teacher model to extract multi-scale features $F_{k}$ from both the student model’s output images and real images.

$$ $F_{k}=Enc_{\phi}\left(z_{t},t,y\right),$ (9) $$

where $Enc_{\phi}$ denotes the encoder part of the teacher model, $z_{t}$ represents the noisy latent code after adding noise to the student model output or the real latent code. We use $F_{k}^{stu}$ and $F_{k}^{gt}$ to denote the features extracted from real latent code and the features extracted from the output of the student model, respectively. We then encode the time step $t$ and map it to a set of time embeddings corresponding to the multi-scale feature dimensions using several linear layers. These time embeddings are used to modulate the features:

$$ $\hat{F}_{k}=Norm\left(F_{k}\right)*(1+\gamma_{k})+\beta_{k},$ (10) $$

where $\gamma_{k}$ and $\beta_{k}$ is a set of learnable modulation parameters produced by projecting time step $t$ into embedding space. After modulation, features of different scales are sent to various discrimination heads $D_{\psi,k}$ for evaluation. The final output is obtained by averaging the results from each discrimination head. The corresponding adversarial loss can be formulated as follows:

$$ $\mathcal{L}_{adv}^{f_{\theta}}=-\mathbb{E}_{z_{0}^{stu}}\left[\sum_{k}D_{\psi, k}\left(\hat{F}_{k}^{stu}\right)\right],$ (11) $$

$$ $\displaystyle\mathcal{L}_{\text{adv}}^{D_{\psi}}$ $\displaystyle=\mathbb{E}_{z_{0}^{stu}}\left[\sum_{k}\text{max}\left(0,1+D_{ \psi,k}\left(\hat{F}_{k}^{stu}\right)\right)\right]$ $\displaystyle+\mathbb{E}_{z_{0}}\left[\sum_{k}\text{max}\left(0,1-D_{\psi,k} \left(\hat{F}_{k}\right)\right)\right]$ (12) $$

The total objective. The student network is trained with the above three losses as follows:

$$ $\mathcal{L}_{f_{\theta}}=\mathcal{L}_{distill}+\lambda_{1}\mathcal{L}_{hsd}+ \lambda_{2}\mathcal{L}_{adv}^{f_{\theta}},$ (13) $$

where $\lambda_{1}$ and $\lambda_{2}$ are the hyperparameters to control the relative importance of these objectives.

**TABLE I: Quantitative results of different methods on the dataset of ImageNet-Test. The best and second best results are highlighted in bold and underline. $*$ indicates that the result was obtained by replicating the method in the paper. Running time is tested on NVIDIA Telsa A100 GPU on the $\times$4 (64$\rightarrow$256) SR tasks. The non-trainable parameters, such as the parameters of VQGAN in LDM, are marked with gray color for clarity.**
| Methods | Metrics |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| 6 |  |  |  |  |  |
|  | LPIPS$\downarrow$ | CLIPIQA$\uparrow$ | MUSIQ$\uparrow$ | Runtime (s) | #Params (M) |
| ESRGAN [] | 0.485 | 0.451 | 43.615 | 0.038 | 16.70 |
| RealSR-JPEG [] | 0.326 | 0.537 | 46.981 | 0.038 | 16.70 |
| BSRGAN [] | 0.259 | 0.581 | 54.697 | 0.038 | 16.70 |
| SwinIR [] | 0.238 | 0.564 | 53.790 | 0.107 | 28.01 |
| RealESRGAN [] | 0.254 | 0.523 | 52.538 | 0.038 | 16.70 |
| DASR [] | 0.250 | 0.536 | 48.337 | 0.022 | 8.06 |
| LDM-15 [] | 0.269 | 0.512 | 46.419 | 0.408 | 113.60 + 55.32 |
| StableSR-15 [] | 0.262 | 0.659 | 59.443 | 1.070 | 52.49 + 1422.49 |
| ResShift-15 [] | 0.231 | 0.592 | 53.660 | 0.682 | 118.59 + 55.32 |
| SinSR-1 [] | 0.221 | 0.611 | 53.357 | 0.058 | 118.59 + 55.32 |
| SinSR*-1 | 0.231 | 0.599 | 52.462 | 0.058 | 118.59 + 55.32 |
| TAD-SR-1 | 0.227 | 0.652 | 57.533 | 0.058 | 118.59 + 57.32 |
|  |  |  |  |  |  |

**TABLE II: Quantitative results of different methods on two real-world datasets. The best and second best results are highlighted in bold and underline.**
| Methods | Datasets |  |  |  |
| --- | --- | --- | --- | --- |
| 5 |  |  |  |  |
|  | RealSR | RealSet65 |  |  |
| 5 |  |  |  |  |
|  | CLIPIQA$\uparrow$ | MUSIQ$\uparrow$ | CLIPIQA$\uparrow$ | MUSIQ$\uparrow$ |
| ESRGAN [] | 0.236 | 29.048 | 0.374 | 42.369 |
| RealSR-JPEG [] | 0.362 | 36.076 | 0.528 | 50.539 |
| BSRGAN [] | 0.543 | 63.586 | 0.616 | 65.582 |
| SwinIR [] | 0.465 | 59.636 | 0.578 | 63.822 |
| RealESRGAN [] | 0.490 | 59.678 | 0.600 | 63.220 |
| DASR [] | 0.363 | 45.825 | 0.497 | 55.708 |
| LDM-15 [] | 0.384 | 49.317 | 0.427 | 47.488 |
| StableSR-15 [] | 0.493 | 59.634 | 0.531 | 56.301 |
| ResShift-15 [] | 0.596 | 59.873 | 0.654 | 61.330 |
| SinSR-1 [] | 0.689 | 61.582 | 0.715 | 62.169 |
| SinSR*-1 | 0.691 | 60.865 | 0.712 | 62.575 |
| TAD-SR-1 | 0.741 | 65.701 | 0.734 | 67.500 |
|  |  |  |  |  |

## V Experiments

### V-A Experimental Setup

Training Details. For a fair comparison, we follow the same experimental setup and backbone design as that in [, ]. Specifically, we use the weights of the teacher model (ResShift) to initialize the student model, and then train the model for 30K iterations based on our proposed loss functions. For the real-world image super-resolution task, we set the weighting factor $\lambda_{1}=1$ and $\lambda_{2}=0.02$. For the blind face restoration task, we set $\lambda_{1}=0.1$ and $\lambda=0.2$.

Compared methods. For real-world SR task, we evalute the effectiveness and effeciency of TAD-SR in comparison to several representative SR models, including BSRGAN [], SwinIR [], RealESRGAN [], DASR [], LDM [], ResShift [] and SinSR []. For blind face restoration task, we compare TAD-SR with seven recent BFR methods, including DFDNet [], PSFRGAN [], GFPGAN [], RestoreFormer [], VQFR [], CodeFormer [], and DifFace [].

Metrics. For real-world super-resolution tasks, we utilize PSNR, SSIM [], and LPIPS [], CLIPIQA [] and MUSIQ [] to measure the performance of proposed method. For face blind restoration task, besides the above metrics, we also evaluate methods with identity score (IDS), landmark distance (LMD) and FID. On the real-world datasets, it is worth noting non-reference metrics such as CLIPIQA and MUSIQ are more convicing, because they are specifically designed to assess the realism of images in the absense of ground truth.

Figure: Figure 5: Qualitative comparisons of different methods on two synthetic examples of the ImageNet-Test dataset. Please zoom in for a better view.
Refer to caption: x7.png

Figure: Figure 6: Qualitative comparisons of different methods on two synthetic examples of the CelebA-Test dataset. Please zoom in for a better view.
Refer to caption: x8.png

Datasets. For the real-world image super-resolution task, we train the models on the training set of ImageNet [] following the same pipeline with ResShift [] where the degradation model is adopted from RealESRGAN []. Then, we evaluate our model on one synthetic dataset ImageNet-Test [, ] and two real-word datasets RealSR [] and RealSet65 []. For the blind face restoration task, We train the models on FFHQ dataset [], and the LQ images are synthesized following a typical degradation model used in []. One synthetic dataset CelebA-Test [, ] and three real-world datasets LFW [], WebPhoto and WIDER [] are adopted to evaluate the performance of face restoration model.

### V-B Experimental Results

Evaluation on synthetic datasets. For the real-world SR task, we conducted a comprehensive comparison between TAD-SR and other state-of-the-art methods on the ImageNet-Test dataset, as summarized in Table I and Fig. 5. The following conclusions can be drawn: i) TAD-SR significantly outperforms other methods in terms of non-reference metrics, specifically CLIPIQA and MUSIQ, and achieves second-best results in the full-reference metric LPIPS. It demonstrates the effectiveness of the proposed method. ii) The substantial improvement in non-reference metrics indicates that TAD-SR has the ability to generate images with high perceptual quality and realism. iii) Visual results show that TAD-SR produces images with higher clarity and better visual perception. In summary, the proposed TAD-SR effectively distills the teacher model of ResShift into a single step, generating more realistic image results while maintaining comparable fidelity to the teacher model. For blind face restoration, We used CelebA-Test as the testing dataset, and results are summarized in Table III and Fig. 6. From the perspective of evaluation metrics, the proposed method achieves state-of-the-art results in terms of FID and comparable results in terms of IDS, LDM, CLIPIQA, and MUSIQ, which demonstrates the effectiveness of TAD-SR on blind face restoration. From the visualization results, the generated faces by the proposed method appear more natural and exhibit richer details. For example, in the images of Fig. 6, the results of TAD-SR show more natural and realistic hair, while the facial clarity is higher, with complete and rich details.

**TABLE III: Quantitative results of different methods on the dataset of CelebA-Test. The best and second best results are highlighted in bold and underline.**
| Methods | Metrics |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 8 |  |  |  |  |  |  |  |
|  | LPIPS$\downarrow$ | IDS$\downarrow$ | LMD$\downarrow$ | FID-F$\downarrow$ | FID-G$\downarrow$ | CLIPIQA$\uparrow$ | MUSIQ$\uparrow$ |
| DFDNet [] | 0.739 | 86.323 | 20.784 | 93.621 | 76.118 | 0.619 | 51.173 |
| PSFRGAN [] | 0.475 | 74.025 | 10.168 | 63.676 | 60.748 | 0.630 | 69.910 |
| GFPGAN [] | 0.416 | 66.820 | 8.886 | 66.308 | 27.698 | 0.671 | 75.388 |
| RestoreFormer [] | 0.488 | 70.518 | 11.137 | 50.165 | 51.997 | 0.736 | 71.039 |
| VQFR [] | 0.411 | 65.538 | 8.910 | 58.423 | 25.234 | 0.685 | 73.155 |
| CodeFormer [] | 0.324 | 59.136 | 5.035 | 62.794 | 26.160 | 0.698 | 75.900 |
| DifFace-100 [] | 0.338 | 63.033 | 5.301 | 52.531 | 23.212 | 0.527 | 66.042 |
| ResShift-4 [] | 0.309 | 59.623 | 5.056 | 50.164 | 17.564 | 0.613 | 73.214 |
| SinSR*-1 [] | 0.319 | 60.305 | 4.935 | 55.292 | 21.681 | 0.634 | 74.140 |
| TAD-SR-1 | 0.341 | 59.897 | 5.050 | 41.968 | 16.779 | 0.735 | 75.027 |
|  |  |  |  |  |  |  |  |

**TABLE IV: Quantitative results of different methods on three real-world human face datasets. The best and second best results are highlighted in bold and underline.**
| Methods | Datasets |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| 7 |  |  |  |  |  |  |
|  | LFW | WebPhoto | WIDER |  |  |  |
| 7 |  |  |  |  |  |  |
|  | CLIPIQA$\uparrow$ | MUSIQ$\uparrow$ | CLIPIQA$\uparrow$ | MUSIQ$\uparrow$ | CLIPIQA$\uparrow$ | MUSIQ$\uparrow$ |
| DFDNet [] | 0.716 | 73.109 | 0.654 | 59.024 | 0.625 | 63.210 |
| PSFRGAN [] | 0.647 | 73.602 | 0.637 | 71.674 | 0.648 | 71.507 |
| GFPGAN [] | 0.687 | 74.836 | 0.651 | 73.369 | 0.663 | 74.694 |
| RestoreFormer [] | 0.741 | 73.704 | 0.709 | 69.837 | 0.730 | 67.840 |
| VQFR [] | 0.710 | 74.386 | 0.677 | 70.904 | 0.707 | 71.411 |
| CodeFormer [] | 0.689 | 75.480 | 0.692 | 74.004 | 0.699 | 73.404 |
| DifFace-100 [] | 0.593 | 70.362 | 0.555 | 65.379 | 0.561 | 64.970 |
| ResShift-4 [] | 0.626 | 70.643 | 0.621 | 71.007 | 0.629 | 71.084 |
| SinSR*-1  [] | 0.640 | 72.457 | 0.641 | 73.357 | 0.654 | 73.556 |
| TAD-SR-1 | 0.768 | 74.085 | 0.718 | 71.952 | 0.770 | 73.739 |
|  |  |  |  |  |  |  |

Evaluation on real-world datasets. For the SR task, in addition to evaluating our method on synthetic datasets, we also evaluated the method in real-world scenarios, and results are reported in Table II. As shown in the table, in terms of the CLIPIQA and MUSIQ metrics, the proposed method significantly outperforms other methods with just a single-step inference. Specifically, compared to the ResShift 15-step model serving as our teacher model, these non-reference metrics show a significant improvement after applying TAD-SR. Additionally, visual comparisons are displayed in Fig. 8. To ensure a comprehensive evaluation, we try to include diverse scenarios, such as buildings, animals, and letters. It can be observed that the images generated by TAD-SR appear more naturalistic, as evidenced by the distinct brick textures, as well as the fine and natural-looking polar bear fur. For blind face restoration, we evaluated recent methods on three datasets: LFW, Webphoto, and WIDER. The results are presented in Table IV, and significant conclusions can be drawn as follows. On all three datasets, the proposed method achieves the best CLIPIQA, which outperforms the other methods by a large margin. On the WIDER dataset, the proposed method achieves the second-best MUSIQ. All these results inform that in terms of blind face restoration, TAD-SR can generate images with really high perceptual quality. We provide visual comparisons in Fig. 11. It can be observed that the results obtained by TAD-SR show more realistic hair details, clearer facial contours, and better skin textures.

### V-C Ablation Study

The aforementioned experiments have confirmed the effectiveness of our method in image super-resolution tasks. This section is dedicated to presenting ablation studies that aim to further validate the importance of the crucial modules introduced within our framework.

**TABLE V: Ablation studies of the proposed methods on RealSR and RealSet65 benchmarks. The best results are highlighted in bold.**
| Methods | Settings | RealSR/RealSet65 |  |
| --- | --- | --- | --- |
| 4 |  |  |  |
|  |  | CLIPIQA$\uparrow$ | MUSIQ$\uparrow$ |
| (a) | SDS + GAN | 0.489/0.528 | 57.290/57.567 |
| (b) | SDS + t-GAN | 0.538/0.554 | 60.223/59.627 |
| (c) | HSD + GAN | 0.711/0.729 | 63.550/66.904 |
| Ours | HSD + t-GAN | 0.741/0.734 | 65.701/67.500 |
|  |  |  |  |

High-frequency enhanced score distillation. We first investigate the importance of high-frequency enhanced score distillation. Recall that in Section IV-B, we analyzed how high-frequency enhanced score distillation can provide a meaningful guidance for optimizing student model compared to score distillation sampling (SDS). Here, we further validate its effectiveness on real-world benchmarks. As shown in Table V, compared with SDS, our proposed high-frequency enhanced score distillation can significantly improved the CILIPIQA and MUSIQ scores on both the RealSR and RealSet65 datasets. This indicates that our proposed high-frequency enhanced score distillation can indeed improve the quality of image generation and is superior to SDS.

Time-aware discriminator. It has been proven that introducing generative adversarial training in latent space is easier to optimize and more cost-effective compared to pixel space []. Now, we demonstrate the importance of introducing time injection into the discriminator. Intuitively, when the discriminator does not have time injection, it needs to distinguish the distribution between real data and generated data under different noise disturbances, which is undoubtedly extremely challenging. Adding time injection to the discriminator is equivalent to providing additional information related to the level of noise disturbance, which should improve the performance of the discriminator and provide more effective supervision for the generator. We further validated the above analysis through experiments. As shown in Table V, performance improves with the replacement of the standard discriminator by our proposed time-aware discriminator, regardless of the score distillation technique used.

## VI Conclusion

In this paper, we propose a time-aware distillation method that accelerates diffusion-based super-resolution models to a single inference step. We introduce a high-frequency enhanced score distillation technique that optimizes the generator by calculating the score difference between the outputs of the teacher and student models following noise perturbation, thereby enhancing the high-frequency details in the student model’s output. To elevate the student model’s performance ceiling, we incorporate generative adversarial learning into the diffusion model framework. Specifically, we design a time-aware discriminator that distinguishes between generated and real data in latent space, providing more efficient and effective supervision for the student model. Extensive experiments demonstrate that our method can achieve performance on par with or surpassing that of the teacher model in a single inference step.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: N/A
soundness: 3
strengths: 1. The paper targets at an important problem of distillation of SR diffusion models. While diffusion distillation is a popular research area, it is interesting to see some insight particularly designed for SR models

2. The paper introduces a novel technique to reduce the bias of the score estimate of generated samples in SDS, which particularly fits in the insights from SR.

3. Empirical results shows promising improvements.
summary: This paper proposed a method to distill a super-resolution diffusion model into one step, by combining 3 losses: direct regression loss, GAN loss, and a modified score distillation loss. The main contribution is the score distillation part.
weaknesses: 1. The biggest concern is insufficient baselines. The method compare against a large number of non-diffusion based methods or diffusion based iterative methods, but it lacks comparisons against the most closely related methods: other diffusion distillation algorithms. This method distill a pre-trained SR diffusion model into one step with some specific design for SR, but there are many distillation methods designed for general diffusion models, such as consistency model and the family of distribution matching distillation. The authors should run controlled experiment with the same teacher model with different algorithms to emphasize the relative advantage. For example, personally I found CM works well in distilling SR model into one step, and DMD and its variant can distilled the more complicated T2I model into one step. Their relative performance on SR diffusion is what we really care.

2. It seems like the method requires teacher model to generate clean samples, which can be computationally expensive, even if you pre-compute the data off-line. 

3. The background of SDS and how to reduce the bias is unclear to readers without prior knowledge.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the weakness part.
soundness: 3
strengths: * The paper is well-written.
* Experimental results demonstrate that the proposed method achieves state-of-the-art performance with high efficiency.
summary: This paper introduces a time-aware diffusion distillation method named TAD-SR, which enables the student model to focus on high-frequency image details at smaller time steps and eliminates inherent biases in score distillation sampling. The authors also design a time-aware discriminator that fully leverages the teacher model’s knowledge by injecting time information to differentiate between real and synthetic data. Experimental results demonstrate the effectiveness and efficiency of the proposed method.
weaknesses: * The evaluation is not comprehensive. Some image fidelity metrics are lacking, such as PSNR and SSIM on ImageNet-Test, where the competing methods ResShift and SinSR all reported.

* The improvement over the previous single-step distillation method SinSR is minor. Considering that LPIPS—a crucial metric for perceptual quality—is very important, the increase from 0.221 to 0.227 represents a big drop in quality and is not slight.

* The ablation study examines only the presence or absence of the discriminator, neglecting other important aspects—for example, the number of scales used in the discriminator.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.	Since this is a distillation method, please compare more diffusion-based distillation SR methods, like OSEDiff [1], quantitatively and qualitatively. (Why are the comparison with diffusion-based distillation SR methods missing in some tables and figures?)

2.	Since you claim that TAD-SR can achieve better reconstruction of high-frequency information, please present the spectrum images of the LR input, GT, baseline methods’ reconstruction, and TAD-SR’s reconstruction. Examine the differences in the high-frequency patterns around the periphery of the spectrum images.

3.	Please compare the inference time of TAD-SR and baseline methods.

4.	In Fig. 10 and Fig. 12, TAD-SR’s results appear to contain many fragmented particles, which make the images look sharper at first glance; however, this is actually due to the addition of pseudo-textures or unnatural details. Could you explain the cause of this? For instance, could it be due to the adversarial loss?

5.	Following the concern raised in my 4th question, could you please provide more qualitative  comparisons that contain fine details or small textures?

[1] Rongyuan Wu, et al. One-Step Effective Diffusion Network for Real-World Image Super-Resolution. 


(I apologize for my previous review comments, which were not fully aligned with your article due to a heavy review workload. I am providing corrected feedback here, and if your response addresses these points well, I will consider adjusting the score.)
soundness: 3
strengths: 1.	This paper proposes a time-aware distillation method that accelerates diffusion-based SR models into a single inference step.
2.	The writing of this paper is good.
summary: This  paper proposes a time-aware diffusion distillation method, TAD-SR, to achieve one-step SR inference with competitive performance. It applies a score distillation strategy make efforts to eliminate the inherent bias SDS focus more on high-frequency image details when sampling at small time steps. A time-aware discriminator is also designed to differentiate between real and synthetic data.
weaknesses: See the questions.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: The motivation is not clear. If the proposed method wants to achieve one-step SR, why it is important for student model to learn how to deal with the intermediate steps?

Will increase the inference steps contribute to the improvement of the performance?
soundness: 2
strengths: 1. The topic is interesting and meaningful.
2. Extensive experiments demonstrate that TAD-SR achieves results comparable to or exceeding multi-step diffusion models, espeically in some non-reference IQA metrics.
summary: This paper introduces TAD-SR, a time-aware diffusion distillation method designed to enhance the efficiency and performance of diffusion-based image super-resolution (SR) models. By aligning the student and teacher models with the proposed score distillation strategy and incorporating a time-aware discriminator to distinguish real and synthetic data across varying noise levels, TAD-SR achieves strong performance across several metrics.
weaknesses: 1. The organization of the paper needs improvement, as it is challenging to clearly understand the core idea. For instance, Fig. 2, which aims to illustrate the paper's motivation, has a caption that provides limited information.

2. The paper lacks essential metrics, such as PSNR and SSIM, to evaluate model fidelity. As shown in previous works, there is a trade-off between PSNR, SSIM, and CLIPIQA, MUSIQ. Reporting only LPIPS and non-reference IQA metrics is insufficient to demonstrate performance. Both the main results and ablation studies should include these metrics.

3. Although I understand that StableDiffusionXL also employs adversarial loss, it appears less elegant to me due to the inherent limitations of GANs.

4. In addition to the difficulty of assessing performance without PSNR and SSIM, the reported improvements seem marginal compared to existing methods.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See the Weakness part. 
The author should carefully describe the details of the method to enhance the readability and clarity of the paper. In addition, the comparison of the most relevant methods (including complexity comparison) should be added to clarify the innovation and effectiveness of the method, and the advancement of the method should be proved through relevant experiments.

I tend to improve the score if the author can solve my concerns.
soundness: 2
strengths: 1. The proposed distillation strategy is simple and straightforward, which can eliminate the inherent bias in score distillation sampling (SDS) and enable the student models to focus more on high-frequency image details. 
2. The proposed time-aware discriminator can differentiate between real and synthetic data, contributing to the generation of high-quality images.
3. The presentation of this work is written well and is easy to read.
summary: The author proposes a time-aware diffusion distillation method, named TAD-SR, where a novel score distillation strategy is introduced to align the score functions between the outputs of the student and teacher models after minor noise perturbation. Such distillation strategy eliminates the inherent bias in score distillation sampling (SDS) and enables the student models to focus more on high-frequency image details by sampling at smaller time steps.  Furthermore, a time-aware discriminator is designed to mitigate performance limitations stemming from distillation, which distinguishes the diffused distributions of real and generated images under varying noise disturbance levels by injecting time information.
weaknesses: 1. It is confusing which is the final output of the model when inference, z_0^{stu} or z ̂_0^{stu}? It is not clearly indicated in Figure 4. Please explicitly state in the text and figure.
2. The authors should clarify if the teacher model is used at all during inference, or if it is only used during training. If I understand correctly, only the student model samples one step, and then the teacher model is used later to sample multiple steps to get the final clean latent, so the model performance relies heavily on the performance of the teacher model, and is not exactly efficient.
3. What is the purpose of setting the weighting function (ω = 1/CS )? Please provide intuition for why this weighting function was chosen, and what effect it has on the training process or results. 
4. In order to eliminate the dependence of the proposed method on the teacher model of ResShift, the relevant ablation experiments should be conducted by replacing the different teacher models to validate the effectiveness of the proposed method.
5. The experiments lack comparisons with the most relevant distillation methods, including DMD, DEQ[1], DFOSD[2], etc. Among them, DMD, a new diffusion model, utilizes similar score distillation techniques to the proposed HSD. DEQ and DFOSD are both efficient and relevant diffusion models, which require one-step diffusion distillation or even no distillation.
6. In the experimental section, the authors compare many GAN and transformer-related methods. However, the proposed method is a diffusion model and should be compared with the most relevant diffusion models to validate its efficiency, especially accelerated diffusion models, including OSEDiff[3], DPM++[4], Unipc[5], etc. 
7. The authors claim that the method is designed to accomplish effective and efficient image super-resolution, but did not include a complexity comparison of the different methods (including parameters, sampling steps, running time, MACs, etc.), which is crucial for diffusion models. Please provide a Table to compare these computational complexity metrics with the key baselines.
8. Are there any limit conditions for using the method? The author should discuss and analyze the limitations of the proposed method. It is recommended to add a discussion of a discussion of potential limitations or where the proposed method might not perform as well.

References

[1] Geng Z, Pokle A, Kolter J Z. One-step diffusion distillation via deep equilibrium models[C]. Advances in Neural Information Processing Systems, 2024.

[2] Li J, Cao J, Zou Z, et al. Distillation-free one-dtep diffusion for real-world image super-resolution[J]. arxiv preprint arxiv:2410.04224, 2024.

[3] Wu R, Sun L, Ma Z, et al. One-step effective diffusion network for real-world image super-resolution[J]. arxiv preprint arxiv:2406.08177, 2024.

[4] Lu C, Zhou Y, Bao F, et al. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models[J]. arxiv preprint arxiv:2211.01095, 2022.

[5] Zhao W, Bai L, Rao Y, et al. Unipc: A unified predictor-corrector framework for fast sampling of diffusion models[C]. Advances in Neural Information Processing Systems, 2024.

Meta-review: This paper receives mixed ratings of (5, 5, 5, 6, 6). The reviewers generally agree that the area this paper is exploring is interesting and meaningful, and the simplicity of the method, while having concerns about the comparison and improvement over existing works. The AC carefully read the paper, reviews, and rebuttal, and agree with the reviewers overall. In particular, in the response of the authors, the improvement over OSEDiff cannot be regarded as significant given a slower speed. As a result, the effectiveness of the methods could not be fully verified. While the AC agrees that this paper is an interesting exploration, the AC regretfully recommends a rejection.

Decision: Reject