AddSR: Accelerating Diffusion-based Blind Super-Resolution with Adversarial Diffusion Distillation

## Abstract

Abstract Blind super-resolution methods based on stable diffusion showcase formidable generative capabilities in reconstructing clear high-resolution images with intricate details from low-resolution inputs. However, their practical applicability is often hampered by poor efficiency, stemming from the requirement of thousands or hundreds of sampling steps.
Inspired by the efficient adversarial diffusion distillation (ADD), we design AddSR to address this issue by incorporating the ideas of both distillation and ControlNet.
Specifically, we first propose a prediction-based self-refinement strategy to provide high-frequency information in the student model output with marginal additional time cost.
Furthermore, we refine the training process by employing HR images, rather than LR images, to regulate the teacher model, providing a more robust constraint for distillation.
Second, we introduce a timestep-adaptive ADD to address the perception-distortion imbalance problem introduced by original ADD.
Extensive experiments demonstrate our AddSR generates better restoration results, while achieving faster speed than previous SD-based state-of-the-art models (e.g., 7 7 7 7 × \times × faster than SeeSR).

## 1 Introduction

Blind super-restoration (BSR) aims to convert low-resolution (LR) images that have undergone complex and unknown degradation into clear high-resolution (HR) versions. Differing from classical super-resolution [; ; ; ; ], where the degradation process is singular and known, BSR is crafted to enhance real-world degraded images, imbuing them with heightened practical value.

Generative models, e.g. generative adversarial network (GAN) and diffusion model, have demonstrated significant superiority in BSR task to achieve realistic details. GAN-based models [; ; ; ; ; ] learn a mapping from the distribution of input LR images to that of HR images with adversarial training.
However, when handling natural images with intricate textures, they often struggle to generate unsatisfactory visual results due to unstable adversarial objectives [; ].

Figure: Figure 1: Comparisons on effect and efficiency. AddSR-$4$ indicates the result is obtained in $4$ steps, achieving high perception quality restoration performance with the fastest speed among diffusion-based models. In contrast, existing SD-based BSR models suffer from either low perception quality restoration performance (e.g., ResShift) or time-consuming efficiency (e.g., SeeSR-$50$).
Refer to caption: extracted/2404.01717v3/introduction.png

Recently, diffusion models (DM) [; ] have garnered significant attention owing to their potent generative capabilities and the ability to combine information from multiple modalities.
DM-based BSR methods can be roughly divided into two categories: those without Stable Diffusion (SD) prior [; ; ], and those incorporating SD prior [; ; ].
SD prior can significantly enhance the model’s ability to capture the distribution of natural images [], thereby enabling the generated HR images with realistic details.
Given the iterative refinement nature of DM, diffusion-based methods typically outperform GAN-based ones, albeit at the expense of efficiency. Hence, there’s an urgent demand for BSR models that deliver exceptional restoration quality while maintaining high efficiency for real-world applications.

In this paper, we propose a novel method called AddSR based on adversarial diffusion distillation (ADD) [] for blind super-restoration, which enhances restoration effects and accelerates inference speed simultaneously.
Specifically, AddSR consists of a teacher model and a student model, incorporating ControlNet [] and CLIP to receive image and text information to achieve multi-modal restoration. To mitigate the degradation impact of LR images and provide high-frequency information for the student model during training, we design a simple yet effective method called prediction-based self-refinement (PSR), which obtains estimated HR image from the predicted noise to control the model output. Meanwhile, the HR image is directly sent to the ControlNet of teacher model as input, aiming to additionally supply high-frequency priors and force the student model to produce high perception quality results conditioned on LR images or estimated HR image.
Moreover, to alleviate the perception-distortion imbalance issue [; ; ], we develop a timestep-adaptive adversarial diffusion distillation (TA-ADD). The core idea involves employing student and teacher timestep-related weights to balance the GAN loss and distillation loss, thereby enhancing the generative ability of the student model at small inference steps while reducing it at larger inference steps.

Our main contributions can be summarized as threefold:

$\bullet$ We propose a novel method called AddSR based on adversarial diffusion distillation (ADD) for blind super-restoration. AddSR achieves high perception quality within $0.8$s, making it $\times$$7$ faster than SeeSR[], while also providing better restoration performance.

$\bullet$ We introduce prediction-based self-refinement (PSR) to provide high-quality controlling signals for regulating output. Moreover, we propose the TA-ADD to achieve perception-distortion trade-off.

$\bullet$ Experiments on popular benchmarks against extensive competitors demonstrate that our proposed AddSR achieves superior restoration effects and high efficiency simultaneously.

## 2 Related Work

GAN-based BSR.
In recent years, BSR have drawn much attention due to their practicability.
Adversarial training [; ; ; ; ] is introduced in SR task to avoid generating over-smooth results.
BSRGAN [] designs a random shuffle strategy to enlarge the degradation space for training a comprehensive SR model.
Real-ESRGAN [] presents a more practical degradation process called “high-order” to synthesize realistic LR images.
KDSRGAN [] estimates the implicit degradation representation to assist the restoration process.
While GAN-based BSR methods require only one step to restore the LR image, their capability to super-resolve complex natural images is limited.
In this work, our AddSR seamlessly attains superior restoration performance based on diffusion model, making it a compelling choice.

Diffusion-based BSR.
Diffusion models has demonstrated significant advantages in image generation tasks (e.g., text-to-image).
One common approach [; ; ; ] is training a non-multimodal diffusion model from scratch, which takes the concatenation of a LR image and noise as input in every step.
Another approach [; ; ; ; ] fully leverages the prior knowledge from a pre-trained multimodal diffusion model (i.e., SD model), which requires training a ControlNet and incorporates new adaptive structures (e.g., cross-attention).
SD-based methods excel in performance compared to the aforementioned approaches, as they effectively incorporate high-level information.
However, the large number of model parameters and the need for numerous sampling steps pose substantial challenges to their application in the real world.

Efficient Diffusion Models.
Several works [; ; ; ; ; ] are proposed to accelerate the inference process of DM.
Although these methods can reduce the sampling steps from thousands to $20$-$50$, the restoration effect will deteriorate dramatically.
Recnet, adversarial diffusion distillation [] is proposed to achieve $1$$\sim$$4$ steps inference while maintaining satisfactory generating ability.
However, ADD was originally designed for the text-to-image task.
Considering the multifaceted nature of BSR, such as image quality, degradation, or the trade-off between fidelity and realness, employing ADD to expedite the SD-based model for BSR is non-trivial.
In contrast, AddSR introduces two pivotal designs to adapt ADD into BSR tasks, making it both effective and efficient.

Figure: Figure 2: Overview of AddSR. Our proposed AddSR consists of a student model, a pretrained teacher model, and a discriminator. Let $\mathcal{C}=\{x_{LR},\hat{x}_{0}^{1},\hat{x}_{0}^{2},\hat{x}_{0}^{3}\}$, where $c_{i}$ denotes the $i$-th element of $\mathcal{C}$ and $i$ stands for $i$-th inference step.
Refer to caption: extracted/2404.01717v3/overview_2.png

## 3 Methodology

### 3.1 Overview of AddSR

Network Components.
The AddSR training procedure primarily consists of three components: the student model with weights $\theta$, the pretrained teacher model with frozen weights $\psi$ and the discriminator with weights $\phi$, as depicted in Fig. 2. Specifically, both the student model and the teacher model share identical structures, with the student model initialized from the teacher model. The student model incorporates a ControlNet [] to receive $x_{LR}$ or predicted $\hat{x}_{0}^{i-1}$ for controlling the output of the U-Net []. Furthermore, the student model utilizes RAM [] to obtain representation embeddings $c_{rep}$, extracting high-level information (i.e., image content) and sends this information to CLIP [] to generate text embeddings $c_{text}$. These embeddings help the backbone (U-Net and ControlNet) produce high-quality restored images. As for the discriminator, we adopt the same structure as StyleGAN-T [] conditioned on $c_{img}$ extracted from $x_{LR}$ by DINOv2 [].

Training Procedure.
(1) Student model with prediction-based self-refinement. Firstly, we uniformly choose a student timestep $s$ from $\{s_{1},s_{2},s_{3},s_{4}\}$ (evenly selected from 0 to 999) and employ the forward process on the HR image $x_{0}$ to generate the noisy state $x_{s}=\sqrt{\overline{\alpha}_{s}}x_{0}+\sqrt{1-\overline{\alpha}_{s}}\epsilon$.
Secondly, we input $x_{s}$ with the condition $c_{i}$, the $i$-th element of $\mathcal{C}=\{x_{LR},\hat{x}_{0}^{1},\hat{x}_{0}^{2},\hat{x}_{0}^{3}\}$ ($\hat{x}_{0}^{i-1}$ is obtained by PSR to reduce the degradation impact and provide more high-frequency information to the restoration process, as detailed in Sec. 3.2), along with $c_{rep}$ and $c_{text}$, into the student model to generate samples $\hat{x}_{0}^{i}(x_{s},s,c_{rep},c_{text},c_{i})$.
(2) Teacher model. Firstly, we equally choose a teacher timestep $t$ from $\{t_{1},t_{2},...,t_{1000}\}$ and employ forward process to student-generated samples $\hat{x}_{\theta}$ to obtain the noisy state $\hat{x}_{\theta,t}=\sqrt{\overline{\alpha}_{t}}\hat{x}_{0}^{i}+\sqrt{1-
\overline{\alpha}_{t}}\epsilon$. Secondly, we put $\hat{x}_{\theta,t}$ with condition $x_{0}$, $c^{\prime}_{rep}$ and $c^{\prime}_{text}$ into teacher model to generate samples $\hat{x}_{\psi}(\hat{x}_{\theta,t},t,c^{\prime}_{rep},c^{\prime}_{text},x_{0})$. Note that $\hat{x}_{\psi}$ is conditioned on $x_{0}$ instead of $x_{LR}$. The primary reason is that substituting $x_{LR}$ with $x_{0}$ to regulate the output of the teacher model can force student model implicitly learning the high-frequency information of HR images even conditioned on $c_{i}$.
(3) Timestep-adaptive ADD for BSR task. It consists of two parts: adversarial loss and a novel timestep-adaptive distillation loss, which is correlated with both the teacher and student model timesteps. The overall objective is:

$$ $\begin{split}\mathcal{L}_{TA-ADD}=&\mathcal{L}_{ta-dis}(\hat{x}_{0}^{i}(x_{s}, s,\rho,c_{i}),\hat{x}_{\psi}(\hat{x}_{\theta,t},t,\rho^{\prime},x_{0}),d(s,t)) +\\ &\lambda\mathcal{L}_{adv}(\hat{x}_{0}^{i}(x_{s},s,\rho,c_{i}),x_{0},\psi_{c_{ img}}),\end{split}$ (1) $$

where $\rho$ denotes the $c_{rep}$ and $c_{text}$, $\rho^{\prime}$ stands for $c^{\prime}_{rep}$ and $c^{\prime}_{text}$. $\lambda$ is the balance weight, empirically set to 0.02. $\psi_{c_{img}}$ is the discriminator conditioned on $c_{img}$. $d(s,t)$ is a weighting function defined by student timestep $s$ and teacher timestep $t$, dynamically adjusting $\mathcal{L}_{ta-dis}$ and $\mathcal{L}_{adv}$ to alleviate perception-distortion imbalance. Further analysis is provided in Sec. 3.3.

### 3.2 Prediction-based Self-Refinement

As shown in Fig. 3, SD-based methods usually use LR images to control the output of DM in each inference step.
However, LR images may suffer from multiple degradations, which can disrupt the restoration process and limit their ability to provide only low-frequency information for controlling the model output (e.g., the first line of Fig. 4).
Previous methods [; ] employ a degradation removal model to preprocess the LR image, aiming to mitigate the impact of degradation.
However, these approaches often sacrifice efficiency, which is unacceptable when designing an efficient model.

Figure: Figure 3: Illustration of the proposed PSR. The previous SD-based methods usually use LR image to guide model’s output, while our PSR additionally utilizes the predicted HR image from the previous step to provide better supervision with marginal additional time cost.
Refer to caption: extracted/2404.01717v3/PSR.png

Figure: Figure 4: Left: Visual comparisons with and without PSR. Right: Perception quality of the control signal at each timestep. MANIQA is calculated between the input of ControlNet and $x_{0}$.
Refer to caption: extracted/2404.01717v3/pbsr_illu_2.png

To address this issue, we propose a novel prediction-based self-refinement approach, which incurs only tiny efficiency overhead.
The core idea of PSR is to fully utilize the predicted noise to enhance image restoration.
Specifically, we use the following equation: $\hat{x}_{0}=(x_{s}-\sqrt{1-\overline{\alpha}_{s}}\epsilon_{\theta,s})/\sqrt{
\overline{\alpha}_{s}}$
to estimate the HR image $\hat{x}_{0}$ from predicted noise in each step, and then control the model output in next step, where $x_{s}$ is the noisy state and $\epsilon_{\theta,s}$ is the predicted noise at timestep $s$.
The $\hat{x}_{0}$ in each step has more high-frequency information to better control the model output (e.g., Fig. 3-right and also Fig. 4).
Moreover, the additional processing time to solve the above equation is nearly negligible.
By leveraging our simple yet effective PSR, AddSR can notably enhance the restoration outcomes without compromising efficiency.

### 3.3 Timestep-Adaptive ADD

Motivation.
Perception-distortion trade-off [] is a well-known phenomenon in SR task.
We observe that training BSR task with ADD directly exacerbates this phenomenon, as shown in Fig. 5(a).
Specifically, during the first three inference steps, there is a significant decrease in fidelity, accompanied by improvement in perception quality.
In the last inference step, fidelity remains at a low level, while perception quality undergoes a dramatic increase.
The aforementioned scenario may give rise to two issues: (1) When the inference step is small, the quality of restored image is subpar. (2) As the inference step increases, the generated images may exhibit “hallucinations”.

Figure: Figure 5: Illustrations of TA-ADD in perception-distortion trade-off. (a) The perception and fidelity variation trends with and without TA-ADD. MANIQA and PSNR stand for perception quality and fidelity, respectively. (b) The outputs at 1st and 4th timesteps. The final output without $\mathcal{L}_{TA-ADD}$ hallucinates the paw into an animal head, while AddSR retains the appearance of the paw.
Refer to caption: extracted/2404.01717v3/ta_loss.png

Figure: Figure 6: Relation between weighting ratio and timesteps. (a) weighting ratio = $\lambda/(\prod_{i=0}^{t}(1-\beta_{t}))^{\frac{1}{2}}$. Once the teacher timestep is established, the weighting ratio remains constant. (b) weighting ratio = $\lambda/d(s,t)$. Even the teacher timestep is established, the weighting ratio can change to balance perception-distortion across different student timesteps.
Refer to caption: extracted/2404.01717v3/weighting_ratio.png

Analysis.
The primary reason lies with ADD, which maintains a consistent weight for GAN loss and distillation loss across various student timesteps, as depicted in Fig. 6(a).
Once the teacher timestep is established, the ratio of adversarial loss and distillation loss remains constant for different student timesteps.
However, since the perception quality of generated images gradually increases with larger inference steps, the weight-invariant ADD may lead to insufficient adversarial constraints on the student model during small inference steps, resulting in the generation of blurry images.
Conversely, as the inference step increases, the adversarial training constraints become too strong, leading to the generation of “hallucinations" (see Fig. 5(b)).

Approach.
To address this issue, we extend the original unary weighting function $(\prod_{i=0}^{t}(1-\beta_{t}))^{\frac{1}{2}}$ to a binary weighting function $d(s,t)$, allowing for dynamic adjustment of the ratio between adversarial loss and distillation loss based on both student timestep and teacher timestep, as shown in Fig. 6(b).
Specifically, we increase this ratio when only one inference step is performed, and gradually decrease it as the inference step increases.
This alleviates the aforementioned issue of generating blurry images with small inference step and “hallucinations" with larger inference steps. We employ the exponential forms to control the weighting ratio. The function $d(s,t)$ can be defined as follows:

$$ $d(s,t)=(\prod_{i=0}^{t}(1-\beta_{t}))^{\frac{1}{2}}\times\mu\cdot\nu^{p(s)-1},$ (2) $$

where $\beta$ represents the noise schedule coefficient, with $t$ and $s$ denoting the teacher timestep and student timestep, respectively.
The hyper-parameter $\mu$ sets the initial weighting ratio, while $\nu$ controls the distillation loss increase over student timesteps, typically resulting in higher fidelity with larger $\nu$.
The function $p(\cdot)$ serves as a projection function that maps student timesteps to inference steps (e.g., mapping $s=999$ to $1$). We primarily consider the exponential and linear forms to control the weighting ratio. A comparison of preferences and detailed settings for different hyper-parameters are provided in Appendix Sec. C. From these comparisons, we find that the exponential form of $d(s,t)$ yields good results, so we use Eq. (2) as the distillation loss function for the remaining experiments.

**Table 1: Quantitative comparison with SotAs on different degradation cases. ’*’ indicates that the metric is non-reference. The best results are marked in red, while the second best ones are in blue.**
| Datasets | Metrics | BSRGAN | Real-ESRGAN | MM-RealSR | LDL | FeMaSR | StableSR-200 | ResShift-15 | PASD-20 | DiffBIR-50 | SeeSR-50 | AddSR-$1$ | AddSR-$4$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ICCV 2021 | ICCVW 2021 | ECCV 2022 | CVPR 2022 | MM 2022 | Arxiv 2023 | NeurIPS 2023 | Arxiv 2023 | Arxiv 2023 | CVPR 2024 | - | - |  |  |
| Single:<br>SR($\times$4) | MANIQA${}^{*}\uparrow$ | 0.3990 | 0.3859 | 0.3959 | 0.3501 | 0.4603 | 0.4088 | 0.4582 | 0.4405 | 0.4680 | 0.5082 | 0.3894 | 0.6430 |
| MUSIQ${}^{*}\uparrow$ | 66.06 | 63.32 | 64.22 | 61.10 | 65.31 | 65.46 | 65.50 | 66.80 | 67.61 | 68.88 | 63.05 | 71.43 |  |
| CLIPIQA${}^{*}\uparrow$ | 0.5951 | 0.5367 | 0.5967 | 0.5120 | 0.6773 | 0.6483 | 0.6803 | 0.6396 | 0.6934 | 0.7039 | 0.5572 | 0.7794 |  |
| NIQE${}^{*}\downarrow$ | 5.01 | 5.21 | 5.22 | 5.39 | 5.80 | 5.35 | 5.74 | 4.68 | 4.88 | 5.06 | 5.31 | 4.75 |  |
| LPIPS${}^{*}\downarrow$ | 0.2003 | 0.1962 | 0.1934 | 0.1892 | 0.1770 | 0.1944 | 0.1544 | 0.1891 | 0.2388 | 0.3085 | 0.2872 | 0.2812 |  |
| PSNR$\uparrow$ | 25.52 | 25.30 | 24.35 | 25.09 | 23.74 | 24.45 | 25.53 | 25.15 | 23.43 | 24.61 | 22.70 | 21.83 |  |
|  | SSIM$\uparrow$ | 0.7091 | 0.7158 | 0.7232 | 0.7282 | 0.6788 | 0.6904 | 0.7206 | 0.6896 | 0.6025 | 0.6709 | 0.6012 | 0.5651 |
| Mixture:<br>Blur($\sigma$=2)+<br>SR($\times$4) | MANIQA${}^{*}\uparrow$ | 0.3823 | 0.3688 | 0.3796 | 0.3337 | 0.4184 | 0.3587 | 0.4195 | 0.4124 | 0.4648 | 0.4974 | 0.3779 | 0.6340 |
| MUSIQ${}^{*}\uparrow$ | 64.73 | 60.89 | 62.21 | 58.64 | 62.96 | 60.85 | 62.02 | 64.41 | 67.09 | 68.27 | 61.95 | 71.11 |  |
| CLIPIQA${}^{*}\uparrow$ | 0.5752 | 0.5116 | 0.5687 | 0.4910 | 0.6390 | 0.5819 | 0.6375 | 0.6026 | 0.6857 | 0.6892 | 0.5389 | 0.7727 |  |
| NIQE${}^{*}\downarrow$ | 5.17 | 5.54 | 5.66 | 5.72 | 5.62 | 5.96 | 6.25 | 5.01 | 5.18 | 5.32 | 5.81 | 6.11 |  |
| LPIPS${}^{*}\downarrow$ | 0.2240 | 0.2267 | 0.2295 | 0.2226 | 0.1979 | 0.2384 | 0.2029 | 0.2223 | 0.2522 | 0.2124 | 0.3007 | 0.2953 |  |
| PSNR$\uparrow$ | 25.07 | 24.74 | 24.20 | 24.45 | 24.00 | 24.01 | 24.95 | 24.70 | 22.97 | 24.12 | 22.57 | 21.69 |  |
|  | SSIM$\uparrow$ | 0.6820 | 0.6890 | 0.6927 | 0.6973 | 0.6730 | 0.6596 | 0.6926 | 0.6688 | 0.5802 | 0.6508 | 0.5905 | 0.5556 |
| Mixture:<br>SR($\times$4)+<br>Noise($\sigma=$40) | MANIQA${}^{*}\uparrow$ | 0.2645 | 0.3120 | 0.3285 | 0.3138 | 0.3123 | 0.3485 | 0.3741 | 0.4270 | 0.4121 | 0.5537 | 0.4320 | 0.6517 |
| MUSIQ${}^{*}\uparrow$ | 50.47 | 53.43 | 56.53 | 53.30 | 56.55 | 52.24 | 60.99 | 64.20 | 61.85 | 70.32 | 65.54 | 71.26 |  |
| CLIPIQA${}^{*}\uparrow$ | 04543 | 0.4761 | 0.5158 | 0.6208 | 0.5178 | 0.4414 | 0.5949 | 0.5503 | 0.6149 | 0.7557 | 0.6219 | 0.7768 |  |
| NIQE${}^{*}\downarrow$ | 7.04 | 6.00 | 4.40 | 5.61 | 4.27 | 5.12 | 6.10 | 5.02 | 5.09 | 4.95 | 4.87 | 6.29 |  |
| LPIPS${}^{*}\downarrow$ | 0.4611 | 0.3601 | 0.3052 | 0.3138 | 0.3267 | 0.4017 | 0.3129 | 0.3451 | 0.3404 | 0.2999 | 0.3546 | 0.3488 |  |
| PSNR$\uparrow$ | 17.90 | 21.97 | 22.04 | 22.68 | 21.84 | 21.20 | 22.78 | 22.12 | 22.22 | 21.04 | 21.01 | 20.79 |  |
|  | SSIM$\uparrow$ | 0.5210 | 0.6044 | 0.5998 | 0.5838 | 0.5421 | 0.5077 | 0.5979 | 0.5587 | 0.5311 | 0.5388 | 0.5684 | 0.5621 |
| Mixture:<br>Blur($\sigma=$2)+<br>SR($\times$4)+<br>Noise($\sigma=$20)+<br>JPEG(q=50) | MANIQA${}^{*}\uparrow$ | 0.3524 | 0.3374 | 0.3287 | 0.3082 | 0.3271 | 0.3452 | 0.3702 | 0.4024 | 0.4538 | 0.5266 | 0.3930 | 0.6335 |
| MUSIQ${}^{*}\uparrow$ | 59.83 | 55.54 | 55.30 | 52.79 | 60.87 | 61.21 | 56.99 | 63.25 | 64.50 | 69.08 | 62.69 | 70.59 |  |
| CLIPIQA${}^{*}\uparrow$ | 0.5380 | 0.5047 | 0.4978 | 0.4699 | 0.6061 | 0.6010 | 0.5888 | 0.5733 | 0.6626 | 0.7180 | 0.5669 | 0.7703 |  |
| NIQE${}^{*}\downarrow$ | 5.31 | 5.69 | 5.70 | 5.77 | 4.87 | 6.33 | 7.03 | 5.49 | 4.93 | 5.06 | 5.13 | 4.68 |  |
| LPIPS${}^{*}\downarrow$ | 0.3223 | 0.3346 | 0.3372 | 0.3272 | 0.2922 | 0.3429 | 0.3526 | 0.3482 | 0.3502 | 0.3085 | 0.3398 | 0.3368 |  |
| PSNR$\uparrow$ | 23.04 | 22.70 | 22.47 | 22.36 | 22.17 | 22.39 | 22.36 | 22.25 | 21.46 | 21.86 | 21.65 | 21.45 |  |
|  | SSIM$\uparrow$ | 0.5866 | 0.5935 | 0.5950 | 0.5948 | 0.5633 | 0.5704 | 0.5574 | 0.5594 | 0.5029 | 0.5474 | 0.5312 | 0.5210 |

## 4 Experiments

### 4.1 Experimental Settings

Training Datasets.
We adopt DIV2K [], Flickr2K [], first $20$K images from LSDIR [] and first $10$K face images from FFHQ [] for training. We use the same degradation model as Real-ESRGAN [] to synthesize HR-LR pairs.

Test Datasets.
We evaluate AddSR on $4$ datasets: DIV2K-val [], DRealSR [], RealSR [] and RealLR200 [].
We conduct $4$ degradation types on DIV2K-val to comprehensively assess AddSR, and except RealLR200, all datasets are cropped to $512$$\times$$512$ and degraded to $128$$\times$$128$ LR image.

Implementation Details.
We adopt SeeSR [] as the teacher model.
Note that our approach is applicable to most of the existing SD-based BSR methods for improving restoration results and acceleration.
The student model is initialized from the teacher model, and fine-tuned with Adam optimizer for $50$K iterations.
The batch size and learning rate are set to $6$ and $2$$\times 10^{-5}$, respectively. AddSR is trained under $512$$\times$$512$ resolution images with $4$ NVIDIA A$100$ GPUs ($40$G).

Evaluation Metrics.
We employ non-reference metrics (i.e., MANIQA [], MUSIQ [], CLIPIQA []) and reference metrics (i.e., LPIPS [], PSNR, SSIM []) to comprehensively evaluate AddSR.
Non-reference metrics are prioritized as they closely align with human perception.

Compared Methods.
Extensive state-of-the-art BSR methods are compared, including GAN-based methods: BSRGAN [], Real-ESRGAN [], MM-RealSR [], LDL [], FeMaSR [] and diffusion-based methods: StableSR [], ResShift [], PASD [], DiffBIR [], SeeSR [].

### 4.2 Evaluation on Synthetic Data

To demonstrate the superiority of the proposed AddSR in handling various degradation cases, we synthesized $4$ test datasets using the DIV2K-val dataset with different degradation processes.
The quantitative results are summarized in Tab. 1.
The conclusions include:
($1$) Our AddSR-$4$ achieves the highest scores in MANIQA, MUSIQ and CLIPIQA across $4$ degradation cases.
Especially for MANIQA, AddSR surpasses the second-best method by more than 16$\%$ on average.
($2$) Diffusion-based models usually achieve low scores in full-reference metrics like PSNR, SSIM and LPIPS, possibly because of their powful generative ability for realistic details that do not exist in GT.
However, full-reference metrics cannot precisely reflect human preferences (see Fig. 7), as discussed in previous works [; ; ].
($3$) AddSR-$1$ can generate comparative results against other SD-based methods except SeeSR, but significantly reduces the sampling steps (i.e., from $\geq$$15$ steps to only $1$ step).

For a more intuitive comparison, we provide visual results in Fig. 8.
One can see that GAN-based method like FeMaSR fails to reconstruct the clean and detailed HR images of the three displayed LR images.
As for SD-based method DiffBIR, it tends to generate wrong texture. This is mainly because DiffBIR uses a degradation removal structure to remove the degradation of LR images.

Figure: Figure 7: Illustration on disparity between full-reference metrics evaluation and human preference. Despite AddSR achieves lower scores in full-reference metrics, it generates human-preferred images.
Refer to caption: extracted/2404.01717v3/explanation.png

However, the processed LR image is blurry, which may lead to the blurry results.
Thanks to our proposed PSR, AddSR uses the predicted $\hat{x}_{0}^{i-1}$ to control the model output, which has more high-frequency information and nearly no extra time cost.
With TA-ADD, AddSR can generate precise images and rich details.
In a nutshell, AddSR can produce images with better perceptual quality than the state-of-the-art models while requiring fewer inference steps and less time.

Figure: Figure 8: Visual comparisons on synthetic LR images. Please zoom in for a better view.
Refer to caption: extracted/2404.01717v3/synthetic.png

Figure: Figure 9: Visual comparisons on real-world LR images. Please zoom in for a better view.
Refer to caption: extracted/2404.01717v3/real-world.png

**Table 2: Quantitative comparison with state of the arts on real-world LR images.**
| Datasets | Metrics | BSRGAN | Real-ESRGAN | MM-RealSR | LDL | FeMaSR | StableSR-200 | ResShift-15 | PASD-20 | DiffBIR-50 | SeeSR-50 | AddSR-$1$ | AddSR-$4$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ICCV 2021 | ICCVW 2021 | ECCV 2022 | CVPR 2022 | MM 2022 | Arxiv 2023 | NeurIPS 2023 | Arxiv 2023 | Arxiv 2023 | CVPR 2024 | - | - |  |  |
| RealSR | MANIQA${}^{*}\uparrow$ | 0.3762 | 0.3727 | 0.3966 | 0.3417 | 0.3609 | 0.3656 | 0.3750 | 0.4041 | 0.4392 | 0.5396 | 0.4189 | 0.6597 |
| MUSIQ${}^{*}\uparrow$ | 63.28 | 60.36 | 62.94 | 58.04 | 59.06 | 61.11 | 56.06 | 62.92 | 64.04 | 69.82 | 63.56 | 72.25 |  |
| CLIPIQA${}^{*}\uparrow$ | 0.5116 | 0.4492 | 0.5281 | 0.4295 | 0.5408 | 0.5277 | 0.5421 | 0.5187 | 0.6491 | 0.6700 | 0.4929 | 0.7215 |  |
| PSNR$\uparrow$ | 26.49 | 25.78 | 23.69 | 25.09 | 25.17 | 25.63 | 26.34 | 26.67 | 25.06 | 25.24 | 24.22 | 22.73 |  |
| SSIM$\uparrow$ | 0.7667 | 0.7621 | 0.7470 | 0.7642 | 0.7359 | 0.7483 | 0.7352 | 0.7577 | 0.6664 | 0.7204 | 0.6863 | 0.6336 |  |
| DrealSR | MANIQA${}^{*}\uparrow$ | 0.3431 | 0.3428 | 0.3625 | 0.3237 | 0.3178 | 0.3222 | 0.3284 | 0.3874 | 0.4646 | 0.5125 | 0.3873 | 0.6034 |
| MUSIQ${}^{*}\uparrow$ | 57.17 | 54.27 | 56.71 | 52.38 | 53.70 | 52.28 | 50.14 | 55.33 | 60.40 | 65.08 | 57.42 | 68.16 |  |
| CLIPIQA${}^{*}\uparrow$ | 0.5094 | 0.4514 | 0.5171 | 0.4410 | 0.5639 | 0.5101 | 0.5287 | 0.5384 | 0.6397 | 0.6910 | 0.5543 | 0.7381 |  |
| PSNR$\uparrow$ | 28.68 | 28.57 | 26.84 | 27.41 | 26.83 | 29.14 | 28.27 | 29.06 | 26.56 | 28.09 | 27.49 | 26.09 |  |
| SSIM$\uparrow$ | 0.8022 | 0.8042 | 0.7959 | 0.8069 | 0.7545 | 0.8040 | 0.7542 | 0.7906 | 0.6436 | 0.7664 | 0.7588 | 0.7036 |  |
| RealLR200 | MANIQA${}^{*}\uparrow$ | 0.3688 | 0.3656 | 0.3879 | 0.3266 | 0.4099 | 0.3672 | 0.4182 | 0.4193 | 0.4626 | 0.4911 | 0.4215 | 0.6182 |
| MUSIQ${}^{*}\uparrow$ | 64.87 | 62.93 | 65.24 | 60.95 | 64.24 | 62.89 | 60.25 | 66.35 | 66.84 | 68.63 | 65.02 | 72.62 |  |
| CLIPIQA${}^{*}\uparrow$ | 0.5699 | 0.5423 | 0.6010 | 0.5088 | 0.6547 | 0.5916 | 0.6468 | 0.6203 | 0.6965 | 0.6617 | 0.5679 | 0.7724 |  |

### 4.3 Evaluation on Real-World Data

Tab. 2 shows the quantitative results on $3$ real-world datasets.
We can see that our AddSR achieves the best scores in MANIQA, MUSIQ and CLIPIQA, the same as in the synthetic degradation cases.
This demonstrates that AddSR has an excellent generalization ability to handle unknown complex degradations, making it practical in real-world scenarios.
Additionally, AddSR-$1$ surpasses the GAN-based methods, primarily due to the integration of diffusion model with adversarial training.
This integration enables AddSR to leverage high-level information to enhance the restoration process and generate high perception quality images, even through a one-step inference.

Figure: Figure 10: Illustrations of prompt-guided restoration that engages with manual prompts for more precise outcomes. In each group, the prompts for the second and third images are obtained through RAM and manual input, respectively. (a) ‘2ALC515’ is corrected as ‘24LC515’ by manual prompt. (b) The background near mushroom is modified to be blurry, aligning with GT.
Refer to caption: extracted/2404.01717v3/text_edit_2.png

Fig. 9 shows the visualization results.
We present the examples of building and face to comprehensively compare various methods.
A noticeable observation is that AddSR generate more clear and regular line, as evidenced by the linear pattern of the building in the first example. In the second example, the original LR image is heavily degraded, FeMaSR and ResShift fail to generate the human face, showing only the blurry outline of the face.
DiffBIR can generate more details, yet still unclear.
The image generated by SeeSR exhibits artifacts. Conversely, our AddSR can generate comparative results with FeMaSR and ResShift in one-step.
As evaluating the inference steps, AddSR generates more clear and detailed human face, which significantly surpasses the aforementioned methods.

#### Prompt-Guided Restoration.

One of the advantages of diffusion model is to integrate with text.
In Fig. 10, we demonstrate that our AddSR can efficiently achieve more precise restoration results in 4 steps by incorporating with manual prompt, i.e., we can manually input the text description of the LR image to assist the restoration process.
Specifically,
in Fig. 10(a), the word on the chip can be corrected from ‘2ALC515’ to ‘24LC515’ with the manual prompt.
In Fig. 10(b),
the mushroom’s background should appear blurry, while the RAM prompt renders the tree branches sharply.
Conversely, the manual prompt maintains the background’s intended blur, aligning with the Ground Truth.

#### Comparison with the Efficient SeeSR-Turbo.

A very recent efficient SD-based method named SeeSR-Turbo [] has been introduced for blind super-resolution through 2-steps inference.
To demonstrate the superiority of our AddSR, we conduct a visual comparison between SeeSR-Turbo and AddSR.
The qualitative results are shown in Fig. 11-Left.
One can see that our AddSR generates realistic textures by $2$ steps, while SeeSR-Turbo tends to generate blurry results.

### 4.4 Ablation Study

Effectiveness of Refined Training Process.
To enrich the information provided by the teacher model, we refine the training process by substituting the LR image with HR image as inputs of ControlNet, RAM, and CLIP.
Since SeeSR is adopted as the baseline, we also replace the LR image of its RAM input with HR image.
The quantitative results are shown in Tab. 3.
We can see that with the supervision from HR input, the perception quality of restored images becomes better.

Effectiveness of PSR.
As shown in Tab. 4, incorporating PSR improves perception quality notably at only $0.03$s slower than the original LR controlling strategy.
All of the three perception metrics, including MANIQA, MUSIQ and CLIPIQA, are improved on the two popular real-world datasets.

**Table 3: Ablation study on refined training process. The best results are marked in bold.**
| Exp | Condi<br>Image | RAM | RealLR200 | DrealSR |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MANIQA${}^{*}\uparrow$ | MUSIQ${}^{*}\uparrow$ | CLIPIQA${}^{*}\uparrow$ | MANIQA${}^{*}\uparrow$ | MUSIQ${}^{*}\uparrow$ | CLIPIQA${}^{*}\uparrow$ | PSNR$\uparrow$ |  |  |  |
| (1) | ✗ | ✗ | 0.5623 | 70.75 | 0.7431 | 0.5331 | 64.55 | 0.7285 | 26.96 |
| (2) | ✓ | ✗ | 0.6092 | 71.76 | 0.7660 | 0.5372 | 62.67 | 0.6997 | 26.78 |
| (3) | ✗ | ✓ | 0.5772 | 71.33 | 0.7549 | 0.5433 | 65.09 | 0.7087 | 26.87 |
| AddSR | ✓ | ✓ | 0.6182 | 72.62 | 0.7724 | 0.6034 | 68.16 | 0.7381 | 26.09 |

**Table 4: Ablation study on PSR and TA-ADD.**
| Methods | Time[s] | RealLR200 | DrealSR |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MANIQA${}^{*}\uparrow$ | MUSIQ${}^{*}\uparrow$ | CLIPIQA${}^{*}\uparrow$ | MANIQA${}^{*}\uparrow$ | MUSIQ${}^{*}\uparrow$ | CLIPIQA${}^{*}\uparrow$ | PSNR$\uparrow$ |  |  |
| w/o PSR | 0.44$\sim$0.77 | 0.5910 | 71.28 | 0.7541 | 0.5672 | 66.67 | 0.6589 | 25.85 |
| w/o TA-ADD | 0.44$\sim$0.80 | 0.6058 | 72.19 | 0.7630 | 0.5898 | 67.58 | 0.7042 | 25.77 |
| AddSR | 0.44$\sim$0.80 | 0.6182 | 72.62 | 0.7724 | 0.6034 | 68.16 | 0.7381 | 26.09 |

Figure: Figure 11: Left: Comparison against SeeSR-Turbo-2. Right: Visual comparison of TA-ADD.
Refer to caption: extracted/2404.01717v3/ta_loss_exp.png

Effectiveness of TA-ADD.
The proposed TA-ADD aims to balance the fidelity and perception quality of restored images.
The quantitative results are shown in Tab. 4.
Despite we increase the weight of the distillation loss in the later inference steps, the perceptual quality still improves.
This could be attributed to the initial steps producing sufficiently high perception quality images, which offer more informative cues when combined with PSR.
Consequently, the later inference steps can achieve both high perceptual quality and fidelity.

The visual results are shown in Fig. 11-Right.
For the upper 3 images, the content is a statue.
However, without TA-ADD, the model hallucinates its hand as a bird.
For the bottom 3 images, the original background is rock.
Again, without utilizing TA-ADD, AddSR might hallucinate the background as an eye of a wolf.
Conversely, with the help of TA-ADD, the restored images can generate more consistent contents with GTs.
TA-ADD constrains the model from excessively leveraging its generative capabilities, thereby preserving more information in the image content, aligning closely with the GTs.
Specifically, using TA-ADD, texture of the statue’s hand in the upper image remains unchanged, and the background of the bottom image retains the rock with out-of-focus appearance.

## 5 Conclusion

We propose AddSR, an effective and efficient model based on Stable Diffusion prior for blind super-resolution.
In contrast to current SD-based BSR approaches that employ LR images to regulate each inference step’s output, AddSR substitutes the LR image with HR image predicted in the preceding step to control the model output and mitigate the degradation impact. To further augment the model’s restoration capacity, we introduce timestep-adaptive ADD, which assigns distinct weights to GAN loss and distillation loss across different student timesteps, addressing the perception-distortion imbalance issue due to the introduction of original ADD.
Additionally, we use HR image as the teacher model controlling signal, enabling the teacher model to provide better supervision to the student model.
Extensive experiments demonstrate that AddSR can generate superior results within $1$$\sim$$4$ steps in various degradation scenarios and real-world low-quality images.

Limitations.
Although the inference speed of our AddSR surpasses all of the existing SD-based methods remarkably, there still exists a gap between AddSR and GAN-based methods.
The primary factor is that AddSR is built upon SD and ControlNet, which, due to its substantial model parameters and intricate network structure, noticeably hinders the inference time.
In the future, we plan to explore a more streamlined network architecture to boost overall efficiency.

## Appendix A Comparisons among ADD, SeeSR and AddSR

Figure: Figure 12: Comparisons on architecture diagram among ADD, SeeSR and AddSR. $x_{0}$ and y denote HR and LR images, respectively. $\hat{x}_{0}^{i}$ and $\hat{x}_{\phi}$ denote the predicted $x_{0}$ from the timesteps $s$ and $t$, respectively. $\epsilon_{s}$, $\epsilon_{t}$, $\hat{\epsilon}_{s}$ and $\hat{\epsilon}_{s}$ stand for added and predicted noise in timesteps $s$ and $t$, respectively. $\mathcal{L}_{ta-dis}$ is the timestep-adaptive distillation loss.
Refer to caption: extracted/2404.01717v3/Comparison.png

In this section, we provide a comparison among ADD, SeeSR, and our proposed AddSR. Their architecture diagrams are depicted in Fig. 12.
Firstly, the distinctions between ADD and AddSR primarily lie in two aspects:
$1$) Introduction of ControlNet: ADD is originally developed for text-to-image task, which typically only takes text as input.
In contrast, AddSR is an image-to-image model that requires the additional ControlNet to receive information from the LR image.
$2$) Perception-distortion Trade-off: ADD aims to generate photo-realistic images from texts.
However, introducing ADD into blind SR brings the perception-distortion imbalance issue (please refer to Sec. $3.4$ in our submission), which is addressed by our proposed timestep-adaptive ADD in AddSR.

Secondly, the key differences between SeeSR and AddSR are:
$1$) Introduction of Distillation: SeeSR is trained based on vanilla SD model that needs $50$ inference steps, while AddSR utilizes a teacher model to distill an efficient student model to achieve just $1$$\sim$$4$ steps.
$2$) High-frequency Information: SeeSR uses the LR image $y$ as the input of the ControlNet. In contrast, AddSR on one hand adopts the HR image $x_{0}$ as the input of the teacher model’s ControlNet to supply the high-frequency signals since the teacher model is not required during inference.
On the other hand, AddSR proposes a novel prediction-based self-refinement (PSR) to further provide high-frequency information by replacing the LR image with the predicted image as the input of the student model’s ControlNet.
Therefore, AddSR has the ability to generate results with more realistic details.

Figure: Figure 13: Our PSR is able to enhance the high-frequency signals of restored images to generate more photo-realistic details. The high frequency part is obtained using Fourier transform and filtering. Please zoom in for a better view.
Refer to caption: extracted/2404.01717v3/psr2.png

## Appendix B Effectiveness of Prediction-based Self-Refinement

Our PSR is proposed to remove the impact of LR degradation and enhance high-frequency signals to regulate the student model output.
As shown in Fig. 13, the restored images generated with PSR exhibit more details and sharper edges, while the images generated without PSR tend to be blurry with fewer details.

Figure: Figure 14: Visual comparisons between SeeSR-Turbo and AddSR. All the results are generate by $2$ steps. Please zoom in for a better view.
Refer to caption: extracted/2404.01717v3/more_examples2.png

**Table 5: Quantitative comparsion between SeeSR-Turbo and AddSR. All the results are generated by $2$ steps.**
| Methods | RealLR200 |  |  |  |
| --- | --- | --- | --- | --- |
| NIQE$\downarrow$ | MANIQA$\uparrow$ | MUSIQ$\uparrow$ | CLIPIQA$\uparrow$ |  |
| SeeSR-Turbo-2 | 7.87 | 0.3503 | 53.88 | 0.4634 |
| AddSR-2 | 5.08 | 0.6182 | 72.62 | 0.7724 |

## Appendix C Comparison of Timestep-Adaptive ADD Forms

To determine the optimal settings for timestep-adaptive ADD, we conduct experiments on its different forms: exponential and linear.
Specifically, the exponential form is defined as Eq. 2, while the linear form is defined as follows:

$$ $d(s,t)=(\prod_{i=0}^{t}(1-\beta_{t}))^{\frac{1}{2}}\times(\gamma\cdot p(s)+\kappa)$ (3) $$

where the hyper-parameter $\kappa$ sets the initial weighting ratio, while $\gamma$ controls the increase of distillation loss over student timesteps.
The quantitative results of the exponential and linear forms under various settings are listed in Tab. 6 and Tab. 7, respectively.
The best settings for different forms in the tables are highlighted with a gray background.
From these tables, we can draw the following conclusions:
(1) The best results of the exponential form are better than those of the linear form. Therefore, we use Eq. (2) as the distillation loss function. Moreover, when $\mu=0.5$ and $\nu=2.1$, we achieves the best perceptual quality while maintaining good fidelity, so we use this setting for Eq. (2). (2) Increasing the hyper-parameters that control the distillation loss ratio (i.e., $\nu$ and $\gamma$) typically results in higher fidelity. For instance, when we fix $\mu$ to 0.5 and increase $\nu$, the overall trend in 4 step shows a decrease in perception quality and an improvement in fidelity. Consequently, we can achieve a perception-distortion trade-off by adjusting $\nu$.

**Table 6: Comparing the exponential form of timestep-adaptive ADD across different hyper-parameters.**
| $\mu$ | $\nu$ | RealSR |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $1$ step | $2$ step | $3$ step | $4$ step |  |  |  |  |  |  |
| MANIQA$\uparrow$ | PSNR$\uparrow$ | MANIQA$\uparrow$ | PSNR$\uparrow$ | MANIQA$\uparrow$ | PSNR$\uparrow$ | MANIQA$\uparrow$ | PSNR$\uparrow$ |  |  |
| $0.5$ | 1.3 | 0.4202 | 24.11 | 0.6263 | 23.10 | 0.6427 | 22.36 | 0.6453 | 22.33 |
| 1.7 | 0.3986 | 24.18 | 0.6110 | 24.01 | 0.6195 | 23.44 | 0.6307 | 23.40 |  |
| 2.1 | 0.4189 | 24.22 | 0.6339 | 23.29 | 0.6496 | 22.76 | 0.6597 | 22.73 |  |
| 2.5 | 0.4207 | 24.48 | 0.5939 | 23.92 | 0.6081 | 23.33 | 0.6197 | 23.29 |  |
| 0.7 | $2.1$ | 0.3821 | 24.90 | 0.5971 | 24.03 | 0.6078 | 23.39 | 0.6221 | 23.36 |
| 0.9 | 0.4095 | 23.16 | 0.6052 | 22.97 | 0.6062 | 22.88 | 0.6244 | 22.68 |  |

**Table 7: Comparing the linear form of timestep-adaptive ADD across different hyper-parameters.**
| $\gamma$ | $\kappa$ | RealSR |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $1$ step | $2$ step | $3$ step | $4$ step |  |  |  |  |  |  |
| MANIQA$\uparrow$ | PSNR$\uparrow$ | MANIQA$\uparrow$ | PSNR$\uparrow$ | MANIQA$\uparrow$ | PSNR$\uparrow$ | MANIQA$\uparrow$ | PSNR$\uparrow$ |  |  |
| 0.1 | 0.7 | 0.3908 | 24.28 | 0.5849 | 23.64 | 0.6133 | 23.00 | 0.6172 | 22.98 |
| $0.2$ | 0.3 | 0.4574 | 23.69 | 0.6294 | 23.06 | 0.6465 | 22.48 | 0.6480 | 22.41 |
| 0.5 | 0.4338 | 23.87 | 0.6237 | 23.17 | 0.6407 | 22.55 | 0.6443 | 22.52 |  |
| 0.7 | 0.4225 | 24.02 | 0.6064 | 23.24 | 0.6313 | 22.60 | 0.6352 | 22.59 |  |
| $0.4$ | 0.1 | 0.4027 | 24.41 | 0.6077 | 23.25 | 0.6157 | 22.51 | 0.6215 | 22.45 |
| 0.3 | 0.4045 | 24.39 | 0.5981 | 23.30 | 0.6124 | 22.58 | 0.6202 | 22.51 |  |
| 0.5 | 0.4152 | 24.73 | 0.6261 | 23.33 | 0.6495 | 22.65 | 0.6507 | 22.62 |  |
| $0.6$ | 0.1 | 0.4156 | 24.48 | 0.6175 | 23.44 | 0.6261 | 22.82 | 0.6328 | 22.79 |
| 0.3 | 0.3926 | 24.90 | 0.5905 | 23.93 | 0.5857 | 23.51 | 0.5981 | 23.42 |  |
| 0.8 | 0.1 | 0.3887 | 24.92 | 0.5943 | 23.84 | 0.6038 | 23.36 | 0.6130 | 23.28 |

## Appendix D More Comparisons with SeeSR-Turbo

Here, we present more quantitative and qualitative results in Tab. 5 and Fig. 14, respectively.
Our AddSR surpasses SeeSR-Turbo with significant margins on NIQE, MANIQA, MUSIQ and CLIPIQA.

## Peer Reviews

confidence: 1
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. The texts in Figure 6 and Table 1 are too small. Please reformat them.

2. What type of GPU are the testing experiments done?

3. Could you provide the visual comparison results with SUPIR [1]?

4. Could you provide the inference times when comparing with SotAs in Table 1?

5. Please provide more visual comparisons with SotA methods as supplements.

6. How to use HR images as conditions when training prediction-based self-refinement? What are the differences between training and inference when using HR as a condition?

7. Since there is no HR input during inference, what is the significance of the *Effectiveness of Refined Training Process* in the ablation study?

8. Does the ratio of GAN loss to distillation loss vary with different scenarios? If yes, how to adjust the ratio in different scenarios? Do we need to change the ratio for each image?

9. Please describe the effects of using only GAN loss and only distillation loss.

[1] Gu Jinjin, et  al. Pipal: a
large-scale image quality assessment dataset for perceptual image restoration.
soundness: 3
strengths: 1. PSR does not use additional modules to pre-clean LR images. Instead, it uses HR images estimated by Equation (2).

2. AddSR’s teacher model substitutes $x_{LR}$ with $x_0$ to regulate the output, which could force the student model to learn the high-frequency information of HR images implicitly.

3. AddSR can produce images with better perceptual quality than most diffusion-based SR models, requiring fewer inference steps and less time.
summary: This paper presents AddSR, an efficient blind image super-resolution method based on Adversarial Diffusion Distillation (ADD). It introduces a timestep-adaptive adversarial diffusion distillation (TA-ADD) loss to address the perception-distortion imbalance inherent in ADD-based approaches. Additionally, it proposes a prediction-based self-refinement strategy to improve high-resolution (HR) estimation.
weaknesses: 1. The texts in Figure 6 and Table 1 are too small.

2. This paper does not provide the type of GPUs, and inference time when testing.

3. Lack of visual comparisons with SUPIR[1].

4. The visual comparisons are not thorough.


[1] Gu Jinjin, et  al. Pipal: a large-scale image quality assessment dataset for perceptual image restoration.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. This paper emphasizes that the proposed PSR can utilize the HR output to "control" the final output, but the experiment about how to control the intermediate results and influence the final output is missing.

2. PSR leverages the intermediate SR result as the condition for the next iteration. However, as shown in Figure 5(b), SD-based SR methods often have hallucination issues. Did you consider error accumulation in this scenario? Did you apply any additional processing to the intermediate output (e.g., blurring) to prevent error accumulation?

3. Line 230: "The $\hat{x_0}$ in each step has more high-frequency information to better control the model output." However, in Figure 5(b), the hallucinated animal head also contains high-frequency components. How do you prove that high-frequency components necessarily provide better guidance?

4. This work adopts SeeSR as the backbone and teacher model. Did you use manually added prompts when generating images, such as "clean," "high-resolution," or "8k" as used in SeeSR? Since these prompts greatly influence the perception-fidelity trade-off in SR output, their use may also affect your model's performance when intermediate results are regarded as crucial conditions. Could you compare the impact on performance with and without these prompts? 

5. Why does your method, which uses the same framework as SeeSR and reduces the number of steps from 50 to 4, only achieve a 7$\times$ speedup (which ideally should be 12.5$\times$)?

6. The entire training scheme appears to be overfitting to the 4-step setting, however, the ablation about the choice of this hyperparameter is missing. What will happen if you use the trained student model for 5 or more steps during inference?

7. In the design of the $L_{ta-dis}$ loss, why do you use the output of the student model as the input of the teacher model, then enforcing similarity between the teacher output and student output? This implies that the purpose of the $L_{ta-dis}$ is for the teacher to learn identity rather than to improve the student. However, since the teacher does not have any learnable layers, this design cannot derive such effect.
soundness: 2
strengths: 1. Some qualitative results demonstrate high-quality details.
2. The leading results in perceptual-oriented metrics.
summary: This work focuses on enhancing the efficiency of SD-based BSR methods by incorporating the Adversarial Diffusion Distillation (ADD) technique. Additionally, this study points out the perception-fidelity imbalance issue and the impact of image condition when applying ADD, proposing TA-ADD loss and prediction-based self-refinement (PSR) mechanism to address them, respectively.
weaknesses: 1. The main focus of this work is to emphasize acceleration through ADD, but most of the discussions revolve around addressing the limitations of ADD, then introducing certain tricks (i.e., loss function, improving image condition) to tackle the problems, without demonstrating noticeable effects in improving efficiency.

2. Another focus of this paper is to resolve the perception-distortion trade-off using the proposed TA-ADD loss. However, from Tables 1 and 3, it is evident that this framework fails to push the boundary of this trade-off. Instead, this framework achieves improved perceptual quality by heavily sacrificing fidelity, as indicated by low PSNR/SSIM scores.

3. The lack of comparison with other efficient SD-based methods (e.g., OSEDiff) makes it difficult to verify whether the effectiveness of this framework under similar efficiency.

4. On main focus of this work is to address perception-fidelity imbalance. In practice, SD-based SR methods can trade off between fidelity perceptual quality by employing inference tricks (e.g., manually added text prompts). However, this paper does not specify the inference setup of each baseline, nor does it achieve promising results with an acceptable level of fidelity drop.

5. In Table 1, the performance of this framework is not comparable with existing SD-based methods in several metrics. Specifically, while the proposed method shows better visual quality, it significantly underperforms in fidelity-oriented metrics such as PSNR and SSIM.

6. In Table 2, this framework doesn't deliver similar perceptual quality scores as SUPIR, such that comparing other settings at this point does not adequately demonstrate the advantages of this approach.

7. Confusing terminology. For example, in the caption of Figure 3, the term "predicted HR" image is used. Typically, "HR image" refers to ground truth. Since your method also uses the ground truth HR image as a condition for the teacher model, mixing the use of the term "HR" might lead to misunderstanding.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See the weekness part.
soundness: 3
strengths: 1. The task is meaningful and engaging.

2. Achieves good performance on certain non-reference metrics.

3. High efficiency compared to SeeSR.
summary: This paper presents AddSR, a blind super-resolution method using Timestep-Adaptive ADD (TA-ADD) to address challenges of perception-distortion imbalance and high-frequency detail restoration. AddSR achieves high-quality image restoration more efficiently, demonstrating a 7x speed advantage over existing models in experiments.
weaknesses: Performance Drop: There is a noticeable performance gap between the teacher and student models, with LPIPS dropping from 0.2124 to 0.2953—a significant decrease. From my experience, LPIPS is a more crucial metric than the non-reference metrics, where the proposed method shows improvement.

Notation Error: The notation is incorrect. In L271-272, the authors state, "'*' indicates that the metric is non-reference," and label LPIPS as non-reference, although it is actually reference-based.

Missing LPIPS and Comparison: LPIPS is absent from Table 3, and the fidelity performance appears much lower than existing SOTA models, such as ResShift, SeeSR, and even current single-step SR models. Comparisons with these methods are also missing.

Complexity and Missing Details: The proposed method seems somewhat complex, and certain details are unclear. For instance, the caption for Fig. 2 lacks essential information, details on PSR are not provided, and the input for Fig. 3 is unspecified.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: 1. Let the reviewer denote Figure 4's \hat{x}^{1}_{0}...\hat{x}^{4}_{0} to 4-1, 4-2, 4-3, and 4-4. In Figure 4, it appears that 4-1 is more similar to 4-4 than to 4-2. Since 4-1 is generated using 4-2 as a condition, one would expect it to be more similar to 4-2. What could explain this discrepancy?

2. Given that PSR can be applied to previous stable diffusion-based SR models without the need for retraining, could the authors present both quantitative and qualitative results showing the performance of PSR on other models?

3. Instead of using enhanced conditions in PSR, would it be possible to substitute all LR images with predicted SR images from simpler SR methods like RealESRGAN? If the richer conditional information from PSR is a crucial factor, then using a naively super-resolved image could be more beneficial during the initial timesteps. Here, instead of retraining the model with SR image conditions, it might suffice to replace the \hat{x}^{0}​ condition with outputs from RealESRGAN on a pretrained AddSR model.
soundness: 3
strengths: 1. The proposed methodologies appear straightforward yet powerful. In comparison to previous SR methods, AddSR demonstrates superior performance in both qualitative and quantitative analyses.
summary: Previous diffusion-based SR methods face challenges related to perception distortion imbalance and sensitivity to the quality of conditioning inputs. To address these issues, this paper introduces TA-ADD and a self-refinement strategy, respectively. Specifically, instead of using the same LR image as a condition across timesteps, the self-refinement strategy utilizes the predicted \hat{x_{0}}​ from previous steps, offering richer information for subsequent denoising. Additionally, to manage the imbalanced loss weights between adversarial and distillation losses at different timesteps, the authors propose a weighting function to adjust the loss weights accordingly.
weaknesses: 1. While the primary focus of the paper is on PSR and TA-ADD, there are significant concerns regarding its novelty. First, although several works utilize the output from previous steps as a condition for the next denoising process*, what distinguishes the proposed TA-ADD from *? Second, since ** also employs a weighting function to adaptively adjust the loss term weights, what advantages does Equation 3 offer compared to their approach? If possible, it would be helpful to include a comparison using their weighting functions.

*Andreas Lugmayr, RePaint: Inpainting using Denoising Diffusion Probabilistic Models

**Tianwei Yin, One-step Diffusion with Distribution Matching Distillation

**Axel Sauer, Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation

2. What advantages does AddSR-1 gain from self-refinement? Additionally, please provide a diverse comparison with other one-step diffusion methods* in Table 1 for a more comprehensive understanding.

*Y Wang, SinSR: Diffusion-Based Image Super-Resolution in a Single Step 

*R Wu, One-Step Effective Diffusion Network for Real-World Image Super-Resolution

*M Noroozi, You Only Need One Step: Fast Super-Resolution with Stable Diffusion via Scale Distillation

3. While this is not a weakness, the reviewer suggests that the authors align the abstracts in both the OpenReview submission and the paper. The abstract on OpenReview seems to be the earlier version before revisions were made.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: Please referring to the weaknesses above.
soundness: 3
strengths: - The paper introduces AddSR, a model that significantly advances blind super-resolution by employing stable diffusion, showcasing impressive generative capabilities for reconstructing high-resolution images from low-resolution inputs.
- Two innovative aspects of the paper are the prediction-based self-refinement strategy, which efficiently incorporates high-frequency details, and the adversarial diffusion distillation approach, which accelerates the model's inference speed while maintaining quality.
- The authors have meticulously designed experiments that thoroughly evaluate AddSR's performance across a variety of datasets, demonstrating the model's robustness and effectiveness in different degradation scenarios.
- The writing is clear and well-structured, making the complex technical details accessible and the methodology easy to follow, which is commendable.
summary: This paper introduces a model named AddSR, which addresses the challenge of blind super-resolution by leveraging the capabilities of stable diffusion. The authors propose prediction-based self-refinement and adversarial diffusion distillation methods to optimize the model, resulting in significantly improved efficiency and image quality. AddSR demonstrates superior performance on various degradation scenarios and real-world low-quality images, showcasing its effectiveness on tasks such as image restoration within a remarkably reduced number of inference steps.
weaknesses: - Could you point out the extract reasons about that directly applying ADD in the BSR task leads to reduced fidelity? The description in this paper is confusing, only through the observation of experimental results, lack of theoretical analysis.

- The latest diffusion-based super resolution methods[1,2,3,4] have accelerated the inference process into single step, but this paper only obtains the optimal results with 4-steps,  It seems that the method in this paper does not have advantages and practicality. could you compare with them?

- PSR sounds insteresting, but when the estimated HR images are rough and distorted in the bigger steps, PSR will transfer the distortion error to the next step, LR is necessary, How to reduce the error?

- Time-step aware weighting is usefully, but the functions d(s, t) is too empirical, the times-step in TAD-SR[2] shows more simpler and more effective, similar to diffusion time-step embedding.

[1] Wu, Rongyuan, et al. "One-Step Effective Diffusion Network for Real-World Image Super-Resolution." arXiv preprint arXiv:2406.08177 (2024).

[2] He, Xiao, et al. "One Step Diffusion-based Super-Resolution with Time-Aware Distillation." arXiv preprint arXiv:2408.07476 (2024).

[3] Noroozi, Mehdi, et al. "You Only Need One Step: Fast Super-Resolution with Stable Diffusion via Scale Distillation." arXiv preprint arXiv:2401.17258 (2024).

[4] Zhang, Aiping, et al. "Degradation-Guided One-Step Image Super-Resolution with Diffusion Priors." arXiv preprint arXiv:2409.17058 (2024).