SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Testing Datasets. We synthesize a testing dataset that contains 3000 images randomly selected from the validation set of ImageNet [50] based on the commonly-used degradation model, i.e.,  $\pmb{y} = (\pmb{x} * \pmb{k}) \downarrow + \pmb{n}$ , where  $k$  is the blurring kernel,  $n$  is the noise,  $\pmb{y}$  and  $\pmb{x}$  denote the LR image and HR image, respectively. To comprehensively evaluate the performance of ResShift, we consider more complicated types of blurring kernels, downsampling operators, and noise types. The detailed settings on them can be found in the supplementary material. It should be noted that we selected the HR images from ImageNet [50] instead of the prevailing datasets in SR such as Set5 [54], Set14 [55], and Urban100 [56]. The rationale behind this setting is rooted in the fact that these datasets only contain very few source images, which fails to thoroughly evaluate the performance of various methods under different degradation types. We name this dataset as ImageNet-Test for convenience.
Two real-world datasets are adopted to evaluate the efficacy of ResShift. The first is RealSR [57], containing 100 real images captured by Canon 5D3 and Nikon D810 cameras. Additionally, we collect another real-world dataset named RealSet65. It comprises 35 LR images widely used in recent literature [19, 58, 59, 60, 61]. The remaining 30 images were obtained from the internet by ourselves.
Compared Methods. We evaluate the effectiveness of ResShift in comparison to seven recent SR methods, namely ESRGAN [62], RealSR-JPEG [63], BSRGAN [18], RealESRGAN [19], SwinIR [20], DASR [21], and LDM [11]. Note that LDM is a diffusion-based method with 1,000 diffusion steps. For a fair comparison, we accelerate LDM to the same number of steps with ResShift using DDIM [16] and denote it as "LDM-A", where "A" indicates the number of inference steps. The hyper-parameter  $\eta$  in DDIM is set to be 1 as this value yields the most realistic recovered images.
Metrics. The performance of various methods was assessed using five metrics, including PSNR, SSIM [64], LPIPS [65], MUSIQ [66], and CLIPIQA [67]. It is worth noting that the latter two are non-reference metrics specifically designed to assess the realism of images. CLIPIQA, in particular, leverages the CLIP [68] model that is pre-trained on a massive dataset (i.e., Laion400M [69]) and thus demonstrates strong generalization ability. On the real-world datasets, we mainly rely on CLIPIQA and MUSIQ as evaluation metrics to compare the performance of different methods.
We analyze the performance of ResShift under different settings on the number of diffusion steps  $T$  and the hyper-parameters  $p$  in Eq. (10) and  $\kappa$  in Eq. (1).
Diffusion Steps  $T$  and Hyper-parameter  $p$ . The proposed transition distribution in Eq. (1) significantly reduces the diffusion steps  $T$  in the Markov chain. The hyper-parameter  $p$  allows for flexible control over the speed of residual shifting during the transition. Table 1 summarizes the performance of ResShift on ImageNet-Test under different configurations of  $T$  and  $p$ . We can see that both of  $T$  and  $p$  render a trade-off between the fidelity, measured by the reference metrics such as PSNR, SSIM, and LPIPS, and the realism, measured by the non-reference metrics, including CLIPIQA and MUSIQ, of the super-resolved results. Taking  $p$  as an example, when it increases, the reference metrics improve while the non-reference metrics deteriorate. Furthermore, the visual comparison in Fig. 4 shows that a large value of  $p$  will suppress the model's ability to hallucinate more image details and result in blurry outputs.
Hyper-parameter  $\kappa$ . Equation (2) reveals that  $\kappa$  dominates the noise strength in state  $x_{t}$ . We report the influence of  $\kappa$  to the performance of ResShift in Table 1. Combining with the visualization in
Table 3: Quantitative results of different methods on the dataset of ImageNet-Test. The best and second best results are highlighted in bold and underline.
Methods	Metrics
PSNR↑	SSIM↑	LPIPS↓	CLIPIA↑	MUSIQ↑
ESRGAN [62]	20.67	0.448	0.485	0.451	43.615
RealSR-JPEG [63]	23.11	0.591	0.326	0.537	46.981
BSRGAN [18]	24.42	0.659	0.259	0.581	54.697
SwinIR [20]	23.99	0.667	0.238	0.564	53.790
RealESRGAN [19]	24.04	0.665	0.254	0.523	52.538
DASR [21]	24.75	0.675	0.250	0.536	48.337
LDM-15 [11]	24.89	0.670	0.269	0.512	46.419
ResShift	25.01	0.677	0.231	0.592	53.660
(a) Zoomed LR
(c) RealESRGAN
Figure 5: Qualitative comparisons of different methods on two synthetic examples of the ImageNet-Test dataset. Please zoom in for a better view.
(d) SwinIR
(e) DASR
(f) LDM-15
(g) ResShift
Fig. 4, we can find that excessively large or small values of  $\kappa$  will smooth the recovered results, regardless of their favorable metrics of PSNR and SSIM. When  $\kappa$  is in the range of [1.0, 2.0], our method achieves the most realistic quality indicated by CLIPIQA and MUSIQ, which is more desirable in real applications. We thus set  $\kappa$  to be 2.0 in this work.
Efficiency Comparison. To improve inference efficiency, it is desirable to limit the number of diffusion steps  $T$ . However, this causes a decrease in the realism of the restored HR images. To compromise, the hyper-parameter  $p$  can be set to a relatively small value. Therefore, we set  $T = 15$  and  $p = 0.3$ , and yield our model named ResShift. Table 2 presents the efficiency and performance comparisons of ResShift to the state-of-the-art (SotA) approach LDM [11] and three other GAN-based methodologies on ImageNet-Test dataset. It is evident from the results that the proposed ResShift surpasses LDM [11] in terms of PSNR and LPIPS [65], and demonstrates a remarkable fourfold enhancement in computational efficiency when compared to LDM-100. Despite showing considerable potential in mitigating the efficiency bottleneck of the diffusion-based SR approaches, ResShift still lags behind current GAN-based methods in speed due to its iterative sampling mechanism.
Therefore, it remains imperative to explore further optimizations of the proposed method to address this limitation, which we leave in our future work.
Perception-Distortion Trade-off. There exists a well-known phenomenon called perception-distortion tradeoff [70] in the field of SR. In particular, the augmentation of the generative capability of a restoration model, such as elevating the sampling steps for a diffusion-based method or amplifying the weight of the adversarial loss for a GAN-based method, will result in a deterioration in fidelity preservation while concurrently enhancing the authenticity of restored images. That is mainly because the restoration model with powerful generation capability tends to hallucinate more high-frequency image structures, thereby deviating from the underlying ground truth. To facilitate a comprehensive comparison between our ResShift and current SotA diffusion-based method LDM, we plotted the perception-distortion curves of them in Fig. 7, wherein the perception and distortion are mea
Figure 7: Perception-distortion trade-off of ResShift and LDM. The vertical and horizontal axes represent the strength of the perception and distortion, measured by LPIPS and MSE, respectively.
Table 4: Quantitative results of different methods on two real-world datasets. The best and second best results are highlighted in bold and underline.
Methods	Datasets
RealSR	RealSet65
CLIPIA↑	MUSIQ↑	CLIPIA↑	MUSIQ↑
ESRGAN [62]	0.2362	29.048	0.3739	42.369
RealSR-JPEG [63]	0.3615	36.076	0.5282	50.539
BSRGAN [18]	0.5439	63.586	0.6163	65.582
SwinIR [20]	0.4654	59.636	0.5782	63.822
RealESRGAN [19]	0.4898	59.678	0.5995	63.220
DASR [21]	0.3629	45.825	0.4965	55.708
LDM-15 [11]	0.3836	49.317	0.4274	47.488
ResShift	0.5958	59.873	0.6537	61.330
sured by LPIPS and mean square-error (MSE), respectively. This plot reflects the perception quality and the reconstruction fidelity of ResShift and LDM across varying numbers of diffusion steps, i.e., 10, 15, 20, 30, 40, and 50. As can be observed, the perception-distortion curve of our ResShift consistently resides beneath that of the LDM, indicating its superior capacity in balancing perception and distortion.
4.3 Evaluation on Synthetic Data
We present a comparative analysis of the proposed method with recent SotA approaches on the ImageNet-Test dataset, as summarized in Table 3 and Fig. 5. Based on this evaluation, several significant conclusions can be drawn as follows: i) ResShift exhibits superior or at least comparable performance across all five metrics, affirming the effectiveness and superiority of the proposed method. ii) The notably higher PSNR ans SSIM values attained by ResShift indicate its capacity to better preserve fidelity to ground truth images. This advantage primarily arises from our well-designed diffusion model, which starts from a subtle disturbance of the LR image, rather than the conventional assumption of white Gaussian noise in LDM. iii) Considering the metrics of LPIPS and CLIPIQA, which gauge the perceptual quality and realism of the recovered image, ResShift also demonstrates evident superiority over existing methods. Furthermore, in terms of MUSIQ, our approach achieves comparable performance with recent SotA methods. In summary, the proposed ResShift exhibits remarkable capabilities in generating more realistic results while preserving fidelity. This is of paramount importance for the task of SR.
4.4 Evaluation on Real-World Data
Table 4 lists the comparative evaluation using CLIPIQA [67] and MUSIQ [66] of various methods on two real-world datasets. Note that CLIPIQA, benefiting from the powerful representative capability inherited from CLIP, performs stably and robustly in assessing the perceptional quality of natural images. The results in Table 4 show that the proposed ResShift evidently surpasses existing methods in CLIPIQA, meaning that the restored outputs of ResShift better align with human visual and perceptive systems. In the case of MUSIQ evaluation, ResShift achieves the competitive performance when compared to current SotA methods, namely BSRGAN [18], SwinIR [20], and RealESRGAN [19]. Collectively, our method shows promising capability in addressing the real-world SR problem.
We display four real-world examples in Fig. 6. More examples can be found in the supplementary material. We consider diverse scenarios, including comic, text, face, and natural images to ensure a comprehensive evaluation. A noticeable observation is that ResShift produces more naturalistic image structures, as evidenced by the patterns on the beam in the third example and the eyes of a person in the fourth example. We note that the recovered results of LDM are excessively smooth when compressing the inference steps to match with the proposed ResShift, specifically utilizing 15 steps, largely deviating from the training procedure's 1,000 steps. Even though other GAN-based methods may also succeed in hallucinating plausible structures to some extent, they are often accompanied by obvious artifacts.
In this work, we have introduced an efficient diffusion model named ResShift for SR. Unlike existing diffusion-based SR methods that require a large number of iterations to achieve satisfactory results,
(a) LR Image
(b)ESRGAN
(c) RealSR-JPEG-
(d) RealESRGAN
(e) BSRGAN
(f)DASR
(g) SwinIR
(h) LDM-100
(i) LDM-30
(k) ResShift-15
(c) RealSR-JPEG
(f) DASR
(a) LR Image
Figure 6: Qualitative comparisons on four real-world examples. Please zoom in for a better view.
our proposed method constructs a diffusion model with only 15 sampling steps, thereby significantly improving inference efficiency. The core idea is to corrupt the HR image toward the LR image instead of the Gaussian white noise, which can effectively cut off the length of the diffusion model. Extensive experiments on both synthetic and real-world datasets have demonstrated the superiority of our proposed method. We believe that our work will pave the way for the development of more efficient and effective diffusion models to address the SR problem.