ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting
Authors: Zongsheng Yue
Jianyi Wang
Chen Change Loy
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Super-resolution; Diffusion model; Efficient
TL;DR: This paper specifically designs an efficient diffusion model for image super-resolution that significantly reduces the sampling steps.

Abstract: Diffusion-based image super-resolution (SR) methods are mainly limited by the low inference speed due to the requirements of hundreds or even thousands of sampling steps. Existing acceleration sampling techniques inevitably sacrifice performance to some extent, leading to over-blurry SR results. To address this issue, we propose a novel and efficient diffusion model for SR that significantly reduces the number of diffusion steps, thereby eliminating the need for post-acceleration during inference and its associated performance deterioration. Our method constructs a Markov chain that transfers between the high-resolution image and the low-resolution image by shifting the residual between them, substantially improving the transition efficiency. Additionally, an elaborate noise schedule is developed to flexibly control the shifting speed and the noise strength during the diffusion process. Extensive experiments demonstrate that the proposed method obtains superior or at least comparable performance to current state-of-the-art methods on both synthetic and real-world datasets, \textit{\textbf{even only with 20 sampling steps}}. Our code and model will be made publicly.

ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting
Zongsheng Yue
Jianyi Wang Chen Change Loy S S-Lab, Nanyang Technological University
(zongsheng, yue, jan.jjannyooo ccloy] @ @ttu.edu.sg
Abstract
Diffusion-based image super-resolution (SR) methods mainly limited by the low inference speed due to the requirements of hundreds or even thousands of sampling steps. Existing acceleration sampling techniques inevitably sacrifice performance to some extent, leading to over-blurry SR results. To address this issue, we propose novel and efficient diffusion model for SR that significantly reduces the number of diffusion steps, thereby eliminating the need for post- aeseleccelertion during inference and its associated performance deterioration. Our method constructs a Markov chain that transfers between the high-resolution image and the low-resolution image by shifting the residual between them, substantially improving the transition efficiency. Additionally, an elaborate noise schedule is developed to flexibly control the shifting speed and the noise strength during the diffusion process. Extensive experiments demonstrate that the proposed method obtains superior or at least comparable performance to current state-of-the-art methods on both synthetic and real-world datasets, even only with 15 sampling steps. Our code and model are available https:/ // //gi thuth...... com/ 2otm///yOOAA//eshitttt ResShiff.
1 Introduction
Image super-resolution (SR) is fundamental problem in low-level vision, aiming at recovering the high-resolution (HR) image given the low-resolution (LR) one. This problem is severely ill-posed due to the complexity and unknown nature degradation models in real-world scenarios. Recently, diffusion model [1, 2]. newly emerged generative model, has achieved unprecedented success in image generation [3]. Furthermore, has also demonstrated great potential solving several downstream low-level vision tasks, including image editing [4, 5], image inpainting [6, 7], image colorization [8, 9]. There is also ongoing research exploring the potential of diffusion models to tackle the long-standing and challenging SR task.
One common approach [10, 11] involves inserting the LR image into the input current diffusion model (e.g., DDPM [2]) and retraining the model from scratch the training data for SR. Another popular way [7, 12, 13, 14] is to use an unconditional pre-trained diffusion model as prior and modify its reverse path to generate the expected HR image. Unfortunately, both strategies inherit the Markov chain underlying DDPM, which can inefficient in inference, often taking hundreds or thousands of sampling steps. Although some acceleration techniques [15, 16, 17] have been developed to compress the sampling steps in inference, they inevitably lead to a significant drop in performance, resulting in over-smooth results shown in Fig. 1, in which the DDIM [16] algorithm employed to speed up the inference. Thus, there is a need to design a new diffusion model for SR that achieves both efficiency and performance, without sacrificing one for the other.
Let us revisit the diffusion model the context image generation. the forward process, it builds up Markov chain to gradually transform the observed data into pre-specified prior distribution, typically a standard Gaussian distribution, over a large number of steps. Subsequently, image
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(b) BSRGAN
(c) DASR
(d) LDM-500
(c) LDM-50
() LDM-15
LR Image
(1) ReaESRGAN
(g) SwiniR
(b) LDM-100
() LDM-30
(K) Ours-15
Figure 1: Qualitative comparisons on one typical real-world example of the proposed method and
recent state of the arts, including BSRGAN [18], RealESRGAN [19]. SwinIR [20], DASR [21], and
LDM [11]. As for LDM and our method, we mark the number of sampling steps with the format of "LDM (or Ours)-A" for more intuitive visualization, where "A" is the number of sampling steps. Note that LDM contains 1000 diffusion steps training and is accelerated to "A" steps using DDIM [16] during inference. Please zoom in for a better view.
generation can be achieved by sampling noise map from the prior distribution and feeding it into the reverse path of the Markov chain. While the Gaussian prior is well-suited for the task of image generation, it may not be optimal for SR, where the LR image is available. In this paper, argue that the reasonable diffusion model for SR should start from a prior distribution based on the LR image, enabling an iterative recovery of the HR image from its LR counterpart instead of Gaussian white noise. Additionally, such a design can reduce the number of diffusion steps required for sampling, therehy improving inference efficiency.
Following the aforementioned motivation, we propose an efficient diffusion model involving a shorter Markov chain for transitioning between the HR image and its corresponding LR one. The initial state of the Markov chain converges to an approximate distribution of the HR image, while the final state converges to approximate distribution of the LR image. To achieve this, carefully design transition kernel that shifts the residual between them step by step. This approach more efficient than existing diffusion-basee based SR methods since the residual information can be quickly transferred in dozens of steps. Moreover, our design also allows for an analytical and concise expression for the evidence lower bound, easing the induction of the optimization objective for training. Based this constructed diffusion kernel, we further develop a highly flexible noise schedule that controls the shifting speed of the residual and the noise strength in each step. This schedule facilitates fidelity-realism trade-off of the recovered results by tuning its hyper-past parameters.
In summary, the main contributions of this work are follows:
We present an efficient diffusion model for SR, which renders an iterative sampling procedure from the LR image to the desirable HR one by shifting the residual between them during inference. Extensive experiments demonstrate the superiority of our approach terms of efficiency, as requires only 15 sampling steps achieve appealing results, outperforming comparable to current diffusion-baat -aaeed SR methods that require a long sampling process. preview of our recovered results compared with existing methods is shown in Fig. 1.
at least being
We formulate a highly flexible noise schedule for the proposed diffusion model, enabling more precise control of the shifting of residual and noise levels during the transition.
2 Methodology
In this section, we present a diffusion model, ResShifi, which is tailored for SR. For ease of presenta- tion, the LR and HR images are denoted as yo and æp, respectively. Furthermore, we assume yo and an have identical spatial resolution, which can be easily achieved through pre-upsampling the LR image yo using nearest neighbor interpolation if necessary.

---

Reverse 2
Ostioaal
YOGAN
VOGAN
HR Image
LR Image
T
Pnis
Residuel
Forward
Figure 2: Overview of the proposed method. It builds up a Markov chain between the HR/LR image pair by shifting their residual.
2.1 Model Design
The iterative generation paradigm of diffusion models has proven highly effective at capturing complex distributions, inspiring method constructs a Markov chain that serves as a bridge between the HR and LR images as shown in Fig. This way, the SR task can be accomplished by reverse sampling from this Markov chain given any LR image. Next, we will detail the process of building such a Markov chain specifically for SR.
approach the SR problem iteratively well. Our proposed
Forward Process. Let's denote the residual between the LR and HR images ep. i.e., Co yo Zo- Our core idea is to transit from Dy to Yo by gradually shifting their residual eg through a Markov chain with length T. A shifting sequence {n) 17 is first introduced, which monotonically increases with the timestep /. and satisfies - 0 and T/r + 1. The transition distribution is then formulated based on this shifting sequence follows:
q(2,,) 24 yo) Nar + ogep, oxep, aT, t= 1,2,
(1)
where O; = 7 for /. > and 0₃ = 7/, x a hyper-parameter controlling the noise variance, the identity matrix. Notably, we show that the marginal distribution at any timestep is analytically integrable, namely
( yo) . (xx + Tu, x² m)d,, t 1,2, ,T
The design of the transition distribution presented in Eq. (1) is based two primary principles. The first principle concerns the standard deviation, i.c., kvaa which aims to facilitate a smooth transition between æ, and æ;-1. This is because the expected distance between æ, and - can be bounded by vor, given that the image data falls within the range of [0, 1], i.e.,
max + m/) (æt + 1e)] max[(xee] < < Vai,
where max represents the pixel-wise maximizing operation. The hyper-parameter is introduced to increase the flexibility of this design. The second principle pertains to the mean parameter, i.e., -o,t-e which induces the marginal distribution Eq. (2). Furthermore, the marginal distributions of Φ: and Dr converges to 6, (-) and (- yo. x²2), which act as two approximate distributions for the HR image and the LR image, respectively. By constructing the Markov chain in such thoughtful way, it is possible to handle the SR task by inversely sampling from it given the LR image yo-
Reverse Process. The reverse process aims to estimate the posterior distribution p( (eol)) via the following formulation:
p( /
((xT1/8)) I pe -z (r,, Mo 1or,
where p(ør (3)) (x (oo x²2 I²21), Pe (w.- Dt, yo) is the inverse transition kernel from to ær-1 with learnable parameter Following most of the literature in diffusion model [1, 2, 8], we adopt the assumption of pe 1 22, yo) = (x 140 (0 yo. t), Ee(x,, yo. t)). The optimization for is achieved by minimizing the negative evidence lower bound, namely,
min Σ DxL. [g(0;-11 [2,, 2o yo) Mo)llp((<r /2,, }0]], 0
(5)
(.)()) denotes the Dirac distribution centered μa.

---

where Dx-[-Il-] denotes the Kullback-Leibeer (KL) divergence. More mathematical details can be found in Sohl-Dickstein al. [1] or Ho al. [2].
Combining Eq. (1) and Eq. (2), the targeted distribution g( 12,
[2, 2o. yo) in Eq. (5) can be rendered
can
tractable and expressed in an explicit form given below:
q( ( R4, mo yo) = N -
1/2
3i.
Ott
2o,
7
The detailed calculation of this derivation presented in the supplementary material, Considering that the variance parameter is independent of at and yo. we thus set Eø yo. A for the mean parameter p yo. f), it reparameterized follows:
7t-1 I. As
no (x Vo: 1) - Dt
at (1 yo: 4)
1/t
where fe is a deep neural network with parameter O. aiming to predict wo. We explored different parameterization forms on μ4 and found that Eq. (7) exhibits superior stability and performance. Based on Eq. (7), simplify the objective function in Eq. (5) as follows,
min Σ. we (o yo. () zz 20112.
(8)
where W/ = Or evident improvement in performance, which aligns with the conclusion in Ho et al. [2].
n In practice, we empirically find that the omission of weight w, results in an
Extension to Latent Space. To alleviate the computational overhead in training, we move the aforementioned model into the latent space of VQGAN [22], where the original image compressed by a factor of four in spatial dimensions. This does not require any modifications our model other than substituting Do and Yo with their latent codes. An intuitive illustration is shown in Fig. 2.
2.2
Noise Schedule
The proposed method employs hhper-parameter k and shifting sequence {ne )2 to determine the noise schedule in the diffusion process. Specifically, the hyper-parameter K regulates the overall noise intensity during the transition, and its impact on performance is empirically discussed Sec. 4.2. The subsequent exposition mainly revolves around the construction of the shifting sequence {7), )
Equation (2) implies that the noise level in state 2; is proportional to vTJ with a scaling factor This observation motivates us to focus on designing /IIt instead of Tj- Song and Ermon [23] show that / should be sufficiently small (e.g., 0.04 in LDM [11] to ensure that ( | yo) g(mr))- Combining with the additional constraint of - O, we set Th to be the minimum value between (x) and 0.001. For the final step T, we set 7/r as 0.999 ensuring 7T - 1. For the intermediate timesteps, i.c., E [2, - 1], we propose non-uniform geometric schedule for TTe as follows:
t t= ., TTe /m x b
(9)
where
log 7T
31 T (T 1), by = exp
(10)
2(T - 1) 11
Note that the choice of 3r and by is based on the assumption of 31 = 0, Br = T 1, and
VIT = / x bn - The hyper-parameter p controls the growth rate of T as shown in Fig. 3(h).
The proposed noise schedule exhibits high flexibility in three key aspects. First, for small values of perturbation around the LR image as depicted in Fig. 3(c)-(d).
k, the final state Φy converges
Compared to the corruption ended at Gaussian noise, this design considerably shortens the length of the Markov chain, thereby improving the inference efficiency. Second, the hyper-paramett p provides precise control over the shifting speed, enabling a fidelity-realism trade-off in the SR results as analyzed in Sec. 4.2. Third, by setting = 40 and p = 0.8, our method achieves a diffusion process remarkably similar to LDM [11]. This is clearly demonstrated by the visual results during the diffusion process presented Fig. 3(e)-(f), and further supported by the comparisons on the relative noise strength shown in Fig. 3(g).

---

Forward Process
Imagge
p=ll.s,
ResSiig (w-2.00 T-15)
(c) ReaShij (x-40, p-1.8, T-1000)
(1) Latent Diftision Model (T-1000)
0h
Figure 3: Illustration of the proposed noise schedule. (a) HR image. (b) Zoomed LR image. (c)-(d) Diffused images of ResShift in timesteps of 1, 3, 5, 7, 9, 12. and 15 under different values of by fixing p = 0.3 and T 15. (c)-(f) Diffused images of ResShift with a specified configuration of = 40, = 0.8, T = 1000 and LDM [11] timesteps of 100, 200, 400, 600, 800, 900, and 1000. (g) The relative noise intensity (vertical axes, measured by v1/>... where A denotes the signal-to-non to-noise ratio) of the schedules in (d) and (e) w.t.. the timesteps (horizontal axes). (h) The shifting speed /Te (vertical axes) w.r.t. to the timesteps (horizontal axes) across various configurations of p. Note that the diffusion processes in this figure implemented in the latent space, but display the intermediate results after decoding back to the image space for the purpose of easy visualization.
3 Related Work
Diffusion Model. Inspired by the non-equilibrium statistical physics, Sohl-Dickstein et al. [1] firstly proposed the diffusion model to fit complex distributions. Ho et al. [2] established novel connection between the diffusion model and the denoising scoring matching. Later, Song et al. [8] proposed unified framework to formulate the diffusion model from the perspective of the stochastic differential equation (SDE). Attributed its robust theoretical foundation, the diffusion model has achieved impressive success in the generation of images [3, 11], audio [24], graph [25] and shapes [26].
Image Super-Resolutton. Traditional image SR methods primarily focus on designing more rational image priors based on our subjective knowledge, such as non-local similarity [27], low-rankness [28], sparsity [29, 30]. and on. With the development of deep learning (DL). Dong al. [31] proposed the seminal work SRCNN to solve the SR task using a deep neural network. Then DL-based SR methods rapidly dominated the research field. Various SR technologies were explored from different perspectives, including network architecture [32, 33, 34, 35], image prior [36, 37, 38, 39], deep unfolding [40, 41, 42], degradation model [18, 19, 43, 44).
Recently, some works have investigated the application of diffusion models in SR. A prevalent approach is to concatenate the LR image with the noise in each step and retrain the diffusion model from scratch [10, 11, 45). Another popular way
utilize an unconditional pre-trained diffusion model as prior and incorporate additional constraints to guide the reverse process [7, 12, 13, 46). Both strategies often require hundreds or thousands of sampling steps to generate a realistic HR image. While several acceleration algorithms [15, 16, 17] have been proposed, they typically sacrifice the performance and result in blurry outputs. This work designs a more efficient diffusion model that overcomes this trade-off between efficiency and performance, as detailed in Sec. 2.
Remark. Several parallel works [47, 48, 49] also exploit such an iterative restoration paradigm in SR. Despite similar motivation, our work and others have adopted different mathematical formulations to achieve this goal. Delbracio and Milanfar [47] employed the Inversion by Direct Iteration (InDI) to model this process, while Luo et al. [48] and Liu et al. [49] attempted to formulate it as SDE. In this paper, we design discrete Markov chain to depict the transition between the HR and LR images, offering more intuitive and efficient solution to this problem.
5

---

Table 1: Performance comparison of ResShift on the ImageNet-Tedt under different configurations.
Configurations
Metrics
T
PSNR
SSIM
LPIPS
CLIPIQAT
MUSIQ1
10
25.20
0.6828
0.2517
0,5492
50.6617
15
25.01
0.6769
0.2312
0,5922
$3.6596
30
0.3
2.0
24.52
0.6585
0.2253
0,6273
55.7904
40
24.29
0.6513
0.2225
0.6468
56.8482
50
24.22
0.6483
0.2212
0.6489
56.8463
25.01
0.6769
0.2312
0,5922
53.6596
0.5
25.05
0,6745
0.2387
0,5816
52,4475
15
1.0
2.0
25.12
0.6780
0.2613
0,5314
48.4964
2.0
25.32
0.6827
0.3050
0.4601
43.3060
25.39
0.5813
0.3432
0.4041
38.5324
0.5
24.90
0.6709
0.2437
0,5700
50.6101
1.0
24.84
0.6699
0.2354
0,5914
52,9933
0.3
2.0
25.01
0,6769
0.2312
0,5922
53.6596
15
8.0
25.31
0.6858
0,2592
0.5231
49.3182
24.46
0.2772
0.4898
46,9794
LR Image
(10, 0.3, 2.0)
(15, 0.3, 2.0)
(30, 0.3, 2.0)
(40, 0.3, 2.0)
(50, 0.3, 2.0)
(15, 0.3, 2.0)
(15, 0.5, 2.0)
(15. 1.0, 2.0)
(15. 2.0, 2.0)
(15. 3.0, 2.0)
(15, 0.3, 0.5)
(.5, 1.0)
(15, 0.3, 2.0)
0.3,0.3, 8.0) (15, 0.3, 16.0)
Figure 4: Qualitative comparisons of ResShift under different combinations of (T. x). For example, "(15, 0.3, 2.0)" represents the recovered result with T = 15, p = 0.3, and 2.0. Please zoom in for better view.
4 Experiments
This section presents empirical analysis of the proposed ResShift and provides extensive exper- imental results to verify its effectiveness on one synthetic dataset and three real-world datasets. Following [18, 19], our investigation specifically focuses on the more challenging x4 SR task. Due to page limitation, some experimental results are put in the supplementary material.
4.1 Experimental Setup
Training Details. HR images with resolution of 256 x 256 in our training data are randomly cropped from the training set of ImageNet [50] following LDM [11]. We synthesize the LR images using the degradation pipeline of RealESRGAN [19]. The Adam [51] algorithm with default settings of PyTorc [52] and mini-batch size of 64 is used to train ResShift. During training, we use fixed learning rate of 5e-5 and update the weight parameters for 500K iterations. As for the
network architecture, employ the UNet structure in DDPM [2]. To increase the robustness of

---

Table 2: Efficiency and performance comparisons of ResShift to other methods on the dataset of
ImageNet- "LDM-A" represents the results achieved by accelerated the sampling steps of LDM [11] to "A" Running time is tested on NVIDIA Tesla V100 GPU on the x4 (64 256) SR task.
Methods
Metrics
BSRGAN
RealESRGAN
SwinIR
LDM-15
LDM-30
LDM-100
ResShijt
PSNR* LPIPS
24.42
24.04
23.99
24.89
24.49
23,90
25.01
0.259
0.254
0.238
0.269
0.248
0.244
0.231
CLIPIQAt
0.581
0.523
0.564
0.512
0.572
0.620
0.592
Runtime (s)
0.012
0.013
0,046
0.102
0.184
0.413
0.105
# Parameters (M)
16.70
16.70
28.01
113.60
118.59
ResShift arbitrary image resolution, we replace the self-attention layer in UNet with the Swin Transformer [53] block.
Testing Datasets. We synthesize testing dataset that contains 3000 images randomly selected from the validation set of ImageNet [50] based on the commonly used degradation model, i.e., y= (x k) + +n. where k is the blurring kernel, is the noise, y and æ denote the LR image and HR image, respectively. To comprehensively evaluate the performance of ResShift, we consider more complicated types of blurring kernels, downsampling operators, and noise types. The detailed settings on them can be found the supplementary material. It should be noted that we selected the HR images from ImageNet [50] instead of the prevailing datasets in SR such as Set5 [54], Set14 [55], and Urban100 [56]. The rationale behind this setting is rooted in the fact that these datasets only contain very few source images, which fails thoroughly evaluate the performance of various methods under different degradation types. We name this dataset as ImageNet-Tee -est for convenience.
Two real-world datasets are adopted to evaluate the efficacy of ResShift. The first is RealSR [57], containing 100 real images captured by Canon 5D3 and Nikon D810 cameras. Additionally, we collect another real-world dataset named RealSe165. It comprises 35 LR images widely used in recent literature [19, 58, 59, 60, 61]. The remaining 30 images were obtained from the internet by ourselves.
Compared Methods. We evaluate the effectiveness of ResShift in comparison to seven recent SR methods, namely ESRGAN [62]. ReaISR-JPEG [63], BSRGAN [18], RealESRGAN [19], SwinlR [20], DASR [21], and LDM [11]. Note that LDM is a diffusion-based method with 1,000 diffusion steps. For a fair comparison, we accelerate LDM to the same number steps with ResShift
using DDIM [16] and denote it as "LDM-A", where "A" indicates the number of inference steps. The hyper-parameter 77 in DDIM is set to be 1 as this value yields the most realistic recovered images.
Metrics. The performance of various methods was assessed using five metrics, including PSNR. SSIM [64], LPIPS [65], MUSIQ [66], and CLIPIQA [67]. is worth noting that the latter two are non-reference metrics specifically designed to assess the realism of images. CLIPIQA, in particular, leverages the CLIP [68] model that is pre-trained on a massive dataset (i.e., Laion400M [69]) and thus demonstrates strong generalization ability. On the real-world datasets, we mainly rely CLIPIQA and MUSIQ as evaluation metrics to compare the performance of different methods.
4.2 Model Analysis
We analyze the performance of ResShifft under different settings on the number of diffusion steps T and the dlyer-paramettrv p in Eq. (10) and in Eq. (1).
Diffusion Steps T and Hyper-parameter p. The proposed transition distribution in Eq. (1) signifi- cantly reduces the diffusion steps 7 in the Markov chain. The hyper-parameter allows for flexible control over the speed of residual shifting during the transition. Table summarizes the performance of ResShift ImageNet-Tor Test under different configurations of T and p. We can see that both of T and p render a trade-off between the fidelity, measured by the reference metrics such as PSNR, SSIM, and LPIPS, and the realism, measured by the non-reference metrics, including CLIPIQA and MUSIQ, of the super-resolved results. Taking p as an example, when it increases, the reference metrics improve while the non-reference metries deteriorate. Furthermore, the visual comparison in Fig. 4 shows that a large value of p will suppress the model's ability to hallucinate more image details and result in blurry outputs.
Hyper-parameter k. Equation (2) reveals that e dominates the noise strength in state 24 We report the influence of k to the performance of ResShift in Table 1. Combining with the visualization in
7

---

Table 3: Quantitative results of different methods on the dataset of ImageNet- Test. The best and second best results are highlighted in bold and underline.
Metries LPIPS,
Methods
PSNR
SSIM
CLIPIQAT
MUSIQT
ESRGAN [62]
20.67
0,448
0,485
0,451
43.615
RealSR-IPEG [63]
23.11
0.591
0.326
0.537
46.981
BSRGAN [18]
24.42
0.659
0.259
0.581
54.697
SwinIR [20]
23,99
0.667
0.238
0,564
53.790
RealESRGAN [19]
24.04
0,665
0,254
0.523
52.538
DASR [21]
24.75
0.675
0,250
0.536
48.337
LDM-15 [1]]
24.89
0.670
0.269
0.512
46.419
ResShift
25.01
0.677
0.231
0.592
53.660
(6) Zoomed LR Figure 5: Qualitative comparisons of different methods on two synthetic examples of the ImageNet- Test dataset. Please zoom in for better view.
(b) BSRGAN
(c) ReaLESRGAN
(4) Swind
(e) DASR
(8) LDM-11
(6) RodSigo
Ohi Ground Truth
Fig. 4, we can find that excessively large or small values of x will smooth the recovered results, regardless of their favorable metrics of PSNR and SSIM. When & is in the range of [1.0, 2.0], our method achieves the most realistic quality indicated by CLIPIQA and MUSIQ, which is more be 2.0 in this work.
desirable in real applications. We thus
Efficiency Comparison. To improve inference efficiency, diffusion steps T. However, this causes decrease in the realism of the restored HR images. To compromise, the and p 0.3, and yield model named ResShift. Table 2 presents the efficiency and performance comparisons ResShift to the
desirable to limit the number of
can be set to relatively small value. Therefore, we set = 15
(SotA) approach LDM [11] and three other GAN-based methodologies ImageNet-Tess dataset. It is evident from the results that the proposed ResShift surpasses LDM [11] in terms of PSNR and LPIPS [65], and demonstrates a remarkable fourfold enhancement in computational efficiency when compared LDM-100. Despite showing considerable potential in mitigating the efficiency bottleneck of the diffusion-based SR approaches, ResShifi still lags behind current GAN-based methods speed due iterative sampling mechanism. Therefore, it remains imperative to explore further opti- mizations of the proposed method to address this limita- tion, which we leave in our future work.
10
Rackkn
Perception-Distortioo Trade-off. There exists well- known phenomenon called perception-distonn distoootonn trade- off [70] in the field of SR. In particular, the augmentation of the generative capability of restoration model, such elevating the sampling steps for diffusion-based method or amplifying the weight of the adversarial loss for a GAN- method, will result in deterioration in fidelity preservation while concurrently enhancing the authenticity of restored images. That is mainly because the restoration model with powerful generation capability tends hallucinate more high-frequent image struc- tures, thereby deviating from the underlying ground truth. To facilitate comprehensive comparison between our ResShift and current SotA diffusion-basse method LDM, we plotted the perception-distortee curves of them in Fig. 7, wherein the perception and distortion are mea-
+ LDN
20
10
30
50
15
30
40
Dianstoe
Figure 7: Perception-distortion trade-off of ResShift and LDM. The vertical and horizontal axes represent the strength of the perception and distortion, measured by LPIPS and MSE, respectively.

---

Table 4: Quantitative results of different methods on two real-world datasets. The best and second best results are highlighted in bold and underline.
Datasels
Methods
RealSR
RealSer65
CLIPIQAT
MUSIQU
CLIPIQAT
MUSIQ1
ESRGAN [62]
0.2362
29.048
0.3739
42.369
RealSR-JPEG [63]
0,3615
36,076
0,5282
50.539
BSRGAN [18]
0.5439
63,586
0.6163
65,582
SwinIR [20]
0,4654
59.636
0.5782
63.822
RealESRGAN [19]
0.4898
59.678
0.5995
63.220
DASR [21]
0.3629
45,825
0,4965
55.708
IDM-15 [11]
0.3836
49.317
0.4274
47.488
RexShift
0,5958
59.873
0.6537
61.330
sured by LPIPS and mean square-error (MSE), respectively. This plot reflects the perception quality and the reconstruction fidelity of ResShift and LDM across varying numbers of diffusion steps, i.e., 10, 15, 20, 30, 40, and 50. As can be observed, the perception-distortion curve of our ResShift consistently resides beneath that of the LDM, indicating its superior capacity in balancing perception and distortion.
4.3 Evaluation on Synthetic Data
We present comparative analysis of the proposed method with recent SotA approaches the ImageNet-Test dataset, as summarized in Table and Fig. 5. Based on this evaluation, several significant conclusions can be drawn as follows: ResShift exhibits superior performance across all five metrics, affirming the effectiveness and superiority of the proposed method. ii) The notably higher PSNR ans SSIM values attained by ResShift indicate its capacity to better preserve fidelity to ground truth images. This advantage primarily arises from our well-designed diffusion model, which starts from subtle disturbance of the LR image, rather than the conventional assumption of white Gaussian noise in LDM. iii) Considering the metrics of LPIPS and CLIPIQA, which gauge the perceptual quality and realism of the recovered image, ResShift also demonstrates evident superiority over existing methods. Furthermore, in terms of MUSIQ, our approach achieves comparable performance with recent SotA methods. In summary, the proposed ResShift exhibits remarkable capabilities in generating more realistic results while preserving fidelity. This is of paramount importance for the task of SR.
least comparable
4.4 Evaluation on Real-World Data
Table 4 lists the comparative evaluation using CLIPIQA [67] and MUSIQ [66] of various methods on two real-world datasets. Note that CLIPIQA, benefiting from the powerful representative capability inherited from CLIP. performs stably and robustly in assessing the perceptional quality of natural images. The results in Table 4 show that the proposed ResShift evidently surpasses existing methods in CLIPIQA, meaning that the restored outputs of ResShifi better align with human visual and perceptive systems. In the case of MUSIQ evaluation, ResShift achieves the competitive performance when compared to current SotA methods, namely BSRGAN [18], SwinIR [20], and RealESRGAN [19]. Collectively, our method shows promising capability addressing the real-world SR problem.
We display four real-world examples in Fig. 6. More examples can be found in the supplementary material. We consider diverse scenarios, including comic, text, face, and natural images ensure comprehensive evaluation. A noticeable observation is that ResShift produces more naturalistic image structures, as evidenced by the patterns on the beam in the third example and the eyes of a person in the fourth example. We note that the recovered results of LDM are excessively smooth when compressing the inference steps match with the proposed ResShift, specifically utilizing 15 steps, largely deviating from the training procedure's 1,000 steps. Even though other GAN- methods may also succeed in hallucinating plausible structures to some extent, they are often accompanied by obvious artifacts.
5 Conclusion
In this work, we have introduced an efficient diffusion model named ResShift for SR. Unlike existing diffusion-based SR methods that require a large number of iterations to achieve satisfactory results,

---

(b) ESRGAN
(c) RealSR-JPEG-
(d) ReaEESRAAN
(c) BSRGAN
(D DASR
LR Image
(g) SwinlR
(h)
④ LDM-30
① LDM-15
ResShiff-15
HUJI
J144
JJC L
J
BJG U
ESRGAN
(c) Rea- (c) ReaIESRGAN
(e) BSRGAN
(1 DASR
I J14 8
JUL
BJD4
BJ44
B
L.R Image
(g)
LDM-100
LDM-15
ReeSSiff-15
(b) ESRGAN
(c) RealSR-IPEG
(d) RealESRGAN
(e) BSRGAN
(1) DASR
(4) LR Image
(g) SwinIR
(h) LDM-100
(i LDM-30
LDM-15
(k) ResShi/f-15
ESRGAN
(e) RealSR-JPEG-
(d) RealESRGAN
(e) BSRGAN
(i DASR
(a) LR Image
(g) SwinIR
(h) LDM-100
① LDM-30
LDM-15
(k) ResSiuf-11
Figure 6: Qualitative comparisons four real-world examples. Please zoom in for better view.
proposed method constructs diffusion model with only 15 sampling steps, thereby significantly improving inference efficiency. The core idea is to corrupt the HR image toward the LR image instead of the Gaussian white noise, which can effectively cut off the length of the diffusion model. Extensive experiments on both synthetic and real- world datasets have demonstrated the superiority of our proposed method. We believe that work will pave the way for the development of more efficient and effective diffusion models to address the SR problem.
This study is supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry ps)tner(s).
Acknowledgement.
10

---

References
[1] Jascha Sohl-Dickstein, Fric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. International Conference on Machine Learning (ICML), pages 2256-2265. PMLR. 2015.
[2] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In Proceedings of Advances in Neural Information Processing Systems (NeurlPS), volume 33, pages 6840-6851, 2020.
[3] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Proceedings of Advances in Neural Information Processing Systems (NewrIPS), volume 34, pages 3780-8794, 2021.
[4] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In Proceedings of International Conference on Learning Representations (ICLR), 2021.
[5] Omri Avrahami, Dani Lischinski. and Ohad Fried. Blended diffusion for text driven editing of natural images. In Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18208-18218, 2022.
[6] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11461-11471 June 2022.
[7] Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Come--loser-liffuue--stee Accelerating conditional diffusson models for inverse problems through stochastic contraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages
2022.
[8] Yang Song, Jascha Sohl- Dickkiick,,,,, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poolc. Score-based generative modeling through stochastic differential equations. In Proceedings of International Conference on Learning Representations (ICLR), 2021.
[9] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: amage-to-imape diffusion models. In Proceedings of ACM SIGGRAPH Conference, pages 1-10, 2022.
[10] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022.
[11] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, 2022.
[12] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjunc Gwon, and Sungroh Yoon. 1lvr: Conditioning method for denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 14367-14376, 2021.
[13] Zongsheng Yue and Chen Change Loy. Difface: Blind face restoration with diffused error contraction. arXiv preprint arXiv:2212.06512 2022.
[14] Jianyi Wang, Zongsheng Yue, Shangehen Zhou, Kelvin C.K. Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution. arXiv preprint arXiv:2305.07015. 2023.
[15] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning (ICML), pages 8162-88 8171. PMLR, 2021.
[16] Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. In Proceedings of International Conference on Learning Representations (ICLR), 2021.
[17] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. Proceedings of Advances in Neural Information Processing Systems (NeurIPS). 2022.
[18] Kai Zhang, Jingyun Liang, Luc Van Gool, and Radu Timofte. Designing a practical degradation model for deep blind image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages
2021.
11

---

[19] Xintao Wang. Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super- resolution with pure synthetic data. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCV-W), pages 1905-1914, 2021.
[20] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang. Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCV-W), pages 1833 1844, 2021.
[21] Jie Liang, Hui Zeng, and Lei Zhang. Efficient and degradatioonadaapive network for real-world image super-resolution. In Proceedings of the European Conference on Computer Vision (ECCV), pages 574-591, 2022.
[22] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 12873-12883, 2021.
[23] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
[24] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi. and William Chan. WaveGrad: estimating gradients for waveform generation. In Proceedings of International Conference on Learning Representations (ICLR), 2020.
[25] Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permutation invariant graph generation via score-based generative modeling. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), volume 108, pages 4484 2020.
[26] Ruojin Cai, Guandao Yang, Hadar Averbuch-Flon Floor Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 364-381, 2020.
[27] Weisheng Dong, Lei Zhang, Guangming Shi, and Xin Li. Nonlocally centralized sparse representation for image restoration. IEEE Transactions on Image Processing (TIP), 22(4); 1620-160,, 2012.
[28] Shuhang Gu, Qi Xie, Deyu Meng, Wangmeng Zuo, Xiangchu Feng, and Lei Zhang. Weighted nuclear norm minimization and its applications to low level vision. International Journal of Computer Vision (ICC)), 121:183-2088 2017.
[29] Weisheng Dong, Lci Zhang, Guangming Shi, and Xiaolin Wu. Image deblurring and super-resolution by adaptive spanse domain selection and adaptive regularization. IEEE Transactions on Image Processing (TTP), 20(7): (338))11881857 2011.
[30] Shuhang Gu, Wangmeng Zuo, Qi Xie, Deyu Meng, Xiangehu Feng, and Lei Zhang. Convolutional sparse coding for image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1823-1831, 2015.
[31] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE Transactions on Puttern Analysis and Machine Intelligence (TPAMI). 38(2): 295-307, 2015.
[32] Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and vidco super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1874-1883, 2016.
[33] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing (TIP), 26(T): 3142-3155, 2017.
[34] Wei-Sheng Lai, Jia-Bin Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings the IEEE/CVF Conference Computer Vision and Pottern Recognition (CVPR), pages 624 632, 2017.
[35] Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. Deep back- projeerooection networks for super- rupoutresolutoon. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages
2018
12

---

[36] Jingyun Liang, Kai Zhang, Shuhang Gu, Luc Van Gool, and Radu Timofte. Flow-based kernel prior with application to blind super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10601-10610, 2021.
[37] Kelvin CK Chan, Xintao Wang. Xiangyu Xu, Jinwei Gu, and Chen Change Loy. GLEANN Generative latent. bank for large-factor image super-resolution. In Proceedings of the IEER/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages (4245-1425a) 2021.
[38] Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation. JEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 44(1 1))7777444) 2021.
[39] Zongsheng Yue. Qian Zhao, Jianwen Xie, Lei Zhang, Deyu Meng, and Kwan- Yee K. Wong. Blind image super-resolution with elaborate degradation modeling noise and kemel. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2128- 2138,, 2022.
[40] Kai Zhang. Wangmeng Zuo, and Lei Zhang. Deep plug-and-play super-resolution for arbitrary blur kemels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1671-1681 2019.
[41] Kai Zhang, Luc Van Gool, and Radu Timofte. Deep unfolding network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3217-3226, 2020.
[42] Jiahong Fu, Hong Wang, Qi Xie, Qian Zhao, Deyu Meng, and Zongben Xu. Kxnet: A model-driven deep neural network for blind super-resolution. In Proceedings of the Europeon Conference on Computer Vision (ECCV), pages 235-253, 2022.
[43] Kai Zhang. Wangmeng Zuo, and Lei Zhang, Learning single convolutional super-resolution network multiple degradations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3262 120122 2018.
[44] Chong Mou, Yanze Wu, Xintao Wang, Chao Dong. Jian Zhang, and Ying Shan. Metric learning based interactive modulation for real-world super-resolution, In Proceedings of the European Conference on Computer Vision (ECCV), pages 723-740, 2022.
[45] Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing. 479:47-59, 2022.
[46] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022.
[47] Mauricio Delbracio and Peyman Milanfar. Inversion by direct iteration: An alternative denoising diffuusinn for image restoration. arXiv preprint arXiv:2303. 11435, 2023.
[48] Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas Schön. Image restoration with mean-reverting stochastic differential equations. arXiv preprint arXiv:2301. 1699, 2023.
[49] Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos A Theodorou, Weili Nie, and Anima Anandku- mar. FSB: tongg-e--mmme schrodinger bridge. In International Conference on Machine Learning (ICML), 2023.
[50] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 248 255, 2009.
[51] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of International Conference on Learning Representations (ICLR), 2015.
[52] Adam Paszke, Sam Gross, Francisco Massa. Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
[53] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 10012- 2021.
13

---

[54] Marco Bevilacqua. Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. 2012.
[55] Roman Zeyde, Michael Flad, and Matan Protter. On single image scale-up using sparse-reprssentation.. In International Conference Curves and Surfaces, pages 711-730 Springer, 2012.
[56] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 5197- 5206, 2015.
[57] Jianrui Cai, Hui Zeng, Hongwei Yong, Zisheng Cao, and Lei Zhang. Toward real-world single image super-resolution: A new benchmark and new model. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3086-3095, 2019.
[58] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), volume 2. pages 416-423, 2001.
[59] Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using mangal09 dataset. Multimedia Tools and Applications, 76: 21811 22118,, 2017.
[60] Andrey Ignatov, Nikolay Kobyshev, Radu Timofte, Kenneth Vanhocy, and Luc Van Gool. Dslr-quality photos on mobile devices with deep convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2017.
[61] Kai Zhang. Wangmeng Zuo, and Zhang. Fidnet: Toward a fast and flexible solution for enn-based image denoising. IEEE Transactions on Image Processing (TIP), 27(9) 4622, 2018.
[62] Xintao Wang. Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European Conference on Computer Vision Workshops (ECCV-)), 2018.
[63] Xiaozhong Ji. Yun Cao, Ying Tai, Chengjic Wang. Jilin Li, and Feiyue Huang. Real-world super-resolution via keree estimation and noise injection. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (CVPR-V) WVPRRWW,.O pages 466-467. 2020.
[64] Zhou Wang, A.C. Bovik, H.R. Sheikh, and F.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEFF Transactions on Image Processing (TIP), 13(4):600-612, 2004.
[65] Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 586-595, 2018
[66] Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5148-5157, 2021.
[67] Jianyi Wang, Kelvin CK Chan, and Chen Change Lay. Exploring clip for assessing the look and feel of images. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
[68] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pages 8748-8763, 2021.
[69] Christoph Schuhmann, Richard Vencu, Romain Bcaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. In Proceedings Advances in Neural Information Processing Systems Workshops (NeurlPS-WV W), 2021.
[70] Yochai Blau and Tomer Michaeli. The perception -iipttiionnittttioinnoooo tradeoff. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2018
14


Reviews:
[{"review_id": "jyaZn3CMZ4", "reviewer": "Reviewer_vFLR", "summary": "This paper introduces a diffusion-based image super-resolution method.\nThe proposed method starts to generate an HR image from a given LR image directly (learns to generate the residual image between LR and HR), not start from a noise.\nTherefore, the method can generate the output image faster than previous works, specifically, with only 20 sampling steps.\nIn addition, there are some hyperparameters to control the fidelity-perceptual quality trade-off of output images, namely, T, p, and k. ", "questions": "- In L87, the image value range of [0,1] but I think the residual image is within [-1,1]. Does the following explanation still hold if I am correct?\n- In L135, how this is achieved? I think it is weird because the method does not learn to generate a noise distribution, and what does this description mean?\n- It would be better if runtimes/parameters are added for other methods in Table 3 (especially vs GAN-based methods).", "limitations": "- Please see the weaknesses and limitations.\n- My rating is a reflection of the weaknesses and I look forward to the authors' feedback on the weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of starting from LR image and utilizing a residual image is proper to the SR task and this is different from conditioning the LR image.\n- This successfully makes the method takes only short steps in the inference using a comparable number of parameters and runtime compared to previous method LDM.\n- In addition, it is practical for users to control the fidelity-perceptual quality trade-off of the SR model by adjusting the hyper-parameters. \n- The paper shows the superiority in terms of perceptual quality (LPIPS, CLIPIQA, MUSIQ) in synthetically degraded images and real images as well.", "weaknesses": "- There is no visual comparison on synthetic SR testsets so it is hard to fully understand the effectiveness of the method, but the paper argues superiority in this aspect.\n- Several previous methods are listed in Sec 3 (Related Work), however, only LDM is compared in experiments. Are there any difficulties \nor problems in comparing with other methods?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">  **Q1. Visual comparison on the synthetic dataset.**\n\nAs suggested, we have presented one visual comparison on the synthetic dataset in Fig. 2 of the associated rebuttal file. Evidently, the proposed method outperforms other competing approaches in terms of both fidelity and realism. In our revised version, we will add more qualitative comparison examples to address this concern.\n\n> **Q2. Performance comparison with other related methods.**\n\nPlease see Q1 of the global response.\n\n> **Q3. I think the residual image is within [-1,1]. Does the following explanation still hold if I am correct?**\n\nYes. The residual image $e_0$ is within [-1,1], and the following explanation still holds on. The shifting sequence $\\lbrace\\eta_t\\rbrace_{t=1}^{T}$ increases monotonically, indicating $\\alpha_t = \\eta_t -\\eta_{t-1} > 0 $ and $\\alpha_t<1$, we thus have\n\n$$\\text{max}[\\alpha_t e_0] < \\alpha_t < \\sqrt{\\alpha_t},$$\n\nwhere $\\text{max}[\\cdot]$ is the pixel-wise maximization across all pixels, $e_0 \\in R^{h\\times w \\times c}$, $h$, $w$, and $c$ represent the image height, width, and channels, respectively.\n\n> **Q4. Explanation of Line 135.**\n\nIn our devised model, the transition distribution defined in Eq. (1) gradually shifts the residual, accompanied by a minor perturbation by Gaussian noise, during the forward process. The intensity of the Gaussian noise is controlled by the hyper-parameter $\\kappa$. For a sufficiently large value of $\\kappa$, e.g., 40,  the noise perturbation becomes predominant within the diffusion process, leading to a convergence towards a Gaussian diffusion model.\n\nFor a discrete Gaussian diffusion model, the progression of the propagation can be equivalently reflected by the signal-to-noise-ratios of the hidden states (SNRs) in each timestep. By setting $\\kappa=40$, $p=0.8$, and $T=1000$, our proposed method demonstrates a closely similar SNR curve to LDM as shown in Fig. 2(g) in our paper. Furthermore, Fig. 2 (e) and (f) visualize the intermediate states of the forward process as observed in our method and LDM. Combining the SNR profile and the visual comparison, it becomes apparent that our method degenerates into a diffusion process that is very close to LDM under such a setting.   \n\n> **Q5. Efficiency comparison to the GAN-based.**\n\nPlease see Q2 of the global response.\n"}, {"review_id": "6MKnIIuxGn", "reviewer": "Reviewer_RmTK", "summary": "The authors propose a new diffusion model, based on the principle of residual shifting, that is able to converge to a good looking image in a low number of diffusion steps, improving in terms of high resolution image inference time at least over the Latent Diffusion Model (LDM) SR variant, another well-established diffusion based solution  for image upscaling. \n\nThe authors provide extensive ablations (both is the main manuscript and the supplementary material) showing that the model is able to achieve a significant level of performance, at least in terms of subjective results analysis. The trade-off between the number of diffusion steps and the achieved \"image quality\" (subjective) is well explored. However, some concerns and comments will be found in the appropriate section of the review. \n\n The metrics chosen for performance quantification (PSNR, SSIM, LPIPS, CLIPIQA,  MUSIQ) show mixed results, with the model characterized by SOTA performance in terms of non-reference image metrics, with strong indicators in terms of CLIPQA and MUSIQ. In terms of LPIPS, the model also shows an advantage over the compared methods. In terms of PSNR and SSIM, the model shows its limitations, however explainable, given the nature of the algorithm and the followed framework.   ", "questions": "1) Where is the difference  in the number of parameters of the diffusion backbone against the LDM coming from?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper is generally well written and easy to follow. The provided details are enough to offer a good overview of the proposed method. \n2) The authors provide ablations showcasing the influence of multiple parameters characterizing the proposed method. \n3) The quality of the results selected for visual comparison matches the performance level in terms of quality assessment related metrics.  \n4) An advantage over the LDM in terms of inference time/diffusion steps needed  to achieve a similar image quality assessment performance can be easily derived from the Table 2 of the main manuscript.  \nThe quality of the results provided for visual comparison and the perspective for more efficient diffusion models explain my rating. ", "weaknesses": "1) The efficiency claim made in the title is not clearly supported by the provided evaluations. Even if the advantage over LDM in terms of inference time and diffusion steps needed is clear, there is no comparison with the other methods considered for comparison (Table 3 and 4).\nThis work should clarify the question mark over the opportunity of using a diffusion model for Efficient Image Super Resolution, show the model strengths and let the readers evaluate the trade-off between reconstruction fidelity and subjective \"image quality\". This is why I added this as the first weak point of the authors claim. \n2) The choice in presenting the results of their ablative study (Table 1) versus the against-SOTA (Table 3,4) is somehow strange. the configurations in terms of data are different making it impossible to quantify the advantage between their proposed models and different configurations of LDM. Also, a fully supervised Image Super Resolution CNN architecture could be also added, to better support the efficiency claim as performance gain per inference-time millisecond. \n3) A certain trend can be observed in the first 5 lines of the Table 1, where the performance of the model in terms of reconstruction fidelity decreases with the number of diffusion steps  followed. This can show an overfitting behaviour on some properties of the training set that are not aligned with the properties of the testing split.\n       3.1)  Have the authors focused their efforts on strategies to supervise the training of their model in order to retain as much as possible \n       from the reconstruction fidelity, while achieving significant subjective performance?\n4) When setting the numbers of diffusion steps to 20, there seems to be a strong correlation between the parameter p value and the \nreconstruction fidelity of the model. However, the value for p chosen for the final configuration was 0.8. Was the choice made given \nthe impact in performance in terms of CLIPQA/MUSIQ? Why? \n\n5) A comparison with another referenced method claiming improved inference time for a diffusion-based algorithm (maybe also Reffusion) would be useful for the potential reader, to understand the advantages of the proposed method (reference 48 of the main manuscript). \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1. Efficiency comparison with the other methods in Table 3 and 4.**\n\nWe have offered more comprehensive comparable analysis on the efficiency as suggested. Please see Q2 of the global response.\n\n> **Q2. The choice in presenting the results of their ablative study (Table 1) versus the against-SOTA (Table 3,4) is somehow strange. the configurations in terms of data are different making it impossible to quantify the advantage between their proposed models and different configurations of LDM.**\n\nWe would like to elucidate that the ablative study in Table 1 and the comparisons with current SotA methods in Table 3 are both conducted on the synthetic testing dataset “ImageNet-Test” (see Sec 4.1 of our manuscript). It is imperative to note that the dataset configuration remains consistent between Table 1 and Table 3. To better evaluate the effectiveness of our method in authentic real-world scenarios, we further made a comparison with existing method on three real-world datasets and presented the results in Table 4. Due to the unavailability of ground truth images in real-world datasets, our scrutiny primarily focused on non-reference metrics, namely CLIPIQA and MUSIQ, as highlighted in Table 4.\n\n> **Q3. A certain trend can be observed in the first 5 lines of the Table 1, where the performance of the model in terms of reconstruction fidelity decreases with the number of diffusion steps followed. This can show an overfitting behaviour on some properties of the training set that are not aligned with the properties of the testing split. 3.1) Have the authors focused their efforts on strategies to supervise the training of their model in order to retain as much as possible from the reconstruction fidelity, while achieving significant subjective performance?**\n\nWe kindly argue that the decline in reconstruction fidelity with an increase in diffusion steps should not be attributed to an overfitting onto the training data. Rather, this is a well-known phenomenon called “Perception-distortion Trade-off” [1] in the field of image restoration. In particular, the augmentation of the generative capability of a restoration model, such as elevating the sampling steps for a diffusion-based method or amplifying the weight of the adversarial loss for a GAN-based method, will result in a deterioration in fidelity preservation, while concurrently enhance the authenticity of restored images. That’s mainly because the restoration model with powerful generation capability tends to hallucinate more high-frequency image structures, thereby deviating from the underlying ground truth.\n\nTo facilitate a more comprehensive comparison between our method and LDM, we have plotted the perception-distortion curves of them in Fig. 3 (b) of the accompanying rebuttal file. Herein, the perception and distortion are measured by the metrics of CLIPIQA and mean square-error (MSE), respectively. The plot reflects the perception quality and the reconstruction fidelity of the proposed method and LDM across varying numbers of diffusion steps, i.e,, 10,  20, 30, 40, 50, and 100. Significantly, the perception-distortion curve of our ResShift consistently resides beneath that of the LDM, thereby indicating its superior capacity to manage the perception-distortion equilibrium. \n\n[1] The Perception-Distortion Tradeoff, CVPR 2018.\n\n> **Q4. When setting the numbers of diffusion steps to 20, there seems to be a strong correlation between the parameter $p$ value and the reconstruction fidelity of the model. However, the value for p chosen for the final configuration was 0.3. Was the choice made given the impact in performance in terms of CLIPQA/MUSIQ? Why?**\n\nYes, you’re right. In our final model, the hyper-parameter of $p$ is set as 0.3, hoping to enhance the perception quality measured by LPIPS, CLIPIQA, and MUSIQ. In the real applications of super-resolution, the prevailing preference consistently gravitates towards outcomes characterized by heightened perceptual quality, in contrast to the conventional emphasis on reconstruction fidelity. This principled inclination is grounded in one common observation that the restored image with superior reconstruction fidelity, as measured by the metric of PSNR, often tends to be blurry. Actually, one can adjust the hyper-parameter $p$ freely according to your requirement.\n\n> **Q5. Performance comparison with the other reference methods.**\n\nPlease see the Q1 of the global response.\n\n> **Q6. Where is the difference in the number of parameters of the diffusion backbone against the LDM coming from?**\n\nRegarding the diffusion backbone, we adopted the codebase of the guided diffusion model (https://github.com/openai/guided-diffusion) and followed its default configurations. The difference in the number of parameters arises from different condition manners on the time embedding within the block of ResNet. In the guided diffusion model, a spatial feature transform layer is employed to modulate the features of ResNet based on the time embedding. In contrast, the time embedding vector is directly added to the features of ResNet in LDM.\n\nIt should be noted that we have not specifically optimized the backbone to pursue more performance gain, just following the settings in existing work. The superiority of our method primarily comes from the elaborate design on the diffusion model, which serves as the core contribution of this work.\n\n\n"}, {"review_id": "8OTNr63zVu", "reviewer": "Reviewer_Mv36", "summary": "The paper presents a new image super-resolution diffusion model called \"resshift\", which aims to address the efficiency issue in diffusion models. Existing acceleration strategies often yield over-smooth results. To combat this, the authors propose a new diffusion model for super-resolution that can produce favorable results in just 20 sampling steps. This method is based on operating on residuals and the paper also proposes a noise-adding strategy to aid this process. The main experiments are performed on real super-resolution settings, imitating the prior setup of Real ESRGAN, and includes testing on real images.", "questions": "see Weakness", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The experimental results appear quite promising. Specifically, the outcomes produced with 20 steps seem slightly better than existing methods. This shows that the proposed method can efficiently achieve super-resolution with fewer sampling steps, and the introduced noise-adding strategy appears to be a beneficial supplement to the overall approach.", "weaknesses": "The major issue with this paper lies in its presentation. The central idea of the paper is not clearly articulated. In the second section, although the method is described meticulously, it does not seem significantly different from existing methods. The authors do not clearly highlight what distinguishes their approach. It is recommended that the authors construct a table to illustrate why the \"shift\" and \"residual\" ideas are useful, and provide a conceptual narrative at the beginning of the second chapter to give readers an overall impression before diving into mathematical expressions.\n\n\nSecondly, Figure 2 could potentially be misleading. It should be the image obtained by adding noise in the VQ-VAE's latent space, but this is not just noise because a conversion to RGB image is performed. This needs further clarification since it's difficult to discern that Gaussian noise is added without any explanation.\n\n\nLastly, the method's versatility needs to be elaborated on. The method does not seem to be confined to super-resolution and appears to be applicable to all image restoration models. The question arises whether it could be applied to other image processing tasks. There also needs to be more comparative analysis, as many recent acceleration methods, including CM models, have not been included in the comparison.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1. The major issue with this paper lies in its presentation. The central idea of the paper is not clearly articulated. In the second section, although the method is described meticulously, it does not seem significantly different from existing methods. The authors do not clearly highlight what distinguishes their approach.**\n\nThe motivation of this work is derived from an intuitive observation, which posits that the transition from an high-resolution (HR) image to its low-resolution (LR) counterpart displays enhanced efficiency, characterized by a reduced number of diffusion steps, in comparison to the transition from the HR image to a Gaussian noise. This assertion stems from the fact that the residual between the HR-LR image pairs are often small. Such a diffusion process can be implemented by gradually shifting this residual, and one corresponding visualization of the hidden states during transition is shown in Fig. 3(a) of the affiliated rebuttal file. The experimental results also well substantiate the rationality of this motivation. Besides, we are very happy on that our motivation and the presentation are highly recognized and appreciated by all the other Reviewers.\n\nFurthermore, we want to kindly clarify the contribution of this work so as to distinguish from existing methods. Based on the aforementioned motivation, we innovate by designing a diffusion model from the mathematical formulation of the forward and reverse processes to the optimization strategy. The devised model enables a seamless transition between the HR-LR image pairs, being different from extant approaches that transition from HR image to Gaussian noise. Beyond the scope of this theoretical model innovation, a more flexible noise schedule is also proposed to better control the perception-distortion trade-off. Collectively, these contributions serve to establish clear differentiation between our approach and previous research.\n\n> **Q2. Figure 2 could potentially be misleading. It should be the image obtained by adding noise in the VQ-VAE's latent space, but this is not just noise because a conversion to RGB image is performed. This needs further clarification since it's difficult to discern that Gaussian noise is added without any explanation.**\n\nWe sincerely appreciate this constructive suggestion. We want to firstly clarify that our proposed method provides a general framework to establish a transition between any two variables, i.e., the HR image $x_0$ and LR image $y_0$ in the context of image super-resolution. Given a pre-trained VQGAN with encoder $E$ and decode $D$, it is naturally to extend our method to model the transition between $z_0$ and $v_0$ in latent space, where $z_0=E(x_0 )$, $v_0=E(y_0)$. It should be noted that when modelling in latent space, the encoding and decoding procedures only occur once in the initial and terminal states during forward and reverse process. For example, in the reverse process, we only need to estimate $z_0$ from $v_0$ that encoded from $y_0$ using the trained diffusion model, and then obtain the restored $x_0$ via $x=D(z_0)$. Thus, it is unnecessary to concern the conversion between the latent space and the RGB space for the introduced Gaussian noise in our model. \n\nNext, we give more details about the visualization in Fig. 2. Specifically, we first mapped $x_0$ and $y_0$ to $z_0$ and $v_0$ via the encoder $E$, and then generate the intermediate states $\\lbrace z_t \\rbrace_{t=1}^T$ using the transition kernel in Eq. (1) in latent space, and finally convert them to the RGB space through $x_t=D(z_t)$. In fact, Figure 2 in our paper visualizes the decoded image $x_t$. We will add this explanation in our revised version for easy understanding.\n\n> **Q3. Whether the proposed method could be applied to other image processing tasks.**\n\nYes. The proposed method is a general methodology to address a wide spectrum of restoration tasks. To verify such a versatility of our method, we further evaluate its effectiveness on the task of blind face restoration. A subset of 3000 images are randomly selected from CelebA as our testing dataset following CodeFormer (NeurIPS 2022). The quantitative comparisons to GFPGAN (CVPR 2021), VQFR (ECCV 2022), and CodeFormer are listed in Table 3 of the associated rebuttal document. Furthermore, the qualitative comparison on one real-world example is also shown in Fig. 1. We can easily see that ResShift achieves the best or at least comparable performance to recent SotA method CodeFormer. This compellingly suggests the potential for the seamless extension of our method to blind face restoration.\n\n"}, {"review_id": "Dpkrw0Mv1b", "reviewer": "Reviewer_k6dY", "summary": "This paper studies diffusion-based image super-resolution (SR) with the goal of reducing the number of diffusion steps. The key intuition is to only learn the residual between an LR-HR image pair, thereby shortening the diffusion path. To this end, the paper introduces ResShift, a novel diffusion framework where the mean of the prior distribution is the LR image. It further provides a derivation of the forward and reverse processes, identifies an effective parameterization of the denoiser network, and analyzes the fidelity-realism tradeoff induced by the noise schedule. Extensive quantitative and qualitative experiments are conducted to verify the effectiveness and efficiency of the proposed method. In particular, ResShift achieves competitive results on the 4x SR task at a low cost of 20 diffusion steps, outperforming the strong baseline of LDM under the same sampling budget.", "questions": "Please see the section above for questions.", "limitations": "The paper discusses limitations of the method. The authors are encouraged to provide a short discussion on the potential negative societal impact of their work, such as hallucination of inappropriate image content.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper makes the key observation that the residual between an LR-HR image pair is often small. It builds a novel diffusion framework around this insight to reduce the number of sampling steps for improved efficiency. Incorporating domain knowledge into the design of a diffusion model is a key strength of the paper.\n\n- Within the proposed diffusion framework, the paper studies the fidelity-realism tradeoff induced by noise schedules through detailed theoretical analysis and ablation experiments.\n\n- The proposed method is evaluated on both synthetic and real datasets. It demonstrates comparable or sometimes better results with respect to strong baselines. In particular, it consistently outperforms LDM under the same sampling budget (20 or 40 steps).\n\n- Overall, the paper is well-written, and the flow of presentation is easy to follow.", "weaknesses": "- My main concern is that the comparison with LDM is unfair. In the main experiments, the number of diffusion steps is limited to 20 or 40. This makes sense, as the goal is to demonstrate that ResShift yields stronger results under a tight sampling budget. However, LDM is put at a significant disadvantage in this comparison, because its default DDIM sampler is not well-suited for short sampling paths. In this regime, more powerful samplers exist (e.g., Heun and DPM solvers) and, according to the reviewer's own experience, will likely produce far better results within 20 or 40 NFEs. This raises an important question: is the proposed diffusion model tailored for SR truly superior to the vanilla model in terms of efficiency, when the latter is paired with a fast solver? This is key to assessing the paper's contribution, yet is not addressed by the current set of experiments.\n\n- DDRM is another diffusion-based model for SR in addition to LDM. The authors are encouraged to compare ResShift against DDRM as well.\n\n- The qualitative results on real data seem inconsistent across different images. In Figure 4, ResShift produces sharper and more detailed output than the baselines, whereas it is more prone to artifacts as shown in Figure 1 in the supplement. I am wondering what causes this discrepancy. Are the images from different real datasets, or is there just a lot of randomness?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1. The comparison with LDM is unfair. More advanced samplers should be considered.**\n\nAs suggest, we conduct a comparison to LDM with 20 sampling steps accelerated by more advanced samplers, including PNDM (ICLR 2022) and DPM (NeurIPS 2022). The quantitative comparison results on the testing dataset of ImageNet-Test are presented as follows:\n\n| Methods | Sampler |PSNR$\\uparrow$ | SSIM$\\uparrow$   | LPIPS$\\downarrow$ | CLIPIQA$\\uparrow$ | MUSIQ$\\uparrow$ |\n| :----:        | :----:      | :----:   | :----:   |:----:    | :----:       | :----:     |\n| LDM-20  | DDIM     | 24.76  | 0.639  |0.284  | 0.630      | 48.355   |\n| LDM-20  | PNDM   | 22.87  | 0.549   |0.285  | 0.738      | 55.820  |\n| LDM-20  | DPM      | 22.28  | 0.526   |0.302  | 0.728      | 54.123  |\n| F-ResShift-20  | -   |23.72   | 0.615   |0.246  | 0.773      | 57.408  |\n\nIt can be easily seen that the incorporation of these advanced sampling techniques brings up pronounced enhancement in perceptual quality, as assessed by CLIPIQA and MUSIQ. However, this performance augmentation comes at a substantial deterioration on fidelity, quantified by PSNR and SSIM. Even in light of this, our proposed method still consistently exhibits evident superiorities across all five metrics. Therefore, the investigations undertaken in this study assume a pivotal and noteworthy significance.\n\n> **Q2. Performance comparison with DDRM.**\n\nDDRM is a non-blind diffusion-base restoration method, and cannot handle the complicated real-world image super-resolution with unknown degradation types. Thus, we omit the comparison with DDRM in the submitted manuscript. To more comprehensively evaluate the proposed method, we conduct a fair comparison with DDRM on the task of x4 bicubic super-resolution. Please see Q1 of the global response.\n\n> **Q3. Artifacts in Fig. 1 of the supplement.  Is there just a lot of randomness?**\n\nWe want to elucidate that the artifacts observed in rare instances,  e.g., the second example in Fig. 1 of the supplement,  can be predominantly attributed to the inadequate\n training. Our model in the submitted version undergoes a training over 500k iterations with a batch size of 64. We empirically find that the artifacts can be substantially mitigated by prolonging the training process to 800k iterations. This extended model is denoted as F-Reshift+, and the revised results are presented in Fig. 4 of the accompanying rebuttal file. Notably, it is necessary to note that LDM is trained with 2560k iteration in a batch size of 256, incurring significantly greater computational resources than our proposed approach.\n\nBesides, the restored results by F-ResShift+ under multiple different random seeds are also shown in Fig. 4 (h)-(k). We can see that the randomness among multiple solutions is exceedingly slight, thus deemed acceptable in the task of image super-resolution.\n"}, {"review_id": "RrFvoqNoYy", "reviewer": "Reviewer_PfAe", "summary": "This paper constructs a Markov chain that transfers between high-resolution and low-resolution by shifting the residual between them. They achieve competitive performance to SOTA methods with only 20 sampling steps. ", "questions": "1. Since there are many math equations, it may be difficult for other researchers to reproduce the method. Would the implementation code be released? ", "limitations": "The authors have addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper structure is clear and the proposed method is well-motivated.\n2. It only requires small iterating  steps to achieve competitive performances.", "weaknesses": "1. A brief figure to introduce Resshift should be provided.\n2.  Why this method is called ResShift? The motivation and function should be discussed. Moreover, the ablation study of the res-shift should be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1. A brief figure to introduce ResShift should be provided.**\n\nThanks for your suggestion. We have provided an overview of our model in Fig 3. (a) in the attached rebuttal document, and will add it to our paper in the revised version.\n\n> **Q2. Why this method is called ResShift? The motivation and function should be discussed.**\n\nThe motivation of this work is derived from an intuitive observation, which posits that the transition from an high-resolution (HR) image to its low-resolution (LR) counterpart is more efficient compared to that from the HR image to a Gaussian noise. This assertion stems from the fact that the residual between the HR-LR image pairs are often small. Therefore, this study develops a novel diffusion model that builds up a Markov chain between the HR-LR images by gradually **Shift**ing the **Res**idual of them. We thus refer to the proposed diffusion model as ResShift, encapsulating the essence of this methodology.\n\n> **Q3. Moreover, the ablation study of the res-shift should be provided.**\n\nThe whole formulation of our proposed model is methodically crafted based on the strategy of residual shifting. It is thus impossible to make an ablation study on the res-shift. However, we have provided sufficient ablation analysis on the other configurations of our model in Sec. 4.2.\n\n> **Q4. Would the implementation code be released?**\n\nWe promise to release the source code of this work after revision."}]

Author rebuttal:
Dear AC and reviewers,

We sincerely thank all reviewers for their constructive comments. Since Reviewer vFLR, Reviewer RmTK, and Reviewer k6dY all concern the performance comparison with other related methods, Reviewer vFLR and Reviewer RmTK both require to supplement the efficiency comparison to GAN-based methods. Thus, we address these two issues here.

>  **Q1. Performance comparison with other related methods.**

Since our proposed method primarily focus on the real-world image super-resolution, we exclusively compare with recent approaches that share a similar orientation. Regarding IRSDE [48] or its enhanced version Refusion (CVPR Workshop 2023), the publicly accessible models are specifically trained for the bicubic super-resolution or stereo super-resolution tasks, and cannot handle the general real-world image super-resolution. Correspondingly, I2SB [49] is similarly limited to bicubic super-resolution. DDRM (NeurIPS 2022) is a non-blind image restoration method, thus lacking the capability to address the real-world image super-resolution with intricate and unknown degradation. For InDI [47], the source code and the pre-trained model are not publicly available. Therefore, our paper omits the comparison to these methods. 

Furthermore, we train a model tailored for x4 bicubic super-resolution, thereby facilitating a fair and comparison against IRSDE, I2SB, and DDRM. For I2SB and DDRM, we expedite the inference process to 20 steps using the default sampler. For IRSDE, we retain the sampling steps same with its setting in training, , i.e., 100 steps,  because accelerating this process during inference will yield a severe performance drop. To conduct a comparative analysis, 3,000 images were randomly selected from the validation dataset of ImageNet to serve as our testing dataset. The comprehensive quantitative comparison results are listed in Table 1 of the accompanying rebuttal document. We can easily observe that the proposed method outperforms the aforementioned approaches across various evaluation metrics, the number of parameters, and the inference speed. This indicates the efficacy of our meticulously designed diffusion model.

> **Q2. Efficiency comparisons to other competing methods.**

We provide a comprehensive comparison on the performance and efficiency of our proposed method to LDM and the GAN-based methods in Table 2 of the rebuttal document. For a more holistic assessment, the efficiency evaluations of our method against recent diffusion-based techniques are also presented in Table 1 of the rebuttal document.

Combining these comparative results, we obtain the following conclusions: i)  In contrast to existing diffusion-based methodologies, our proposed method have achieved significant improvement in both performance and efficiency. ii) While current diffusion-based methods have shown notable superiority due to their powerful generation capability, they still lag behind GAN-based models in terms of efficiency. Notably much faster than the diffusion-based techniques as shown in Table 1, our proposed method yet remains twice as slow as SwinIR, the present SotA GAN-based model. We will explore how to further speed up our method in the future work.  


Meta-review: The work proposes an efficient diffusion model for image super-resolution by shifting the residuals and not (re)building the image from scratch like prior work does. This leads to significant benefits as supported by the reported experimental results.

The authors provided satisfactory responses and convinced Reviewers vFLR, RmTK and k6dY to upgrade their ratings. Thus, for this work there is a consensus on the positive side (4xWeak Accept, 1xBorderline Accept) from all five Reviewers.

After reading the paper and carefully checking the reviews and the authors' responses the ACs agree with the reviews that the present work makes important contributions and is of interest for the community.

The authors are invited to further refine their paper for the camera ready by including (part of) information/details from their responses to the reviewers' comments.
Decision: Accept (spotlight)