Taming Diffusion Prior for Image Super-Resolution with Domain Shift SDEs
Authors: Qinpeng Cui
Yi'xuan Liu
Xinyi Zhang
Qiqi Bao
Qingmin Liao
liwang Amd
Lu Tian
Zicheng Liu
Zhongdao Wang
Emad Barsoum
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Models; Super-Resolution
Primary area: diffusion_based_models

Abstract: Diffusion-based image super-resolution (SR) models have attracted substantial interest due to their powerful image restoration capabilities. However, prevailing diffusion models often struggle to strike an optimal balance between efficiency and performance. Typically, they either neglect to exploit the potential of existing extensive pretrained models, limiting their generative capacity, or they necessitate a dozens of forward passes starting from random noises, compromising inference efficiency. In this paper, we present DoSSR, a $\textbf{Do}$main $\textbf{S}$hift diffusion-based SR model that capitalizes on the generative powers of pretrained diffusion models while significantly enhancing efficiency by initiating the diffusion process with low-resolution (LR) images. At the core of our approach is a domain shift equation that integrates seamlessly with existing diffusion models. This integration not only improves the use of diffusion prior but also boosts inference efficiency. Moreover, we advance our method by transitioning the discrete shift process to a continuous formulation, termed as DoS-SDEs. This advancement leads to the fast and customized solvers that further enhance sampling efficiency. Empirical results demonstrate that our proposed method achieves state-of-the-art performance on synthetic and real-world datasets, while notably requiring $\textbf{\emph{only 5 sampling steps}}$. Compared to previous diffusion prior based methods, our approach achieves a remarkable speedup of 5-7 times, demonstrating its superior efficiency.

Taming Diffusion Prior for Image Super-Resolution with Domain Shift SDEs
Qinpeng Cui 12, Yixuan Liu', Xinyi Zhang", Qiqi Bao?, Qingmin Liao²
Li Wang', Tian Lu', Zicheng liu', Zhongdao Wang ? Emad Barsoum'
AAdvanedd Micro Devices Inc. TTinghua University
fqinpeng. cui, yixuan.liu, 1i. Li.wang, lu.tian, zicheng liu, ebarsoum Searsoum-@aadd com; {cqp22, xinyi-zh22, bqq19, liaoqm, wcd17)@tsinghua eew.ghuuaeeu....e.... cn
Abstract
biffusion-based image super-resolution (SR) models have attracted substantial interest due their powerful image restoration capabilities. However, prevailing diffusion models often struggle to strike an optimal balance between efficiency and performance. Typically, they either neglect exploit the potential of existing extensive pretrained models, limiting their generative capacity, or they necessitate dozens of forward passes starting from random noises, compromising inference efficiency. In this paper, we present DoSSR, Domain Shift diffusion-based SR model that capitalizes on the generative powers of pretrained diffusion models while significantly enhancing efficiency by initiating the diffusion process with low-resolution (LR) images. At the core of our approach a domain shift equa- tion that integrates seamlessly with existing diffusion models. This integration not only improves the use of diffusion prior but also boosts inference efficiency. Moreover, we advance our method by transitioning the discrete shift process to continuous formulation, termed as DoS-SDEs. This advancement leads the fast and customized solvers that further enhance sampling efficiency. Empirical results demonstrate that our proposed method achieves state-of-the-arr performance synthetic and real-world datasets, while notably requiring only 5 sampling steps. Compared to previous diffusion prior based methods, our approach achieves remarkable speedup of 5-7 times, demonstrating its superior efficiency. Code: https::/giihub..cm///a Bat.:a//.oo/.DD-...A--A-- AIG-AAMMA/DDSSSS
Introduction
Image super-resolution (SR) is a classical task in computer vision that involves enhancing a low- resolution (LR) image create a perceptually convincing high-resolution (HR) image (28). Tradi-
tionally, this field has operated under the assumption simple image degradations, such as bicubic down-sampling, which has led to the development of numerous effective SR models 6 25 57 12
However, these models often fall short when confronted with real-world degradations, which are typically more complex than those assumed in academic settings. Recently, diffusion models has emerged as pivotal research direction in real-world SR, using their robust generative capabilities to enhance perceptual quality. This shift highlights their superior performance in practical applications.
Currently, diffusion-based SR strategies can be broadly categorized into two approaches. The first approach leverages large-scale pretrained diffusion models (e.g.. Stable Diffusion (41)) as generative prior, using LR images (or preprocessed LR images) as conditional inputs to generate HR images 45 51] Despite achieving remarkable results, it exhibits low inference efficiency,
"Work done during an internship at AMD. t CCorrepoondig author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Latency vs. MANIQA Parameters DoSSR(oura):5
0.6
SccSR-50
DoSSR(ours)-1
StableSR-200
NANNDA 1
0.5
Resshin-15
DirTBIR-50
Image
RealE
LDL
DASR
0.4
LDL
Real-ESRGAN
Diameter
DASR
[#Params:1000M]
0.3
2
17.5
20.0
Latency(s)
StableSR-200
Resshift-15
SeeSk-50
DoSSR
(a)
Figure I: (a) Latency, MANIQA, and complexity of model comparison on RealLR200 [5]] dataset in x4 SR task (for 128x128 LR images). (b) Qualitative comparisons DoSSR and recent state-of- the-art methods on one typical real-world example. For diffusion suffusion-bbsed methods, the suffix "-N" appended to the method name indicates the number of inference steps. Zoom in for a better view.
as the inference starting point is a random Gaussian noise instead of the LR image. Although techniques such as sampler optimization [33 38 proposed to mitigate this issue, they inevitably compromise SR performance. The second approach involves redefining the diffusion process and retraining model from scratch for the SR task [53] 36
30 9 22 or model distillation (3213) have been
Consequently, the generative prior from pretrained diffusion models is not leveraged. ResShift [53]. as a typical representative, revises the forward process of DDPM [16] to better accommodate the SR task. By starting from LR rather than Gaussian noise, it improves inference efficiency. However, its modification of the diffusion pattern, which deviates significantly from existing noise schedules in diffusion models, hinders its integration with large-scale pretrained diffusion models for leveraging their generative prior. The diffusion generative prior has been proven to be highly beneficial for SR tasks 45), enabling models transcend the limitations of knowledge learned solely from the training dataset, thereby equipping them to handle various complex real-world scenarios. Thus, crafting diffusion process tailored for the SR task that also remains compatible with established diffusion prior presents significant challenge.
To tackle this challenge, initially view the SR task as a gradual shift from the LR domain to the HR domain, describing this transition with linear equation, which is called domain shift equation. Then, combine this domain shift equation with existing diffusion equations, facilitating the fine-tuning of large-scale pretrained diffusion models harness diffusion prior effectively. Moreover, by carefully designing shifting sequence, inference can begin from LR images rather than Gaussian noises, thereby boosting inference efficiency. To further enhance efficiency, we employ sampler optimization techniques, but not previously tailored for diffusson SR tasks. Specifically, we expand the customized diffusion equation from discrete continuous, enabling its formulation as stochastic differential equations (SDEs). We subsequently present the corresponding backward-time SDE as Domain Shift SDE in the reverse process and provide an exact formulation of its solution. Based on our formulation, we customize fast solvers for sampling. Experimental results demonstrate that our method achieves superior or comparable performance compared to current state-of-the-an methods on both synthetic and real-world datasets, with only 5 sampling steps, striking an optimal balance between efficiency and effectiveness. Furthermore, our approach can match the performance of previous methods even with just single step.
propose DoSSR, Domain Shift diffusion-based SR model. We
extensively explored in image generation (38 30 91
In summary, the main contributions of our work are as follows:
We propose a novel diffusion equation, which models SR from the perspective of domain shift, enabling inference to start from LR images and leveraging diffusion prior to ensure both efficiency and performance.
We further propose the SDEs to describe the process of domain shift and provide an exact solution for the corresponding reverse-time SDEs. Based the solution, we design customized fast samplers, resulting in even higher efficiency, thereby achieving the state-of-the-an efficiency- performance trade-off.

---

2 Related work
Neural Network-based Super-Resolution. Neural network- based methods have emerged as the dominant approach in image SR tasks. The introduction of convolutional neural networks (CNNs) and Transformer architecture, with the primary focus on network architecture design [2 I0 25 26
588 21 [56 57]. have demonstrated superior performance over traditional methods. This improvement is facilitated by the introduction of residual blocks, dense blocks and attention mechanisms. These methods primarily aim for better image fidelity measures such as PSNR and SSIM [49] indices, therefore, they often yield over-smoothed outcomes. To enhance visual perception, Generative adversarial network -GAN)-based SR methods have been developed. By incorporating adversarial loss during training, many SR models [3] 23 77 can generate perceptually realistic details, thereby enhancing visual quality. To further study SR problems in real-world scenarios, some studies [54] 46 24 have proposed simulating the intricate real-world image degradation process through random combinations of fundamental degradation operations. Despite the remarkable advancements, GAN- based SR methods can introduce undesirable visual artifacts.
Diffusion-based Super-Resolution Recently, diffusion- baasdd SR methods [35 36 45 27
have demonstrated excellent performance, especially in terms of perceptual quality. These methods can generate more authentic details while avoiding unpleasant visual artifacts like GAN-based methods. Current diffusion models for super-resolution can be broadly categorized into two main approaches. The first approach involves leveraging large-scale pretrained diffusion models, such as Stable Diffusion [11] as prior, and then using LR images as conditional inputs to generate HR images. StableSR
451 and DiffBIR 27 represent representative works that leverage diffusion prior, leading to enhanced fidelity when conditioning on LR or preprocessed LR. SeeSR [5]] and CoSeR [4] demonstrate that extracting semantic text information from LR images as additional control conditions for the T2I model helps improve performance. The second approach involves redefining the diffusion process and retraining a model from scratch for SR [8] 36). To address the slow inference speed issue of diffusion-based SR methods, ResShift [53] constructs a Markov chain that transitions between HR and LR images by shifting residuals between them, enabling accelerated sampling. SinSR 48) proposed a method of distilling ResShift to achieve comparable performance in a single step. Despite the remarkable advancements achieved by ResShift and SinSR, they necessitate retraining from scratch for SR tasks (or further distillation) and are unable leverage diffusion prior. Therefore, improving the inference efficiency while leveraging the potential of large-scale pretrained diffusion models to assist SR requires thorough investigation, which is the goal of this work.
3 Methodology
We aim to optimize the balance between efficiency and performance in diffusion-based super- resolution (SR) models. Our approach is grounded in two key principles: First, initiating inference from LR images rather than noise; Second, effectively harnessing pretrained diffusion prior. In Section 3.1 we introduce a novel diffusion equation designed to fulfill both criteria simultancously. Subsequently, in Section 3.2. we extend this diffusion process to continuous scenarios, formulating it through Stochastic Differential Equations (SDEs). Building on these SDEs, we develop an efficient solver detailed in Section 3.3 further enhancing inference efficiency.
3.1 Diffusion Process with Domain Shift
Our goal is to characterize the shift from the source domain to the target domain as a diffusion process. In the task of SR, the distribution of LR images Paata (mo) represents the source domain, while the distribution of HR images Pasta (Za) represents the target domain. Firstly, we conceptualize domain shift gradual transition from the source domain to the target domain through linear
drift coefficient 7). the domain shift equation is formulated as
D (at, xo) = n: + (1 Ne 70) 0 < nh < 1, 11tt=10 1,2, .......... ,T,
(I)
where shifting sequence ( ) monotonically non-decreases with timestep I. order to enable linear combination, we can interpolate 2 to match the same dimensions as æpp if necessary. Secondly, we combine this domain shift with the diffusion equation. To integrate with pretrained diffusion models, we adopt the most commonly used diffusion scheme from DDPM [16] and express the
3

---

(6) DOSSR forward
[c) Score field of SD (leff) and DoSSR (right)
HRR:
= =a,00,,, +o
comsin
Dodaka
[.R:
Lomned diffssnn
HE dosEa
path for
D(e 14)
We (1
D fe)
(d) Shifting suquence T].
(b) DoSSR backward
Starting poin
Optional staring point,
%
X--4
Xy
cos(or//1))
DoSER sampler((..))
Figure 2: Illustration of the proposed diffusion process with domain shift. (a) In the forward process. we merge the gradual shift from HR to LR domain with standard diffusion process. (b) In the reverse process, we initiate inference from LR domain (t f1) and use our fast sampler to generate SR images. (c) Comparison of the estimated score between SD and DoSSR. DoSSR inherits the capability of SD in ambient space and enhances learning a pathway from LR to HR domain. (d) The design of the shifting sequence which enables us to initiate inference from t
formula of marginal distribution at any timestep as follows:
n( #o) (xx (u, ma), T),, 1 1,2,
(2)
where at >. 0 and aj of 1, I is the identity matrix. Based proposed marginal distribution Eq. (2) we demonstrate the transition distribution as follows:
21-1, 2o) Nax
Ot xt + an(n M/-1) 0,-1
of a?
t= ,T (3)
where eo 20 vo the residual between the source and target domain.
The formulation of Eq. (2 is based on the DDPM 16] forward process, with crucial difference lying in its mean a; (@o 2o) instead of 0g This integration encapsulates the domain shift within the variation of its mean, while the diffusion process with added noise maintains consistency with it, thereby smoothing this transformation. Meanwhile, it enhances the diffusion model to learn the pathway from the source domain to the target domain. For an intuitive understanding, we plot and compare the score function V. log q((z) learned by a vanilla Stable Diffusion (SD) model and DoSSR in Fig. [cc. While SD learns reasonable score field in the whole space, DoSSR inherits capability in the ambient space and further learns more accurate scores along the path between LRR and HR domains. Therefore sampling efficiency is improved. Furthermore, the choice of a and o,, referred to as the noise schedule, follows the existing pretrained diffusion model, allowing to fine-tune it rather than training it from scratch.
Relation to DDPM [6].
Relation to ResShift [53]. The form of equation Eq. (3) suggests that this shift essentially constructs Markov chain in a manner similar to that described in ResShift [53]. However, the equation constructed by ResShift adopts an entirely different noise schedule compared the pretrained diffusion model. This makes it difficult to apply pretrained diffusion models for subsequent fine- tuning, necessitating training from seratch instead. Therefore, it unable utilize the diffusion prior, thereby limiting the model's performance. See Appendix A.7 for detailed theoretical differences from ResShift. Appendix C.I] we present experimental results on ImageNet [1] showing that DoSSR uses two orders of magnitude less training data than ResShift while achieving superior performance.
Shifting Sequence. The parameter 7/ plays a crucial role in guiding the diffusion process, serving bridge between the source and target domains. Specifically, π7. represents standard diffusion

---

forward perturbations in the source domain, whereas 7): 0 corresponds to the target domain. The transition between these domains occurs for 0 < 7 < 1. indicating domain shift. To effectively utilize the diffusion prior, we adopt the noise schedule from DDPM. This adoption dictates that as i approaches the final time step 7, the scale parameter orr tends towards zero, and the distribution ((TT) approximates a standard Gaussian, (0, I). To retain prior information from the source domain while shortening the diffusion path, we set 7) = 1 for t [f,,T], as defined by:
cos( )
if tfte 0, 7 1 if t [T.
(4)
The advantage of such a setting lies in the fact that during the reverse process, the values of æe for t [t,, T] are known and can be obtained through the forward process Eq. (2). Consequently, the inference does not need to start from time step T, but can commence at f1, thereby preserving the prior information of the source domain while enhancing the efficiency of inference. An overview of the impact of is presented in Fig. 2
3.2 Diffusion DoS-SDEs
To improve the efficiency of inference in diffusion models, many prior works 30 31
have designed efficient samplers by solving the diffusion SDEs. Therefore, in this section, we extend the aforementioned discrete shift process an SDE for description, in preparation for designing efficient samplers in the following section. Specifically, inspired by the work of 40), we generalize this finite shift process further an infinite number of noise scales, such that the data distribution of domain shift evolves according to an SDE as noise intensifies. Then we provide the corresponding reverse- time SDE and elucidate the training of diffusion models from the perspective of score matching [39]. Next, we will elaborate extensively on how to describe diffusion models using SDEs.
9
Forward Process. Expanding the time variable in Eq. (2) to a continuous range, E [0, we have that Ot2, ot, 7/ are differentiable functions of t with bounded derivatives. Furthermore, Song et al. [4) have demonstrated that the diffusion process can be modeled the solution to an Itô SDE and we formulate the SDE as follows:
dæ; [f(t),, + h(1)@o]dt + g(t)du,, 2o (2()))
where w; is the standard Wiener process, and (2o) is the target domain data distribution. It has the same marginal distribution ((az)/zo @o) as Eq. (2) for any & E [0, T' with the coefficients satisfying (proof in Appendix
A.2
f(t) d log (1 - -m)) A(t)) = 0t dap 1- The di"
do? d do ar(1 m)) of.
g(t)
= dt
(6)
dt
dt
Reverse Process. The reverse of diffusion process is also diffusion process 2 which can similarly be described by reverse-time SDE (proof in Appendix A.3):
dx;
f(t) + h(t) - g² log qe (x)) dt + g())
where W. is also standard Wiener process when time flows backwards. In this paper, we refer to this SDE as Domain Shift SDE (DoS-SDE).
Score Matching. The only unknown term in Eq. 7 is the score function V log gr (ar:) that be estimated by training a score-based model on samples with score matching 39 In practice, we use neural network eo (z,, -tn, t) conditioned function (alternatively referred to as noise), following [16] 40). The parameter 0 is optimized by minimizing the following objectives:
to. parameterized by 0, to estimate the scaled score
0- arg min E, (()) (w:) [I ( So. 1) Vn (x:)(
0
(8)
Eg(e) Eo (xr, Do, <1l]}.
arg min E,
where w() is weighting function, æ, a @n (1 7/1 )mn) + o,e, and NO0,
Thus, we have completed the expression of the diffusion model using SDEs. Sampling from diffusion models can alternatively be seen solving the corresponding diffusion DoS-SDEs.
5

---

3.3 Solvers for Diffusson DoS-SDEs
In this section, we present an exact formulation of the solution of diffusion DoS- SDEs and design efficient samplers for fast sampling. To facilitate the solution of equation Eq. 7 we utilize the data prediction model zy (2,, e, &o. t). which directly estimates the original target data æy from the noisy samples. The relationship between score function and data prediction model is as follows (proof in Appendix A.
VR log (x,, 21 - (as -7) (a Co. t) ac7))o))
(as
(9)
of
In practice, we employ our trained noise prediction model EØ ( t) for data prediction
(a do, t) as described in Appendix A..4 By substituting Eq. (6) and Eq. (9) into Eq. 7 de
2t and introducing the substitutions X = oc(1 and y: a.(1-n)
along with the notation
dw>, = V ddi de
dur, ax == (())), wx == W), we rewrite Eq.
7 w.r.t À as
1
2 as do
2 mo S AjdA + 22ddux
dyx UadA+ +
d1yx 1 7/
(10)
We propose the exact solution for Eq. (10) using the variation-of-consten
formula, following 31
Proposition 3.1 (Exact solution of diffusion DoS-SDEs) Given initial value ag at time the solution H, for the diffusion DoS-SDEs defined in Eq. 7 at time E [0, s] is as follows:
0,
xt 0r 111))
X
as + ar(1 -11](
4)s
X
a,
17,))
X
71
1 - 7
X
(1I)
^e 2X,
A
az 71))
Iz (ax, Co. A)dA + (
11mmm)) me)
X?
\
x
2
where X a1(1-1)) and 2, 1).
The detailed derivation of this proposition is provided in Appendix AA.5 Notably, the nonlinear term in Eq. (11) involves the integration of non-analytical neural network æo 2o. A), which can be challenging to compute. For practical applicability. we employ Itô-Taylor expansion to approximate the integral of 2t from A to d to compute Kt, thereby approximating 4t, Additionally, we approximate the derivatives of æg using the forward differential method. These approximations allow us to derive SDE solvers of any order for diffusion DoS-SDEs. For the sake of brevity, we employ first-order solver for demonstration. In this case, Eq. (IT) becomes
(1 - 17) X? x. + a1 np)( Tl
X?
7)s
T a1 -7,) X2
@ao
7) X2
Shift Guidance(DoSG)
(12)
X² ar (z.. Co. 5) an(1 me X2
+ aa - (1
m
X2
X2 Z
The detailed derivation, as well as high-order solvers, are provided in Appendix A.6 and detailed algorithms are proposed in Appendix [B] Typically, higher-order solvers converge even faster because of more accurate estimation of the the nonlinear integral term. The solvers provided for sampling allow us to iteratively generate HR images using trained diffusion model. It is worth noting that Eq. (12) comprises four terms, including the additional linear term øo. as compared to the ancestral sampling algorithm [16]. We refer to this additional term the domain shift guidance (DoSG) which leverages prior information from the source domain and enhances the efficiency of inference.
Experiments
4.1 Experimental setup
For training, we train our DoSSR using a variety of datasets including DIV2K M. DIVSK [S). Flickr2K [43) and OST 47). To synthesize LR and HR training pairs, we adopt the degradation

---

Table 1: Quantitative comparison with state-of-the-art methods on both synthetic and real-world benchmarks, as well as comparison of latency and number of model parameters. NFE represents the number of function evaluations in the inference of diffusion models. The best and second best results of each metric are highlighted in red and blue, respectively.
Real- 16
Datasets
Metries
BSRGAN
54
LDL 23
DASR
StableSR 45
ResShif: 53
DiftBRR
Sce5R []]
DoSSR
ESRGAN
PSNR
24.58
24.29
23.83
24.47
23.36
24.65
23.67
23,68
23.98
SSIM 1
0.6241
0.6338
0.6312
0.6277
0.5654
0.6148
0,5592
0.5987
0.6073
LPIPSS
0.3351
0,3112
0.3256
0.3114
0.3349
0.3516
0.3195
0.3371
161
CLIPIQA
0.5276
0.5179
0.5016
0.6771
0.6065
0.6693
0.6955
0.7014
MUSIQ 1
61.19
61.06
60.04
55.19
65.92
61.07
65.78
68.68
66.54
MANIQA
0.3547
0.3795
0.3736
0.3165
0.4193
0.4107
0.4568
0.5041
0.5294
TOPQQ
0.5142
0.5974
0.5383
0.6142
0.6854
0.6766
PSNR1
26.38
25.28
24.65
26.26
24.81
25.14
SSIM
0.7655
0,7615
0,7565
0,7714
0,7060
0,7404
0.6571
0,7194
0,6839
LPIPS
0.2656
0.2209
0.2750
0.3134
0.3002
0.3469
0.3607
0.3007
0.3374
RealSR
CLIPIQA
0.5114
0,4485
0,4556
0.3198
0.6234
0.5473
0.6448
0.6699
0.7025
MUSIQ
63.28
60.37
60.93
41.21
65,88
58.47
64.94
69.82
69.42
MANIQA
0.3764
0,3733
0,3792
0.2161
0.1260
0.3836
0.4539
0.5406
0,5781
TOPIQ T
0.5502
0.5147
0.5124
0.3207
0.5743
0,4883
0.5722
0.6887
0.6165
PSNR
26.24
28.62
28.17
29.72
28.03
28.42
26.67
27,89
26.82
SSIM 1
0,8033
0,8050
0.8126
0,8264
0,7523
0.7629
0.6548
0.7565
0,7298
LPIPS
0,2858
0.2811
0.2792
0.3099
0.3284
0.1036
0,4517
0.3273
0.3689
CLIPIQA
0,4507
0.6357
0.5286
0.6391
0.6708
0.6776
DRealSR
MUSIQ 1
$7.16
54,28
$3.95
42.41
58.51
49.73
60.91
65.09
64..4
MANIQA
0,3424
0.3436
0.3444
0,2845
0.3867
0.3322
0.4486
0.5115
0.5214
TOPIQ
0.5058
0,4621
0,4518
0.5320
0.4380
CLIPIQA T
0,5910
0,5554
0,5508
0.5157
1h222
0.6759
0,7170
0.7167
0,7437
MUSIQU 7
67.65
66.12
65,80
61.26
70.63
66.98
68.92
72.14
21.62
Real200
MANIQA
0.3861
0.3921
0,4713
0.4869
0,5588
TOPIQ
0,5966
0.5530
0,5478
0,4793
0.6517
0.6124
0.6235
0,7142
0,7176
NTE 1
200
15
50
50
#
16,70M
16.70M
16.70M
8.17M
173.9M
1716.7M
2283.7M
171666
Lasency/Image 4
0.06s
0.08s
0.08s
0.04s
18.90s
1.12s
5.85s
7,24s
1.03s
pipeline from RealESRGAN [46]. For the network architecture, we employ the LAControlNet (7 with SD 2.1 -base as the pretrained T2I model. In cases where LR images are severely degraded, potentially leading to the diffusion model mistaking degradation for semantic content, we implement RealESRNet [46] as a preprocessing step. This ensures our source domain consists of preprocessed LR images, thereby refining the input quality for better model training and performance. The model fine-tuned for 50k iterations using the Adam optimizer (20). with a batch size of 32 and learning rate set to 5 10-5, on 512 512 resolution images.
For testing, we evaluate our method on both synthetic and real-world datasets, employing the same configuration as Sta For synthetic data, we randomly crop 3K patches with a resolution of 512 x 512 from the DIV2K validation set I. and degrade them following the degradation pipeline of RealESRGAN [46]. For real-world datasets, we generate LR images with resolution of 128 x 128 by center-cropping on RealSR [4), DRealSR (50) and RealLR200 (11)
4.2 Comparisons with State-of-the-Arts
We compare DoSSR with the state-of-thh-art real-world SR methods, including BSRGAN [54] Real- ESRGAN [46), LDL. (33, DASR (24), StableSR [45), ResShift (53), DiffBIR (27). and SeeSR (51]
We use the publicly available codes and pretrained models facilitate fair comparisons.
Quantitative Comparison. We show the quantitative comparison on the four synthetic and real- world datasets Table To comprehensively evaluate the performance of various methods, we utilize the following metrics for quantitative comparison: reference-busee metries PSNR, SSIM 49
LPIPS [55), and non-reference metrics CLIPIQA [44], MUSIQ [19]. MANIQA 52). TOPIQ 5
2 http:////innniniiteeooaalaaaiair http.///a.liiiii//aall//iiiiiioo 2-1-base
4 hhtps:/huggingface https///wggnngeeeccooo/aaatsctte cwldataset/lccelaa//Saaieees Testtees
We use the repository available http://ggitbbbocom///caattnngggp
7

---

Zoomed
B5KGAW
RealESKGAN
LDL
DASR
LR
StablesR
ResShife
DITBIR
D6SSR
Zoomed LR
8SRGAN
RealESRGAN
LDL
DASR
LR
Stable5R-200
ResShift
DITBIR-50
SecSR.50
DoSSR(ours].5
Figure 3: Qualitative comparisons of different steps of our DoSSR and other diffusion- -based SR methods. The "-N" suffix denotes inference steps. Please zoom in for a better view.
Notably, DoSSR consistently achieves the highest scores in CLIPIQA, MANIQA, and TOPIQ, with the exception of being second TOPIQ DIV2K, and attains the second highest score in MUSIQ across all four datasets. At the same time, we also note that diffusion- based methods generally achieve poorer performance reference metrics compared GAN-based methods due to their ability to generate more realistic details at the expense of fidelity. Additionally, our DoSSR manages to achieve improved no-reference metric performance compared the data presented in Table as NFE increases slightly, detail further elaborated on in Section
Qualitative Comparison. Figs. [b), 3 present visual comparisons on real-world images. By leveraging learning of domain shift and introducing DoSG, our DoSSR efficiently generates high- quality texture details consistent with contents of the LR image. In the example of Fig. lb), GAN-based methods fail to faithfully reconstruct the grid texture of clothing, leading to notable degradation. StableSR and ResShift produce specific erroneous textures. Both SeeSR and ours successfully restore correct textures, while our results display clearer textures. Similarly, in the first example of Fig. 3 our DoSSR generates more perceptually convincing Spider-
Spider-Man Man face as well as textures, while in the second example, it produces more realistic and high-quality details of ground-lan bricks compared other methods. More visual examples are provided in Fig. 7
Efficiency Comparison. The comparative analysis of model parameters and latency for competing SR models is shown Fig. a and Table The lateney calculated on the x4 SR task for 128x 128 LR images with V100 GPU. StableSR, DifiBIR, SeeSR, and our DoSSR utilize the pretrained SD model, resulting
similar parameter count, with SeeSR incorporating a prompt extractor to enhance SR results, making it the largest among these methods. ResShift, utilizing the network structure from LDM [35). is trained from scratch and has significantly fewer parameters. It employs a 15- step process to achieve faster inference speeds. Among the pretrained SD-based methods, DoSSR demonstrates superior performance efficiency, requiring only function evaluations to achieve speeds 5-7 times faster than previous SDD-based models such as SeeSR. Additionally, DoSSR not only demonstrates faster or comparable latency ResShift but also achieves significantly better super-resolution performance.

---

0.80
75
SSIK
71.10
0.75
0.7369
SSIM
0.70
7000
MUSIQ
Zoomed LR
LDL
SeeSR-10 -00
SeeSR-50
64.37
5
0.6495
0.6377
3
Step
NoSsR-onester
DoSSR-3
DoSSR-5
DOSSR-7
(a)
Figure 4: (a) Quality metrics vs. steps on RealSR Dataset. (b) Qualitative comparisons of different steps of our DoSSR with other methods. The suffix "-N" appended the method name indicates the number inference steps. Please zoom in for a better view.
4.3 Ablation Study
Effectiveness of DoSG. To verify the effectiveness of the DoSG introduced in the diffusion equation, we conduct an experiment using identical network architectures but with two different diffusion equations: the original diffusion equation as described by Ho el [16 and our newly formulated equation (Eq. (2. To isolate the impact of DoSG from our shifting sequence design, we set 1 = T, ensuring that the starting point of our inference in both scenarios approximates Gaussian noise. Quantitative comparisons can be found in the first two rows of Table 2 It is evident that the introduction of DoSG leads to a significant improvement across highlighting the effectiveness of DoSG in enhancing the performance of inffusion-baseed SR models. Additionally, it is worth noting that the original diffusion equation can be considered special case within our framework where 7 0 and t T. Therefore, our sampler original diffusion equation, and for a fair comparison, we employ the same sampler for both models. More comprehensive comparison is provided in Appendix Table where it can be seen that DoSSR demonstrates superior performance compared to the corresponding order solver with DDPM, benefiting from the inclusion of DoSG in DoS SDE-Solver.
metrics in table,
accommodate the
The selection of tj. The strating point t serves as a pivotal parameter in DoSSR. We explore several options the value of t and show the corresponding final SR per- formance in Table It can be observed that SR performance improves as 4 gradually decreases from 7 2.. However, further decreasing 4 from T/2 to 3/T conversely compromises SR performance. Intuitively, a larger & means less LR prior is preserved due a larger magnitude of added noises, and the model behaves more like the vanilla pretrained model by hallucinating plausi- ble HR contents; In contrast, smaller t means less noises, so the prediction is prone to be more consistent with the LR image, without HR details. Hence, we set 4 T/2 by default for good trade-off.
Method
CLIPIQAT
MUSIQU
MANIQAT
TOPIQU
DDPM 16
0.5379
54.09
0.3932
0.5180
0.5427
Domain Shift
27/3
0.6337
59,30
0.4589
0.5987
Diffusiom 4g
T/2
0.5214
7/3
0.6490
61.76
0.4895
0.6260
Table 2: Comparison across various selections of start- ing point f,, evaluated on the DRealSR dataset. The baseline method is DDPM, which employs the original diffusion equation. In all setups, inference is carried out over 5 steps.
The number of step. We assess the impact of different inference steps on DoSSR by analyzing changes in representative metrics for both reference- based and non-reference-bant based evaluations, as shown in Fig. [a). As the number of inference steps increases, reference-basee metrics tend to decline, suggesting loss in fidelity, while non-reference metrics improve, indicating enhanced realism and detail in the generated images. We also conduct visual comparisons in Fig. bb. Our DoSSR achieves performance comparable to SeeSR just steps and produces more realistic details in 7 steps. Remarkably, DoSSR is capable of delivering satisfactory results with just

---

single step, achieving 0.5115 MANIQA score and 0.6258 CLIPIQA score on the RealSR dataset, significandly boosting the efficiency of diffusion- -aased methods. More visual examples are provided in Fig. where it can be observed that increasing the number of steps yields more realistic details.
The order of our sampler. We provide suite of solvers for sampling in our DoSSR model, including
Order
CLIPIQAT
MUSIQU
MANIQA1
TOPIQ
first-order solver presented in Eq. [T]) and more ad-
0.5907
59.12
0.4686
0.5907
vanced second- and third-order solvers detailed in Ap- pendix | We investigate the impact of samplers with different orders on our experimental results through qualitative and quantitative comparisons, as illustrated in Table 3 and Fig. T0 From Table 3 it becomes ev- ident that high-order samplers can achieve superior non-reference metrics under the same limited inference step conditions. This is because the acceleration of higher-order samplers allows diffusion models to generate more details, as demonstrated in the first example of Fig. I0 where the tower generated by the high-order sampler exhibits richer textures. More comprehensive comparison is provided in Appendix Table 6 In our implementation, we use third-ordes sampler by default.
0.6749
64.09
0.5196
0.6571
0.6776
64.40
0.5214
0.6618
Table 3: Comparison of performance of different sampler orders on the DRealSR dataset. In all setups, inference is carried out over 5 steps.
5
Conclusion
In this paper, we present DoSSR, diffusion-based super-resolution framework that significantly enhances both efficiency and performance by integrating a domain shift strategy with pretrained diffusion models. This approach not only enhances generative capacity but also enhances further inference efficiency through our novel proposed DoS-SDEs formulation and customized solvers. Empirical validation on diverse SR benchmarks confirms that DoSSR achieves a 5-7 times speed improvement over existing methods, setting new state-of-the-art. Our work paves the way for more efficient diffusion- based solutions in image super-resolution.
Limitation. Despite the strong overall performance demonstrated by the proposed DoSSR, occasionally generates visually unfriendly details when employing an unfavorable random seed, a challenge also encountered by other diffusion- methods. Typically, we fix the random seed for all image super-resolution tasks to stabilize the results, but this particular seed may not be suitable for certain specific images. As depicted in Fig- [ different initializations of random seeds result in significant variations in the details of the lion's eyes. Some of the initialized random seeds produce eyes that are reasonable and acceptable, while others exhibit noticeable inconsistencies with LR. For bad cases, we can also obtain satisfactory result by adjusting the random seed multiple times. However, this often requires numerous attempts, and the quality of the results heavily relies on luck. This inspires to find suitable initialization for each specific LR image, which can enhance the performance the model. Hence, for diffusion baasedd methods, exploring how to obtain a reasonable random seed based known LR images may be a future research direction.
Societal impact. Our advancements in the diffusion-based image super-resolution model, DoSSR, present both positive and negative societal impacts. On the positive side, it enhances medical imaging, potentially leading to more accurate diagnoses and reducing the need for invasive procedures. In surveillance, it aids in better identification and tracking, improving public safety. Moreover, in remote sensing and environmental monitoring, it facilitates informed peaiiinn-making for disaster management and environmental conservation. However, there are concerns regarding privacy and surveillance. Enhanced resolution capabilities could infringe upon privacy rights and lead increased surveillance in public spaces, raising questions about civil liberties. Additionally, in digital media, while high-resolution imagery enhances visual content, may perpetuate unrealistic beauty standards and digital manipulation, impacting self-esteem. In summary, while DoSSR brings promising advancements, it's crucial to address concerns around privacy, security, and digital ethics to ensure responsible and ethical deployment of the technology.
10

---

References
[1] Eirikur Agustsson and Radu Timofte. Núre 2017 challenge on single image super-resolution: Dataset and study. Proceedings of the IEEE conference on computer vision and puttern recognition workshops. pages 126-135, 2017.
[2] Brian DO Anderson. Reverse- time diffusion equation models. Stochastic Processes and their Applications. 12333313--22,, 1982.
[3] Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. Advances in Neural Information Processing Systems, 36, 2024.
[4] Jianrui Cai, Hui Zeng, Hongwei Yong. Zisheng Cao, and Lei Zhang. Toward real-world single image super- resolution:::: A new benchmark and new model. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3086-3095, 2019.
[5] Chaofeng Chen, Jiadi Mo, Jingwen Hou, Haoning Wu, Liang Liao, Wenxiu Sun, Qiong Yan, and Weisi Lin. Topiq: A top-down approach from semantics to distortions for image quality assessment. IEEE Transactions on Image Processing, 33:2404-2418, 2024.
[6] Xiangyu Chen. Xintao Wang. Jiantao Zhou, Yu Qiao, and Chao Dong. Activating more pixels in image super-resolution transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 22367-22377, 2023.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong. Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.02%0 2021.
[8] Hyungjin Chung. Byeongsu Sim, and Jong Chul Ye. Come-closer-diflss-
Accelerating conditional diffuusio models for inverse problems through stochastic contraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12413-12422. 2022.
[9] Qinpeng Cui, Xinyi Zhang, Zongging Lu, and Qingmin Liao. Elucidating the solution space of extended reverse-time sde for diffusion models. arXiv preprint ar 2x31v:2309900//11 1619,, 2023. 2023.
[10] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang. Second-order attention network for single image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogmition, pages 1065-11074, 2019.
[11] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. Tece, 2009.
[12] Chao Dong, Chen Change Loy, Kaiming Hc, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. In Computer Visi 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part IV 13, pages 184-199. Springer, 2014.
[13] Dario Fuoli, Luc Van Gool, and Radu Timofte. Fourier space losses for efficient perceptual image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2360-2369, 2021.
[14] Martin Gonzalez, Nelson Fernandez Pinto, Thuy Tran, Hatem Hajri, Nader Masmoudi, et al. Seeds: Exponential sde solvers for fast high-quality sampling from diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[15] Shuhang Gu. Andreas Lugmayr, Martin Danelljan, Manuel Fritsche. Julien Lamour. and Radu Timofte. Div8k: Diverse 8k resolution image dataset. In 2019 JEEE/CVF International Conference Computer Vision Workshop (ICCVW), pages 3512-3516. IEEE, 2019.
[16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[17] Huaibo Huang, Ran He. Zhenan Sun, and Tieniu Tan. Wavelet-smet: A wavelet-based enn for multi-scale face super resolution. In Proceedings of the IEEE international conference on computer vision, pages 1689-1697, 2017.
[18] Babjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems, 35:23593-23666 2022.

---

[19] Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5148-5157, 2021.
[20] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: /4126980, 2014.
[21] Haoran Li, Long Ma, Yong Liao, Lechao Cheng, Yanbin Hao, and Pengyuan Zhou, 3d-goi: 3d gan omni-inversion for multifaceted and multi-object editing. arXiv preprint arXiv:2311. 2023.
[22] Haoran Li, Haolin Shi, Wenli Zhang. Wenjun Wu, Yong Liao, Lin Wang, Lik- hang Lee, and Pengyuan Zhou. Dreamscene: 3d gaussian- based ext-to-3d 3d scene generation via formation pattern sampling, arXiv preprint arXiv:2444.0 03575, 2024.
[23] Jie Liang, Hui Zeng, and Lei Zhang. Details artifacts: A locally discriminative learning approach to realistic image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5657-5666, 2022.
[24] Liang, Hui Zeng, and Lei Zhang, Efficient and degradation-adaplis network worldworld image super-resolution. In European Conference on Computer Vision, pages 574-591. Springer, 2022.
[25] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1833-1844, 2021.
[26] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In Proceedingy of the IEEE conference on computer vision and pattern recognition workshops, pages 136-144, 2017.
[27] Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Ben Fei, Bo Dai, Wanli Ouyang, Yu Qiao, and Chao Dong. Diffhir: Towards blind image restoration with generative diffusion prior. arXiv preprint ar3iv:2308. 1,5070, 2023.
[28] Anran Liu. Yihao Liu, Jinjin Gu, Yu Qiao, and Chao Dong. Blind image super-resolution: A survey and beyond. IEEE transactions on pattern analysis and machine intelligence, 45(5):5461-544 2022.
[29] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference Learning Representations (ICLR), 2023.
[30] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances Neural Information Processing Systems, 35:5775 357777 2022.
[31] Cheng Lu,, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01000 2022.
[32] Simian Luo, Yigin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference, 2023.
[33] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffuusion probabilistic models. In International Conference on Machine Learning, pages 8162-81111 PMLR, 2021.
[34] Hannes Risken and Hannes Risken. Fokker-planck equation. Springer, 1996.
[35] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 10684-10695, 2022.
[36] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):471334722, 2022.
[37] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311.17042, 2023.
[38] Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
12

---

[39] Yang Song, Sahaj Garg. Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pages 574-584. PMLR, 2020.
[40] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[41] Stability.ai. htttss:/sstailittyy.
[42] Haoze Sun, Wenbo Li, Jianzhuang I. Liu, Haoyu Chen, Renjing Pei, Xueyi Zou, Youliang Yan, and Yujiu Yang. Coser: Bridging image and language for cognitive super-ress reperrreooution. In Proceedings of the IEEE conference on compuder vision and pattern recognition. 2024.
[43] Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming- Hsuan Yang, and Lei Zhang, Ntire 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE conference on computer vision and puttern recognition workshops, pages 114-125, 2017.
[44] Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 2555-2563, 2023.
[45] Jianyi Wang. Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image resoluteso arXiv preprint arXiv:2305.07015 2023.
[46] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real- wollw blind super- resolution with pure synthetic data. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1905-1914, 2021.
[47] Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Recovering realistic texture in image super- resolution by deep spatial fcature transform. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 606-615, 2018.
[48] Yufei Wang, Wenhan Yang. Xinyuan Chen, Yaohui Wang. Lanqing Guo, Lap-Pui Chau, Ziwei Liu, Yu Qiao, Alex C Kot, and Bihan Wen. Sinsr: Diffusion- based image super-resolution in single step. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2024.
[49] Zhou Wang. Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
[50] Pengxu Wei, Ziwei Xie, Hannan Lu, Zongyuan Zhan, Qixiang Ye, Wangmeng Zuo, and Liang Lin. Component divide-aand-oonuuer for real-world image super-resolution. In Computer Vision-ECCV 2020: 16th Furopean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII 16, pages 101-117. Springer, 2020.
[51] Rongyuan Wu, Tao Yang, Lingchen Sun, Zhengqiang Zhang, Shuai Li, and Lei Zhang. Seesr: Towards semantics-awee aware real-world image super-rosolution. In Proceedings the IEEE conference on computer vision and pattern recognition, 2024.
[52] Sidi Yang, Tianhe Wu, Shuwei Shi. Shanshan Lao. Yuan Gong, Mingdeng Cao, Jiahao Wang, and Yujiu Yang, Maniqa: Multi-dimension attention network for no-reference image quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1191-1200, 2022.
[53] Zongsheng Yue, Jianyi Wang, and Chen Change Loy. Resshift: Efficient diffusion model for image super-resolution by residual shifting. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[54] Kai Zhang, Jingyun Liang, Luc Van Gool, and Radu Timofte. Designing a practical degradation model for deep blind image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4791 4791-4800, 2021.
[55] Richard Zhang, Phillip Isola, Alexci Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[56] Xinyi Zhang, Qinpeng Cui, Qiqi Bao, Wenming Yang, and Qingmin Liao. Geometry- guided diffusson model with masked transformer for robust multi-view 3d human pose estimation. In ACM Multimedia 2024, 2024.
13

---

[57] Yulun Zhang. Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super-resolution using very deep residual channel attention networks. Proceedings of the European conference on computer vision (ECCV), pages 286-301, 2018.
[58] Yulun Zhang. Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2472-2481, 2018.
[59] Yixuan Zhu, Wenliang Zhao, enhancement via rectified flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13 13-22, 2024.
Yansong Tang, Jie Zhou, and Jiwen Lu. Flowie: Efficient image
14

---

A Mathematical Details
A.1 Derivation of Eq.(3)
According to Eq. 2 we can express 2t as a linear combination of æo. 2o and noise variable e: 3t - az (m,do + (1 - 7h ()) + o,6, where (),
(13)
Subsequently, the relationship between at; and - derived as follows:
3t ar (11- + (1 10 + (7= 7 -x0) OrE
af at
( + 7=-1)=0)) + 01-1E1 aar(m 1)-0) + of 0 12
at-l
at --
Ott
-1-1 + ac(Tc 77-1))0 of
a;-1
0
where En Eg - Un. and E, E E2 can be further simplified as follows:
W00, I). Taking into account aj + of 1, the above equation
2
at
#t-1 + an 77-1))0 + 1
E2.
(14)
at11
a
Hence, the transition distribution between 2, and 2;-1 is as follows:
g(2,|2,-1, = N(xx;
Oit
- + at(1) 7--1)00
at
I). t=
(15)
ar-1
A.2 Derivation of (q.(6)
In this section, we derive the coefficients of the forward SDE. Discretizing Eq. (5 yields:
Tt At 3t =
+
where 21 ~ ((0
E++ (f(t) h(t +
(16)
Substituting Eq.
13 into Eq.
16
we have
B:+Ar (f(t) + (n:
= =ailf(t)At+ + 1) - m + nh + 0: + (tt)
(17)
+ (f(tt) + 1)20, 2t))2
where 22,E~ ~ NO, I). For Eq. 13 at time + Al, we have:
E++ = 044s:( To + (1 T142))20)) + OctACE.
(18)
Equating the corresponding parts of Eq. (17) and Eq. 18 yields:
artar M+++) (f)((t) 1)(1 - 7c)
a++AM++a ann + +) h(t)
(19)
2 (s(()// + 1J²03 + =())
Then, letting At + 0, the aforementioned three equations can be solved separately to yield:
f(t) = d a1 mp)
dt
at d1j
h(t) I - Ttt dt
(20)
do?
dl
dogog
ar (1 n) of dt
g(t) V dt
15

---

A.3 Derivation of Eq.(7)
As outlined in Sec. 3.2 the forward process can be expressed as the SDE shown in Eq. (5) In accordance with the Fokker- Plank Equation [4], we obtain:
(xe( Ot
20000,2220m00 (t)(((()))
+
0x,0x,
+
V 0 2
(t) V[(((a))
= V((lf(())z+++ V
= (z) Va log q (2,)
+() - (t)
log qi az (x)))
where ((x;) denotes the probability density function of state Most process defined by forward- time or conventional diffusion equation model possess a corresponding reverse-time model (2), which can be formulated as:
dæ, ju(t, xz xt + (t, xt duw
(21)
According to the backward Fokker-Plank Equation [34. we have:
(xx) V
0
ar,,0r (1, (x))
Øt
V=lu((,,z))u((3r)) Vn 10 (1, xe Vanet (xx))
V=lm((,,2:)90(10; V 1 (1, Wc )r (x: V log (xx))
-V=((u(t, at) + (t, æz - log ge ((
2
Our goal for the reverse process have the same distribution as the forward process, specifically:
p(t, xx) 2 o (t, Ve Vo log (xx)) f(t)2; + h(t)a0 5 (t) (x;)
(22)
Typically, set ot,, x,) g(t) 40 yielding:
(tt, x.) = f())x + h(t)ag - ( log qe (xx))
(23)
Therefore, the reverse- reverse-time SDE can be expressed as follows:
dx;
f(t)x; h(t) g² V log (z;) dt g(t)
(24)
A.4 Derivation of Eq.(9)
The data prediction model my (ar; Do 1) directly estimates the original target data 2n from the noisy samples, indicating that æe (x,, Lo, x Φo- Based on Eq. (2). the expression for gx(x)) can be formulated as follows:
Fr (ar (1 n n2)o0 + as
ge (ax))
exp(
(25)
V2TO,
20
Hence, score function is:
V log (x())) = x2 (az 7t 30 + amm))))
(26)
of
Substituting acp (x,, 2o 1) Fo, establish the relationship between the score function and the data prediction model:
- (ax 17e mx) (z,, Lu. t) + au7))))
V x log q((x))
(27)
of
Furthermore, Eq. shows that the noise prediction model is estimate
(x Lo. t) x -a v log q, (xx). Hence, the relationship between the noise prediction model and the data prediction model is:
(28)
cØ (It, Et. Øt (ax 1/, )ay (x:, tt t) + a.))
(29)
Tt indicating that casily trained noise prediction model for data prediction through the equation.
16

---

A.5 Proof of Proposition 3.1
In this section, we derive the solution the equation Eq. 77. into Eq. 77. we obtain:
By substituting Eq.
and Eq. (0
dlau(1 - -1t:]]
at dit
dx,
ax
+
aa1(1 Th.)
dt
1 f/t dt
20: do:
-20, adla((1-1) mi]l
2t
(as
mi l (ze, $o + amn))oo jat (30)
dt
dt
oī
+ dož log an 11 n1)
drw
dt
Combining like terms, we get:
dx, 2 do.
dlar (1 113]
0, dt
a, 77))
dt
+ {
dn
2 do.
2 dla((1-ml) -mml] 2
auT
T)t dt
ot dt
ar(1 11)
dt
(31)
1 dō
1
d(ar(1 - mill [mm)]]]]]]]
20, (1 71))
1o 20, 1)
os dt
ar(1-1)
dt
d dlog
an(1 - m1) of
+ dt
dt
Subsequently, dividing both sides by ac(1 n) simultaneously, we have:
1
2 do:
dlox(1 - 1/)]
St
da
ar(1
11)
oe dt
a4 71)
dt
04 1 71)
1
dn 1/1 do,
2
dlor(1 (1 mn)]
20
1122 dl
(1 1p))
dl
aie1 71)
di
do,
1
d d[ar(1 (1 Th]]
(32)
2 dl
Vp (e.. 20011)
0ra(1 - m1)
dt
20. do. 20
d(cr,() - m1e] dww
[a m dt
[ar(1 n)]s
dt
de Let A
mx) Then A is monotonically increasing, and we have:
dd,
1 da:
az dlar(1 -n)]
(33)
dt
ar - 11)) dt
[0+ - 11)12
dt
Therefore, have:
1 do:
dlar(1 (1 ne) dd,
(34)
de dt By performing the variable substitution y: ar(1-m) and then substituting Eq. (34) into Eq. 32
o2 dt
a, - n))
dt
m
we can simplify to obtain:
dy
dd, yet (1-11)) dt 1 Mt dd
dnj.
2o A, dt x xo(x; (1,, 20, 1) dtt y 22x dt
2 dd,
dd,
dw..
A dt
de dt
(35)
Denoting dw>,
V
dhe di
die == x(()), Wt = wx.+ rewrite the equation above w.n. À as
2 dys ス Axdd+ + [1 (1-may nA) dna 1 m A
17x di 2o
xe <2 do A)dA + v 22ddux.
(36)
Utilizing the variation-of--o offooontnn---ontatts formula to solve the equation above, we obtain
yt = elli &dl Edly, +
//m c 2dr
1
?
(1- m=) dns - 1- n X
dA to
(37)
e.l 2dr 2
2o Co, 2)d1 L.m e Edr 22ndux.
17

---

Simplifying and substituting back 2, = as(1 17: 17:yy,, we obtain
a+(1 1/))
X
U
1/s 2
xt a,(1 1 - 17s) X
T + ar (1- 17t)
201
1- 1 - f7 2
(38)
Ae 2X
x
0r. - 77:))
X Dg (xx, to AldA + ar(1 - m1)1
Z
x2
A
Thus, we obtain the exact solution to the DoS-SDEs.
A.6 Derivation of Solvers for Diffusion DoS-SDEs
Denote 2g k 1, the 1-th order Itô Taylor expansion of xy (2x Do. A) w.r.t À at s
(x)) (æx, Lo. A) == (2t,,_t)_ as the n-th order total derivative of x2(2xx A) w.r. A. For
dA
(A- mo (æ Lo, 8) + RR:
no (x,, Co A)
(39)
n!
==
where the residual R2 comprises of deterministic iterated integrals of length greater than & and all iterated with at least one stochastic component.
Substituting the above Itô-Tayloy expansion into Eq.
38 yields
(1 m/ X
xs + ac(1 -no)
1/t
1/s X?
zt ax(1 170)
1-17
1 1 17s X3
k-1 1
d 2x (A As)
X
- -0+(1 - 11h))
(xs, 20, 8) 0
dd + az -W
X3
X 2a +
Re,
13 n!
n=U
(40)
where R₂ can be easily obtained from R₂ and the integral Ja 2X (A-2a)"
computed by repeated applying n times of integration-by-patte X n! dd can be analytically By dropping the R2 eπтоr and approximating the first 1-th total derivatives with forward differential method, we can derive k-th order SDE solvers for diffusion DoS-SDEs. In fact, it is inaccurate to call it "order" when 2, because the proposed algorithm has global error of least O() X₂) [4]. Thus, only when k = 1, it is referred to first-order solver with strong convergence guarantee, stated in [4]. Nevertheless, for practical convenience, we still refer to this approximation as k-th order. Here present the expressions for first-order as well as second and third-order solvers. We name such solvers as DoS-SDE Solver overall, and DoS-SDE Solver-k for a specific order k.
DoS-SDE Solver-1
When k = 1, the integral becomes
2X?
A3 Io (x A)dA X2 // ddxoo (I., Lo s) = =(1-
X? xz (x., Lo. s).
(41)
X3
Substituting into Eq. 38 we obtain first-order solver for DoS-SDEs
ae(1 776) X₂
X?
x. + ac(1 n1){ 1tt
n7s
x a,(1 - n.) X2
17
7o
X2
(42)
X²
A
+ + - (1)
og (x, &o. 8) + aa(1 me
X3
X?
2
8'
DoS-SDE Solver-2
When 1: the integral in Eq. 40
becomes
A
2X (A Aa)=
Σ
(n) (x,,
8) 8)
dA
A3
n!
n=0
2X2 dAos (Ess 200 8)
Ae 2x
() --xa)dda (a., 8)
(43)
A3
(1
22
(20, 20. 8)
():
Xa)
I (x,,, L0. 8)
(1)
X?
X
18

---

Substituting into Eq. 40),
we obtain 2-th order solver for DDS-
zx a,(1 17,) X²
a1(l 17)) X7
v + az(1 -mt}{ -mo) 71 17s
A
a, 11))
22
2
1/.
173 X
(44)
(Ar - 220) (1) A
+ a1 - m1)((1
)ao (xs: 20. 8) a0(1 71)
Big
(ar,, 8)
23
where (1) (x,, Lu s) can be estimated by forward differential method. We have
(I,, Lo, 8) ~ De ( to. 8) Dg (Tr,, $o, (1)
(45)
X Xr
where time < 8 < r and as (Zr, To. r) represents the output of the network at the previous time step.
DoS-SDE Solver-3 Samely, when = 3, the integral (4.) becomes
(n)
( To. 8) /
2X
(A
Am)a
dA
A3
n!
===
22
(Me x,)2 (1)
A1 A?
(1
jz (Ie, s)
Fg
(2,1 k0, 8)
A3 (1 dd) (x,;
X
X
(1
X2
Jzog
(Ie: |a
8)
(de 2aa²²
I [5s t. s)
(1)
X
A
(xg () -11)
Xb2n
de
(2)
(Tes To. 8)
(46)
Substituting into Eq. 40), we obtain 3-th order solver for DoS-SDEs
x a.(1 m)) X?
vs + a0(1-1m)ll
-mo)(
ne 17s
2o
ar n1) mn)
X?
A
x2 2
a,(1 - -n/) X2
1)
X
+ a1(1 (1
X?
x (x:: æu. - acc(1 m1)
(de
2x02
Ig
(1)
(I s)
(47)
X2
(Ag 3At) -Xr)
de (2) >2In(
+ a1 - mm)
(x.,, aro,
2
where (1) (x,, 20, 3) and 2g (x., Do. s) can be estimated by forward differential method. We have
(I., Zp. s) No (x'; Lo, 8) DP (Zrr Lo,
x
(48)
A Ay
where time and we (ar, To. represents the output of the network the previous time step. And
xgg [I. ty. s) Zzg (2 to. 8) ar (2,, (x,;,
(2)
(49)
h.-da
where time < q and a (æ,, 8) and ar (e,, respectively represent the approximations of the first-order derivatives at the current and previous steps.
Detailed algorithms for our solvers are proposed in Sec. B
A.7 Comparative Analysis of DoSSR and ResShift
Previous work has introduced method called ResShift 53 Markov chain in the diffusion process through residual shifting to achieve efficient super-resolution in diffusion models. this section, we theoretically elaborate on the similarities and differences between our method and ResShift.
which shortens the length of the
19

---

ResShift expresses the forward diffusion process in the form of residual shifting, as shown in the following equation:
(x2; mo yo) (2 To neCo, me t == 1,2,- ,T,
(50)
where yo Lo respensents the residual between the LR image yo and the HR image zo. Hence, can be rewrite as,
(20 yo) (a nryo (1 7e) 1fT,,,,, t 1,2, 1,2,...,,1 T.
(51)
Therefore, essentially, the ResShift concept also represents a linear combination of the source domain and the target domain. However, the crucial factor lies in our design of the diffusion equation, which determines whether we can effectively utilize the diffusion prior. This is our most significant differentiating point. Currently, the mainstream approach to leveraging diffusion prior involves fine-tuning large-scale pretrained diffusion models to achieve SR. As we all know, Stable Diffusion, typical representative of large-scale diffusion models, employs the diffusion equation of DDPM [66] Its forward process can be expressed as:
<( mo) = N (m o0 T) 1 1,2, ..2.
(52)
where at, oz 2 0 and a + = 1, referred to as noise schedule. If we consider y0 + (1 - T/p 200
as whole, we can intuitively observe that Eq. (51) lacks decay coefficient at compared Eq. (52)
Therefore, applying Eq. (5T) to fine-tune pretrained Stable Diffusion model poses significant challenges. The equation proposed by our DoSSR is restated as follows:
q(0;|2o, .b) (Nx am (1
t = 1.2......7 ,T,
(53)
Our diffusion equation incorporates the noise schedule of Stable Diffusion, enabling seamless compatibility with existing DDPM-type diffusion models. Therefore, in this sense, our method is specifically designed as diffusion equation tailored to adapt to Stable Diffusion.
In fact, Eq. (51) and Eq. (52) (or Eq. (53) belong two different types of diffusion models, which correspond to the discretizations of two types of SDEs VE SDE and VP SDE) 44). respectively.
The diffusion equation that satisfies the discretizations of VE SDEs typically takes the following form:
( (zn) N( oo, If t t 1,2,....,7 ,,
(54)
where ot increases with t. and is typically a very large number, ensuring that ((á)) = N( I) = ((; I). Therefore, Eq. (5I) can be considered as a variant of the above equation. By setting hyperparameters such that R. =of, it appears possible to fine-tune a diffusion model pretrained with Eq. (54) as the diffusion equation, thereby leveraging the diffusion prior. However, this contradicts its original intention, as the starting point of inference almost approximates Gaussian noise, retaining little of the prior information of LR. Our design of shifting sequence {nt) is aimed at addressing this issue, which the ResShift lacks.
The second type of diffusion equation that satisfies the discretizations of VP SDEs is of the DDPM type. In Eq. (52). the noise schedule satisfies the noise schedule 0 < ar < and it is typically set to or R 1, oT
to ensure that q(<)) (ar I) R A I). To my knowledge, most scage pretrained diffusion models, represented by Stable Diffusion, adopt the VP-type (DDPM-type) diffusion equation. Therefore, our design is highly significant. Moreover, our theory not limited to the analysis of discrete cases but extends to more general continuous cases, expressed as SDEs. Based on this, we have developed our fast samplers for our DoSSR. These are the distinctions between our work and ResShift.
To further demonstrate the effect of the diffusion prior, we conducted an additional comparative experiment with ResShift, as detailed in Appendix C
20

---

B Pseudocode
Here, algorithms for first, second, and third-order solvers for DoS-SDEs are presented follows.
Algorithm 1 DoSSR Solver-1.
Require: starting point 1,, used time steps (1+) nosie schedule oz and St preprocessed LR image 4oo data prediction model Zo.
1: Pt, f- Cte do + E
initial value
2: for 4- 2 to N do
3: DoSG + ar, (1 TT, I-11;
1-9/;
X2 7
100
> domain shift guidance
4: Linear Term + ore, (1-ne,)) 11
X2;
Iti-1
x²
5: Noise Term f- Oxt, [1t,)
X?
Xi,-1
xt
270
6: PAT Ott, (1 111)(1
)ao (ag Lo, 1i-1)
D Prediction Approximation Term
x7
7: Tt, Linear Term + DoSG
PAT + Noise Term
8: end for
9:
Return: Zt
Algorithm 2 DoSSR Solver-2.
Require: starting point 4, used time steps (4 noise schedule da and dt, preprocessed LR image .og data prediction model Fg.
1: Ct, t- Ot, do + 0
initial value
2: Q <- None
3: for + 2 to N do
4: DoSG + at,
(I-
m
X7
domain shift guidance
1-14,-1 X
5: Linear Term cie, (1-1),)
x?
Itt--1
01--((-1111
x/
X1
6:
Noise Term ar, (1, - 11,)
12
40
X
7:
if = None then
8:
PAT + a (1 )1.) x²
)m (It
Lo. ti-1)
9:
else
D 2a( @p.ts 1) (2 -, (0.14-2)
first order derivative
Ac-- Ao.-2
x7
)a:o (Ft Lo 4-11) 04,(1 mh: (de, d D;
11:
PAT + a1, (1 - m1
x2
12:
end
if
13:
Q
xo Lo Do. 1-11)
save last network output
14:
Ct.
Linear Term + DoSG PAT' + Noise Term
15:
end for
Return: Vtn
21

---

Algorithm 3 DoSSR Solver-3.
Require: starting point (1, used time steps (4: (4:) J
noise schedule oz and Øt preprocessed LR
image Zo, data prediction model Zzg.
2t f- at, 2o + o
initial value
2: Q - None, Qd f- None
3: for i 2 N do
4: DoSG t at, (1 T1,] mi
my
> domain shift guidance
I-m4
1-91;
(1-1))
x2
5: Linear Term f ae (2-1t,, -,,- 1)
-tt-1
x
Nt.
6: Noise Term +- ac, - (1)() )
X2
500
Xr_-1
7: if Q None and Qu N None then
X2
8:
PAT at,(1 (1 n1,((1 Xi
Ja (1) :o. 1i-1)
9:
else Q + None and Qd = None then
D, æe 20( .b.ti 2)- (2), 2o.4 2)
first order derivative
A4-1 -
(A,, Ac;-1 D;
11:
PAT an, (1 m1,
kop ( (
Lo, ti-1) at,(1 7h.
A1;-1
Qa D;
13:
else
D; (- 00( 20.ti ti 2010,
10.h
14:
first order derivative
Ae-- A
15:
U, t be -
D.-D.-1
D second order derivative
(1:; Ad; 1)² D
PAT t ar, -1,,)
Jz (1 20,t4-1) ar, (1 me.
dz 1
ar, (1 nc.)
(Ae; -3de)(0 -
Xo. Inl In(5re
do
U
17:
Qd - D:
end if
19:
Q + xn xo(xt Lo, 1
save last network output
St. Linear Term + DoSG + PAT + Noise Term
21: end for
Return: IiN
C Additional Experiment Results
C.1 Ablation on Diffusion Prior
To demonstrate the impact of the diffusion prior, we conduct a comparative experiment with the ResShift model (53) which does not leverage pretrained diffusion model. Considering that ResShift trained ImageNet [TT] while DoSSR model is trained commonly used datasets for super-resolution tasks (e.g., DIV2K (), to eliminate the influence the training dataset, we retrain our DoSSR model using ImageNet as well. To highlight the effect of the diffusion prior, we only utilize subset of ImageNet as our training data. This subset consists of randomly selected 10 images from each of the 1000 categories in ImageNet, totaling 10,000 images, as illustrated in T It can be observed that our DoSSR achieves superior metrics compared to ResShift despite utilizing significantly fewer training data and epochs for iteration. This indicates that leveraging the diffusion prior is highly beneficial for super-resolution tasks, even without utilizing commonly used high-resolution datasets for training, we can still achieve satisfactory results. We also perform qualitative comparisons between our retrained DoSSR and ResShift, as illustrated in Fig. 5 It is evident that the utilization of the diffusion prior significantly enhances the quality of the generated images, both in terms of fidelity and realism.
22

---

Table 4: Comparison of performance between retrained DoSSR and ResShift models the DRealSR dataset. For fair comparison, we employ our first-order sampler for inference, running it 15 times match ResShift's default setting.
Diffission
Training Setup
Evalation Metries
Method
Prior
Training Dataset
Num of Iters
Traing Method
SSIM
LPIPS
MUSIQ
MANIQA f
ResShift
ImageNet(-1280k)
500k
Train from scrutch
0,7629
0.4036
49.73
0.3322
DoSSR
sub-ImageNen((0k)
Finctune
0.7824
0.2943
53.94
0,3672
Table 5: Comparison of performance with other methods on the RealSRSet and RealSR datasets, NFE represents the number of function evaluations in the inference. involves retraining using the same training data and identical network architecture as our model.
RealSRSet
RealSR
Method
Training Dataset
NFE
MUSIQ
MANIQAT
MUSIQI
MANIQA
ColdDiff
DYVXK-DVVSK-FiiscXX--OTT-- 15k)
58.19
0.3194
47.42
0.2783
ResShift
DNV2K-DIVVSK-PPcccXXX--0077 15k)
63.90
0.4505
56.01
0.4001
DoSSR
DVVX+-DVa---ick224005T-- 15k)
73.35
0.6169
69.42
0.5781
S
FlowIE
ImageNet(~ 12800)
61.63
0.3611
56.51
0.3284
FlowIE*
DIVX<+DIV&&+FIidXXX440STT- 15k)
60.48
0.3644
50.82
0.3228
DOSSR
DVV2K4DVVKK4PicckXX4405557 15k)
69.42
0.5554
62.69
0.5115
C.2
Compare with other formulations of diffusion process
Aside from ResShift, we also compare our method with other plausible alternative formulations of diffusion processes, which encompass ColdDiff (3 and Rectified Flow [29]. ColdDiff is similar to DDPM, but differs by employing alternative degradation methods, such as blurring and masking, rather than additive Gaussian noise used in DDPM. In case for image super-resolusion, select the blur degradation. Note that ColdDiff is originally applied in the pixel space, while we have to implement in the latent space for fair comparison with our method. Rectified Flow defines the forward process as æ, = Lăn + (1 1) where æp is the data sample and ætp is Gaussian noise. In our case for image super-resolusion, we choose FlowIE (59) as an implementation of rectified flow which replaces 2o with the LR image as the starting point.
For fair comparison, we reimplemented the three methods- ResShift [53. ColdDiff 33 and FlowIE [59]-using the same network architecture, training dataset, and initialization as our method. The results shown in the T Results of ColdDiff is significantly worse than other methods. The main reason can be summerized as two aspects. First, applying bluring kernels to latent features equivalent applying them to raw images, while ColdDiff is originally designed for the latter. Second, the blurring kernel can only be designed in a hand-craft manner and may not represent the real-world degradation, The limilation of hand-crafted degradation is also well-known in many previous image restoration works. contrast, other methods, including our DoSSR, does not assume fixed degradation, so the performance is much better. Because FlowlE emphasizes single step evaluation, we follow its setting and compare DoSSR with 1-setp evaluation with It can be see that FlowIE is also not satisfactory and largely underperforms our DoSSR (-11.87% MUSIQ). This is mainly attributed to FlowlE's bad adaptation from the DDPM pretrained T2I model, since the learning objective of flow matching differs from score matching. By contrast, our design makes full use of the DDPM pretrained weights so performs much better. ResShift also significantly underperforms our method, similar to FlowIF, because its formulation does not account for adaptation from the pretrained diffusion prior, as discussed in Appendix A.7 and C.T To summarize, our formulation differs from other alternatives especially in terms of better leverage and adaptation from DDPM pretrained T2I models.
when tested with real LR images there would be domain gap.
and
C.3 Network Structure
The network structure of our model is illustrated in Fig. 6 In general, we adopt the same model architecture as in DifIBIR (27) using LR images as conditional inputs for the ControlNet module. The ControlNet initialized from the Stable Diffusion 2.1 Unet encoder and trained to generate HR images given LR inputs.
23

---

Table 6: Comparison of performance: w/o DoSG vs. different accelerated samplers on the RealSR and DRealSR datasets with same model(RealEESRNer preprocessing DifiBIR). In all setups, inference is carried out over steps.
ReolSR Dataset
DreolSR Dotaset
Method
Corr. Sampler
CLIPIQAT
MUSIQI
MANIQA
TOPIQT
CLIPIQA"
MUSIQI
MANIQAT
TOPIQT
DDIM(y 1) EDM
0.5176
$7.95
0.4293
0,5286
0.4732
48.22
0.3518
0.4316
w/o DoSG
0.5351
62.08
0.4445
0.5789
0.5341
53.13
0.3917
0.5082
(DDPM)
DPM-Solver+++ -3
0.5323
62.67
0,4384
0,5807
0,5379
54.09
0,3932
0,5180
DoS SDE Solver I
0.6874
66.55
0.5574
0.6588
0,5907
59.12
0,4686
0.5907
w/ DeSG
DoS SDE-Solver -2
0,7025
69.27
0.5794
0.6966
0.6749
64.09
0,5196
0.6571
(DoSSR)
Do5 SDE-Sollve
0.7025
69.42
0.5781
0.6985
0.6776
64.40
0.5214
0.6618
Image
DoSSR
Image
ResShift
Figure 5: Qualitative comparisons between our retrained DoSSR and ResShift. The utilization diffusion prior noticeably enhances the realism and visual appeal of the generated high-resolution images. Please zoom for a better view.
LR
HR
SD UNet
Proprocest
staps
2o
Zt
Zo
D
(2)
ControlNet
Trainable
Frozen
8 VAE encoder
VAE decader
Figure 6: The overall framework of DoSSR. During training, introduce noise facilitate the gradual transition from the HR to LR domain, integrating it with the standard diffusion process, and incorporate preprocessed LR as conditioning input for the denoising process, following the ControlNet approach. During inference, we add noise to LR latent according to Eq. (2) and perform inference starting from t
24

---

aid
Today
aid
aid
aid
aid
aid aid
young
community
RSRGAN
got
bette
fair Rep
har
aid
aid aid
aid
LR Image
Stable5R-200
ResShirt-15
DIMBIR-50
DoSSRjoursj-5
Zoomed LR
BSRGAN
Real ESRGAN
LDL
DASR
LR Image
Stable5R-200
Messhift-15
DITBIR-50
SeeSR-50
DoSSR
17
Zoomed LR
BSRGAN
RSaIEN
LOL
DASR
+
LR Image
Stable5R-200
ResShift-15
DirTBIR-50
SeeSR-50
DoSSRtours
Zoomed LR
BSRGAN
ReaIESRGAN
LDL
DASR
LR Image
Stable5R-200
ResShife11
DiTTBIR:50
SeeSR-50
DoSSR\ours)-
Zoomed LR
BSRGAN
ReaIESRGAN
LDL
DASR
LR Image
StaDRSR-200
Rasshii-15
SeRSR-50
DasSR 1-5
Figure 7: Qualitative comparisons of different steps of our DoSSR and other diffusion-basse SR methods. The suffix "-N" appended to the method name indicates the number of inference steps. Please zoom in for a better view.
25

---

LR Image
DifiBIR-50 (seed 512)
DiffBIR-50 (seed = 13)
SeeSR-50 (seed 12)
SeeSR-50 (seed 42)
ResShift-15 (seed 512)
ResShift-15 (seed = 42)
DoSSR-5 (seed = 512)
DoSSR-5 (seed 321)
Figure Visualizing the impact of random seed on diffusion-based methods. The "-N" suffix denotes inference steps. Please zoom in for a better view.
Zoomed LR
DOSSR-3
DOSSR-5
DoSSR-7 7
Figure 9: Qualitative comparisons of different inference steps of our DoSSR. The "-N" suffix denotes inference steps. Please zoom for a better view.
Zoomed LR
order=1, 1, step=5
order=1, step=10
order=22 step=5
order= step=5
Figure 10: Qualitative comparisons of different sampler orders of our DoSSR, Please zoom in for better view.
26

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract provides a concise summary of the main findings and contributions of the paper, while the introduction elaborates the problem statement and research objectives, thereby clarifying the contributions.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Appendix 5 we expound upon the limitations of the work conducted and provide brief discussion thereof.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
27

---

Answer: [Yes]
Justification: In Appendix [ we provide detailed mathematical derivations for the formulas appearing in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Section 4. we introduced the details of experimental setup and model training to ensure reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
28

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
The code will be released once the submission is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ccppp..cc/pulli///gg.c guwdes/ddeeeee CooeeSumeeieeeeeeeeeee ssi onyylicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: In Section we introduced the details of experimental setup and model training, and conducted ablation experiments in Section 4.3 to elucidate the selection of hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: The experiments conducted in our paper do not involve the use of error bars or statistical significance analysis, thus this aspect not applicable to our study.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: For the latency testing experiments, we furnished detailed specifications the GPU models used along with their corresponding tasks. Furthermore, we included specific information regarding the model training batch size and the number of training epochs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: principles.
We have carefully reviewed the NeurIPS Code of Ethics and adhere to its
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive societal impacts and negative societal impacts of the work performed in Appendix.
30

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymemt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators original owners of assets, such code, data, models, used in the paper, are properly credited. Additionally, the license and terms of use associated with these assets explicitly mentioned and respected in accordance with ethical and legal standards.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
31

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing research with human subjects.
Guidelines:
32

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "oMcp8nEBEP", "reviewer": "Reviewer_QMdK", "summary": "This paper presents DoSSR, a novel framework for diffusion-based image super-resolution that aims to balance efficiency and performance. A domain shift equation that integrates with existing diffusion models, allowing inference to start from low-resolution images while leveraging pretrained diffusion priors. A continuous formulation of the domain shift process using stochastic differential equations (SDEs), enabling fast and customized solvers.", "questions": "About the issue mentioned in Weaknesses, can we have some experiments to show the reason of doing such interpolation?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The domain shift equation and SDE formulation provide a solution and enables efficient inference.\n2. DoSSR outperforms existing methods on multiple benchmarks, both in terms of quality metrics and computational efficiency.", "weaknesses": "I have doubts about the interpolation between LR and HR. Theoretically, from the moment of interpolating, the image contains both the degradation of LR and the details of HR. For details, please refer to the article [Interpreting Super-Resolution Networks with Local Attribution Maps], which also mentions similar gradient and difference operations. If each image contains information about LR and HR, is it reasonable to use this for training? Based on the proposed method and the actual effect of LR&HR, I guess that using only LR or adding various noise/blur methods to LR may also achieve similar results. Since the topic of the paper is related to this approach, this concept is still important. In addition, skipping the first half of the diffusion model is a more intuitive method. I don’t know if there is a paper discussing this, but in practice this method seems to have appeared for some time (this will not affect my score much).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Regarding interpolation between LR and HR.**\n\n**A1:**  We want to clarify the necessity of interpolation between LR and HR from the following aspects.\n1. _**HR is necessary:** All diffusion-based SR method consturct training samples using HR images._ This is because that our objective is to recover the HR images, thus the training samples are constructed by add noises on HR images.  The forward diffusion process can be seen as linear combinations (more general than interpolation) of HR and Gaussian noises.\n\n2. _**LR is beneficial:** Using LR as starting point instead of Gaussian noise can shorten the diffusion path._ Imagine the original DDPM is to find a path from noise towards the HR domain, what we do by replacing noises with LR is to shorten the diffusion path by pushing forward the starting point from pure noise to the LR domain.\n\n3. _**Interpolation is explored in existing works:**_  In fact we are not the first to introduce the interpolation diffusion process. As pointed by Reviewer tfJj and dM7n, the interpolation form has been explored in Rectified Flow [1], FlowIE [2], and ResShift [3].  Results of each work show good performance, and more importantly, better sampling efficiency. Though we share some similar spirits with these work in the interpolation formulation, our main contribution lies more on how to better leverage pretrained diffusion priors and introducing a novel fast solver. We suggest to read our **general response** for more comparisons.\n***\n\n**Q2: Using only LR or adding various noise/blur methods to LR may also work?**\n\n**A2:**  Sorry we did not find an existing diffusion-based method that uses only LR to construct training samples, as by doing so it is only possible to recover the LR images theoritically. \nSimilarly, adding various noise/blur methods to LR is not feasible, either. However, we do find a diffusion process, ColdDiff [4], that adds various blur to **HR** images and try to recover from the blured samples. In general, ColdDiff generates reasonable results, but significantly underperforms ours. Please see results in our general response and Reviewer 7zor Q2 for more details. \n\nUsing only LR images as **conditions** rather than **inputs** can achieve super-resolution, which is the focus of comparative methods in our paper, such as StableSR [6] and DiffBIR [7]. However, these methods do not effectively address the trade-off between efficiency and performance, which is a key issue that our proposed method aims to resolve. \n***\n\n**Q3: Comparison with methods can also skip half the inference steps.**\n\n**A3:** Indeed, there have been previous works implementing this idea, such as SDEdit [5]. However, their approach adopts an approximation that  $LR + noise \\approx HR + noise$, thereby enabling inference to start from the intermediate node. Although this approximation can be used, it introduces a gap between training and inference, leading to a decrease in model performance. In contrast, our method meticulously designs the forward diffusion equation to enable precise calculation of this skip, thereby eliminating the training-inference gap and achieving excellent performance. We conducted a comparative experiment as shown in the table below:\n\n| Method   | from corrupted LR input | inference steps | MUSIQ$\\uparrow$ | CLIPIQA$\\uparrow$ | MANIQA$\\uparrow$ |\n| ------ | :------------: | --------------- | --------------- | ----------------- | ---------------- |\n| StableSR | \\                       | 200             | 65.88           | 0.6234            | 0.4260           |\n| StableSR | starting point  = 9T/10 | 180             | 55.45           | 0.4288            | 0.3314           |\n| StableSR | starting point  = T/2   | 100             | 23.40           | 0.2098            | 0.1873           |\n| DiffBIR  | \\                       | 50              | 64.94           | 0.6448            | 0.4539           |\n| DiffBIR  | starting point  = 9T/10 | 45              | 50.09           | 0.4009            | 0.3176           |\n| DiffBIR  | starting point  = T/2   | 25              | 38.12           | 0.2751            | 0.2669           |\n| DoSSR    | starting point  = T/2   | 5               | **69.42**       | **0.7025**        | **0.5781**       |\n\nAs seen from the table above,  the performance of other methods drops heavily when skipping inference steps.  Skipping half of the diffusion model (starting point from t = T/2) leads to a noticeable decline in the model's performance (-64.48\\%  MUSIQ in StableSR , -41.29\\%  MUSIQ in DiffBIR)  .  Skipping fewer steps (starting point from t = 9T/10)  still results in a performance decrease for the model ( -15.8\\%  MUSIQ in StableSR, -22.87\\%  MUSIQ in DiffBIR).\n\n***\nReference:\n\n[1] Liu, Xingchao, and Chengyue Gong. \"Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow.\" The Eleventh International Conference on Learning Representations.\n\n[2] Zhu, Yixuan, et al. \"FlowIE: Efficient Image Enhancement via Rectified Flow.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[3] Yue, Zongsheng, Jianyi Wang, and Chen Change Loy. \"Resshift: Efficient diffusion model for image super-resolution by residual shifting.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[4] Bansal, Arpit, et al. \"Cold diffusion: Inverting arbitrary image transforms without noise.\" _Advances in Neural Information Processing Systems_ 36 (2024).\n\n[5] Meng, Chenlin, et al. \"SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations.\" _International Conference on Learning Representations_.\n\n[6]  Wang, Jianyi, et al. \"Exploiting diffusion prior for real-world image super-resolution.\" International Journal of Computer Vision (2024): 1-21.\n\n[7]Xinqi Lin, Jingwen He, et al. Diffbir: Towards blind image restoration with generative diffusion prior. arXiv preprint\n385 arXiv:2308.15070, 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qMW2rmbIYL", "reviewer": "Reviewer_7zor", "summary": "This paper proposes a Domain Shift diffusion-based SR model, named DoSSR, that capitalizes on the generative powers of pretrained diffusion models while significantly enhancing efficiency by initiating the diffusion process with low-resolution (LR) images. Introducing the SDEs to describe the process of domain shift and provide an exact solution for the corresponding reverse-time SDEs. Designing customized fast samplers, resulting in even higher efficiency, thereby achieving the state-of-the-art efficiency performance trade-off.", "questions": "1. As mentioned in Weakness, why do the resulting graphs look very beautiful, but the reference indicators are so far behind?\n\n2. Explain the differences with DPM-solver, since ito-taylor expansionin also exists in your paper.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.This paper is innovative and combines a domain shift equation with the existing diffusion model to obtain better inference results and efficiency.\n\n2.This paper has a solid theoretical foundation, the author gives a strict and sufficient formula derivation process.\n\n3.The fusion results obtained are very superior in the non-reference index.", "weaknesses": "1.\tIn Table 1, the results of the proposed method on the reference index seem unsatisfactory, even much lower than other diffusion model methods.\n\n2.\tIn colddiff [r1], Bansal et al proposed a method of reasoning from an arbitrary starting point. What is the difference between the method of reasoning from LR mentioned in this article and it? Such clarification would be useful.\n\n3.\tIn order to facilitate observation, it is recommended to perform a partial zoom on Fig. 4 (b) for closer examination.\n\n[r1] Bansal, Arpit, et al. \"Cold diffusion: Inverting arbitrary image transforms without noise.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Regarding the reference index.**\n\n**A1:**  For real-world image super-resolution tasks, using reference metrics is not always an effective way to measure the quality of the generated results, as the 'groundtruth' image is merely one of many plausible outcomes. For methods based on diffusion models, particularly those utilizing large-scale pretrained diffusion models, the models' strong generative capabilities often produce outputs that, while differing from the ground truth, remain reasonable. Consequently, this can lead to relatively low reference metrics, as similarly discussed in other papers [1,2,3]. \n\nAdditionally, the reference metrics of our method is reasonable as expected. Compared to our baseline model DiffBIR, we show improvements across three test datasets and is generally comparable to other SD-based models in Table 1. \n\n***\n\n**Q2: Comparion to ColdDiff [4].**\n\n**A2:** The core idea of ColdDiff is to perturb the data distribution by sampling non-Gaussian noise (such as using blur, snow, or pixelation) to degrade it into a known data distribution. \nTaking the super-resolution task as an example, it allows for reasoning from an arbitrary starting point because LR images can be approximated as a node in the forward degradation process. \nThus, one can initiate the reverse process from this node to perform degradation inference. \nHowever, doing so introduces a gap between training and inference, because the degradation is hand-crafted in training, differing from real-world degradations. This will lead to a significant drop in model performance.\n In contrast, our method, through a carefully designed forward diffusion equation, allows for precise calculation of the insertion inference from the intermediate node, as the latter half of the forward process is seen in the training process.\n\nAnother drawback of ColdDiff is that it is designed on the pixel space, while modern diffusion models mostly operate on a latent space. Degradation in the pixel space is not always equivalent to degradation in the latent space, makes it difficult for adaptation to latent diffusion models . \n\nWe also provide experimental results in the **general response**, comparing our method to ColdDiff. It can be seen even using the full sample scheme (not skipping the first half steps), ColdDiff largely underperforms ours.\n\n***\n\n**Q3: Regarding differences with DPM-solver.**\n\n**A3:** DPM-solver is derived under the DDPM framework. Since our forward diffusion process is different from that of DDPM, DPM-solver cannot be simply applied to our method. The proposed DoS-SDE solver is to our DoSSR as DPM-solver is to DDPM. \nThe derivation of the two is similar as they both use ito-taylor expansion as a tool.\n\nIn fact, there are some interesting mathematical connections between our proposed DoS SDE-Solver and DPM-Solver series. When $\\eta_t \\equiv 0$ ,our DoS SDE-Solver Eq.(12) becomes \n$$\nx_t = \\frac{\\alpha_t}{\\alpha_s}\\frac{\\lambda_t^2}{\\lambda_s^2}x_s+\\alpha_t(1-\\frac{\\lambda_t^2}{\\lambda_s^2})x_\\theta(x_s,\\hat{x}_0,s)+\\alpha_t\\sqrt{\\lambda_t^2-\\frac{\\lambda_t^4}{\\lambda_s^2}}z_s, \\lambda_t = \\frac{\\sigma_t}{\\alpha_t(1-\\eta_t)}\n$$ \nthe DoSG introduced by us becomes ineffective, and our DoS SDE-Solver is equivalent to the SDE DPM-Solver++[5], \n$$\nx_t=\\frac{\\sigma_t}{\\sigma_s}e^{-h}x_s + \\alpha_t (1-e^{-2h}) x\\_\\theta(x_s,s) +\\sigma_t\\sqrt{1-e^{-2h}}, h = log(\\alpha_t/\\sigma_t)-log(\\alpha_s/\\sigma_s).\n$$\n\n*** \n\n**Q4: Zoom-in for better visualization.**\n\n**A4:** Thanks for the suggestion! We will include a partial zoom-in on Fig. 4(b) in the next version.\n\n***\n\nReferences:\n\n[1] Wu, Rongyuan, et al. \"Seesr: Towards semantics-aware real-world image super-resolution.\" _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 2024.\n\n[2] Wang, Jianyi, et al. \"Exploiting diffusion prior for real-world image super-resolution.\" _International Journal of Computer Vision_ (2024): 1-21.\n\n[3] Yang, Tao, et al. \"Pixel-aware stable diffusion for realistic image super-resolution and personalized stylization.\" _arXiv preprint arXiv:2308.14469_ (2023).\n\n[4] Bansal, Arpit, et al. \"Cold diffusion: Inverting arbitrary image transforms without noise.\" _Advances in Neural Information Processing Systems_ 36 (2024).\n\n[5] Lu C, Zhou Y, Bao F, et al. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models[J]. arXiv preprint arXiv:2211.01095, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2upkhGRlTw", "reviewer": "Reviewer_tfJj", "summary": "The existing diffusion-based image restoration models do not fully exploit the generative prior of the pre-trained diffusion models and produce high-quality images starting from Gaussian noise, leading to inefficiency. To address these issues, the paper a Domain Shift diffusion-based SR model (DoSSR) which can leverage the generative power of the pre-trained diffusion models and reconstruct super-resolved images based on the input corrupted images. It shows that the proposed method can effectively improve efficiency with few evaluation steps. Specifically, experiment results show that the proposed model achieves a slightly better performance on some benchmark datasets, including synthesized data and real-world data, in terms of various perceptual quality measurements.", "questions": "1. It is better to illustrate the overall structure of the adopted models. For example, how to adopt the corrupted input as conditional information for the T2I model. \n2. The novelty of the paper is a significant concern. It is better to demonstrate the advances of the proposed method, compared with the previous studies.\n3. The paper proposes to specifically design the drift term in SDE for image super-resolution, which leads to the reduction of evaluation steps. It easily causes confusion, because the additional drift term is to interpolate the corrupted input and the corresponding high-quality images. The sampling method is the key point. These two points should be clarified.", "limitations": "The paper does not discuss the limitations and social impact of the proposed method. It is hard to evaluate the model fairness, generalization ability, robustness, and transparency of the proposed method.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper clearly describes the limitations of the existing diffusion-based image restoration models, which are hot topics in image restoration based on Gaussian diffusion models. The structure of the paper is well-organized, so the whole paper is understandable and easily follow.\n\n2. How to efficiently perform image restoration based on the diffusion models is a significant challenge. The paper demonstrates that it can achieve better performance using few evaluation steps, which is impressive.\n\n3. The paper evaluates the proposed method on several benchmark datasets, which contain synthesized data and real-world data, and the ablation studies are comprehensive.", "weaknesses": "1. In implementation, this research adopts LAControlNet and SD 2.1-base as the pre-trained T2I model. However, the paper does not illustrate the overall structure of their proposed method, so it is hard to understand how to adopt the corrupted input as conditional information.\n\n2. The novelty of the paper is limited. 1). Leveraging pre-trained large-scale diffusion models for image super-resolution under the framework of ControlNet has been investigated in many previous studies [1-4]. These methods can also produce high-quality restored images from the corrupted input. 2). The goal of the proposed domain shift SDE  is to design a diffusion bridge between the corrupted input images and the corresponding high-quality images, which is very similar to flow matching-based methods, such as rectified flow [5]. However, the paper does not discuss the flow-based models.\n\n3. By leveraging flow-based models, several methods [6-7] have been proposed for image restoration.\n\n4. The performance improvement on several datasets is limited. It is hard to evaluate the visual quality of images generated by the proposed method through visual comparison because no GT image is illustrated.\n\n\n[1] Wang J, Yue Z, Zhou S, Chan KC, Loy CC. Exploiting diffusion prior for real-world image super-resolution. IJCV, 2024.\n\n[2] Chen X, Tan J, Wang T, Zhang K, Luo W, Cao X. Towards real-world blind face restoration with generative diffusion prior. IEEE Transactions on Circuits and Systems for Video Technology. 2024 Apr 1.\n\n[3] Yu F, Gu J, Li Z, Hu J, Kong X, Wang X, He J, Qiao Y, Dong C. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild. In CVPR 2024.\n\n[4] Yang T, Ren P, Xie X, Zhang L. Pixel-aware stable diffusion for realistic image super-resolution and personalized stylization. arXiv preprint arXiv:2308.14469. 2023.\n\n[5] Liu X, Gong C, Liu Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. ICLR, 2023.\n\n[6] Liu J, Wang Q, Fan H, Wang Y, Tang Y, Qu L. Residual denoising diffusion models. CVPR, 2024.\n\n[7] Zhu Y, Zhao W, Li A, Tang Y, Zhou J, Lu J. FlowIE: Efficient Image Enhancement via Rectified Flow. CVPR, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Overall model structure and implementation details**\n\n**A1:**  Thanks for the valuable suggestion! Please find the figure and descriptions about detailed model structure in the attached PDF.  Also, Figure 2(a)(b) in the main text shows training and inference pipelines and may help better understand.  Generally, we adopted the same model structure as in DiffBIR[1], where we use LR as condition input for the ControlNet module. The ControlNet here is initialized from the Stable Diffusion 2.1 Unet Decoder, then trained to generate HR images given LR images as condition inputs. We will add these to our appendix in the next version.\n\n*** \n\n**Q2: Novelty against ControlNet based methods**\n\n**A2:**  Yes, our method follows the ControlNet framework similar to the listed literature. However, we want to emphasize that we did not claim the ControlNet framework as our contribution. Rather, the only reason that we opt to it  is that we adopt DiffBIR, which is based on ControlNet,  as our baseline. It is totally possible to implement our method without ControlNet, since our innovation is mainly on the formulation of domain shift diffusion and a corresponding efficient sampler.\n\nThe effecitveness of our proposed DoSSR can be justified by comparing the DiffBIR baseline with ours, please see Table 1 in the main text. The comparison is fair because the ControlNet part is exactly the same, and the only difference is that we  replace the learning objective and sampler from DDPM/DDIM with our propsed ones. We see a significant boost in generation quality, with $+6.9$\\% MUSIQ and $+27.4$\\% MANIQA in RealSR test set. Meanwhile, The inference efficiency is largely improved also, with number of function evaluations (NFE) decreased from 50 to 5, and lantency reduced by 5x from 5.85s to 1.03s. \n\nWe would like to once again emphasize that our contribution is not the ControlNet-based SR framework. Please see our general response for a better illustration of our contributions.\n\n*** \n\n**Q3: Novelty against Rectified Flow**\n\n**A3:**  Thanks for the suggestion. Rectified Flow[2] (or we take the implementation of FlowIE[3] as an example for the SR task) does share a similar spirit with our formulation, from the pespective of the LR-to-HR interpolation design. However, our work differs from Rectified Flow mainly on that we focus more on better leverage of DDPM pretrained generative prior. Specifically, this results in different design chioces of the interpolation schedule and the noise schedule. Finally, we show under a fair comparison setting, our method remarkbly outpuerforms FlowIE, thanks to the better leverage of pretrained prior. Please see quantitative results in our general response. By the way, FlowIE is **concurrent with our work**, as it was not publicly available prior to the NeurIPS submission deadline.\n\n***\n\n**Q4：Regarding the visual comparison**\n\n**A4:** For real-world image super-resolution tasks, since the ground truth represents just one of many plausible results, many prior arts do not include the GT[1,4,5], and we adhere to this practice as well. However, we appreciate the suggestion and will include a comparison with the ground truth in future versions to better present the visual comparison.\n\n***\n\n**Q5：Clarification on the drift term of DoS-SDE solver**\n\n**A5:** Thanks for the suggestion. We believe this is a misunderstanding. In fact, the interpolation between LR and HR only exists in the training process. During inference, the sampler does not require an interpolation. Please note in Eq (12) the drift term (Domain Shift Guidance) contains only the LR latent $\\hat{x}_0$. \n\n*** \nFinally, regarding the discussion of the limitations and social impact of the proposed method, due to page constraints, we have placed it in Appendix C.2. We intend to move it to the main body in the next version.\n\n***\nReferences:\n\n[1] Lin X, He J, Chen Z, et al. Diffbir: Towards blind image restoration with generative diffusion prior[J]. arXiv preprint arXiv:2308.15070, 2023.\n\n[2] Liu X, Gong C. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow[C]//The Eleventh International Conference on Learning Representations.\n\n[3] Zhu Y, Zhao W, Li A, et al. FlowIE: Efficient Image Enhancement via Rectified Flow[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 13-22.\n\n[4] Sun, Haoze, et al. \"Coser: Bridging image and language for cognitive super-resolution.\" _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2024.\n\n[5] Yu, Fanghua, et al. \"Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild.\" _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KJcoecuZ6I", "reviewer": "Reviewer_dM7n", "summary": "This paper proposed a Domain Shift diffusion-based SR model, namely DoSSR, which capitalizes on the generative powers of pretrained diffusion models while significantly enhancing efficiency by initiating the diffusion process with low-resolution (LR) images. In addition, authors advance their method by transitioning the discrete shift process to a continuous formulation to further enhance sampling efficiency. The proposed method is evaluated on synthetic and real-world datasets and achieved competitive results.", "questions": "1. The method proposed in the paper is based on the RealESRGAN preprocessing and LAControlNet (from DiffBIR). Is the model trained based on the pre-trained model of DiffBIR? What about the results on the baseline1 \"RealESRGAN preprocessing + DiffBIR\" or baseline2 \"RealESRGAN preprocessing + w/o Domian Shift Guidance(DoSG)\"? Are there some experiments to verify the performance improvement of the proposed method compared to the baseline1 or baseline2? \n2. What are the advantages of the DoSSR Solvers proposed in the paper compared to existing accelerated sampling schemes (such as DDIM [1], Dpm-solver++ [2], EDM [3])? Are there any ablation studies to verify it? \n\nThe relevant references are listed as follows, \n\n[1] Song, Jiaming, Chenlin Meng, and Stefano Ermon. \"Denoising diffusion implicit models.\" arXiv preprint arXiv:2010.02502 (2020). \\\n[2] Lu, Cheng, et al. \"Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.\" arXiv preprint arXiv:2211.01095 (2022). \\\n[3] Karras, Tero, et al. \"Elucidating the design space of diffusion-based generative models.\" Advances in neural information processing systems 35 (2022).", "limitations": "The authors fully explain the limitations and potential societal impact in this paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed DoS-SDEs achieve a remarkable speedup in inference, demonstrating its superior efficiency.\n2. The proposed method achieved competitive results on non-reference metrics. \n3. The paper is well-organized and the figure is clear to understand.", "weaknesses": "1. The Domain Shift scheme seems to be similar to ResShift, where noise schedule is replaced with the solution in DDPM. Can the authors further explain the novelty of proposed Domain Shift scheme?\n2. The experimental setting of the paper is insufficient. See questions for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Novelty against ResShift**\n\n**A1:**  Conceptually, our method is similar to ResShift, as both involve the design of a diffusion bridge between the LR and HR images.  \nThe main differences between the two diffusion schemes are that ours are carefully designed in order to better leverage the generative prior of pretrained diffusion models. \nSpecifically, this results in different design chioces of the interpolation schedule and the noise schedule.\nA major advantage that our formulation brings is that we achieve a better trade of between generative quality and inference efficiency. \nFurthermore, we extend our formulation from discrete version to a continuous version and propose a  solver based on  stochastic differential equation (SDE). This solver improves our inference efficiency even furhter. \n\nWe refer the reviewer to see more comparisons between ResShift and our method in our general response, Appendix A.7 and Appendix C.1.\n\n*** \n\n **Q2: More baseline results**\n\n**A2:**  Fisrt, our model is NOT trained based on the pre-trained model of DiffBIR. Instead, we use the DiffBIR architecture and initialize the weights with Stable Diffusion 2.1 weights. Specifically, our baseline should be the baseline2 as you mentioned: \"RealESRGAN preprocessing + w/o Domian Shift Guidance(DoSG)\".  \n\nThe results of the baseline2 here can be found in Table 1 in the attached PDF, where this baseline is represented as \"w/o DoSG (DDPM)\". We compare the baseline with ours under 3 different samplers with varied orders. All results show our method outperforms the baseline by a large margin.\n\n*** \n\n **Q3:Relation to existing solvers**\n\n**A3:**  Our proposed sampler, DoS SDE-Solver, is specifically designed for the model under our forward-noise scheme framework. Existing samplers, such as DDIM, DPM-Solver, and EDM, are not suitable for our model as they are designed for the default DDPM.  Therefore these sovlvers cannot be adapted to our model without bells and whistles. \n\nHowever, we can still show some analogy comparisons by comparing DDIM to the 1st order version of the DoS SDE-Solver, EDM to the 2nd order version, and 3rd order DPM-Solver++ to our 3rd order version.\n\nIn fact, there are some interesting mathematical connections between our proposed DoS SDE-Solver and DPM-Solver series. When $\\eta_t \\equiv 0$ ,our DoS SDE-Solver Eq.(12) becomes \n$$\nx_t = \\frac{\\alpha_t}{\\alpha_s}\\frac{\\lambda_t^2}{\\lambda_s^2}x_s+\\alpha_t(1-\\frac{\\lambda_t^2}{\\lambda_s^2})x_\\theta(x_s,\\hat{x}_0,s)+\\alpha_t\\sqrt{\\lambda_t^2-\\frac{\\lambda_t^4}{\\lambda_s^2}}z_s, \\lambda_t = \\frac{\\sigma_t}{\\alpha_t(1-\\eta_t)}\n$$ \nthe DoSG introduced by us becomes ineffective, and our DoS SDE-Solver is equivalent to the SDE DPM-Solver++[1], \n$$\nx_t=\\frac{\\sigma_t}{\\sigma_s}e^{-h}x_s + \\alpha_t (1-e^{-2h}) x_θ(x_s,s) +\\sigma_t\\sqrt{1-e^{-2h}}, h = log(\\alpha_t/\\sigma_t)-log(\\alpha_s/\\sigma_s).\n$$\n\n**We show quantitative results in Table 1 in the attached PDF**. In all setups, inference is\ncarried out over 5 NFEs(number of function evaluations) with roughly same latency . From Table 1, two conclusions can be drawn: Firstly, regardless of DDPM or DoSSR, higher orders lead to better performance. Secondly, our DoSSR demonstrates superior performance compared to the corresponding order solver with DDPM, benefiting from the inclusion of DoSG in our DoS SDE-Solver.\n\n***\n\nReference:\n\n[1] Lu C, Zhou Y, Bao F, et al. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models[J]. arXiv preprint arXiv:2211.01095, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response
We thank all reviewers for their positive feedback:  solid theoretical foundation (Reviewer 7zor), outperforming existing methods on multiple benchmarks (Reviewer QMdK), better performance using few evaluation steps (Reviewer tfJj), and well-organized, clear to understand (Reviewer dM7n).
***

## **Regarding contributions / novelty.**

We here first want to clarify the main contributions of our work, especially in a way by comparing our method with other plausible alternative formulations of diffusion processes. The diffusion formulations that we compare with include:
- **ColdDiff [1] :** Similar to DDPM. The distinction is that it opts to other degradation forms, such as blur and masking, instead of additive Gaussian noises as in DDPM. In our case for image super-resolusion, we select the blur degradation. Note that ColdDiff is originally applied in the pixel space, while we have to implement it in the latent space for fair comparison with our method.
 
- **Rectified Flow [2]** defined the forward process by $x_t = t \hat{x}_0 + (1-t) x_0$ where $\hat{x}_0$ is the data sample and $x_0$ is Gaussian noise. In our case for image super-resolusion, we choose FlowIE[4] as an implementation of rectified flow which replaces $x_0$ with the LR image as the starting point.  For comparison, our forward pass is defined by $x_t = \alpha_t (\eta_t  \hat{x}_0 + (1-\eta_t) x_0 ) + \sigma_t$. The main differences are two-fold: First, the interpolation schedule is diffferent; Second, we introduce an outer degradation parameterized with $\alpha_t, \sigma_t$. Both differences are crucial for making it possible to  seamlessly adapt the DDPM pretrained weights to image super-resolution. 

- **ResShift [3]**  defines the forward pass by $x_t = \eta_t \hat{x}_0 + (1- \eta_t) x_0 + k^2 \eta_t$. It can be seen as a weaken case of our formulation when we set $\alpha_t \equiv 1$ and $\sigma_t = k^2 \eta_t$, thus similar to our method.  However, the devil is in the details, our noise and scale factors are crucial for making it possible to effectively utilize the diffusion prior while ResShift fails.

We implement the above three forms of diffusion formulations under the exact same experiment setting as our DoSSR. Below we show the quantitative comaprison results on RealSR test set. More comprehensive results can be found in Table 2 in the attached PDF.
|  Method  |  MUSIQ$\uparrow$ | MANIQA$\uparrow$ | NFE$\downarrow$ |         
| ------ |  :-----------------: | :-----------------: | :-----------------: |    
| ColdDiff |  47.42           | 0.2783           | 5               |  
| FlowIE |   50.82           | 0.3228           | 1               |  
| ResShift |   56.01           | 0.4001           | 5               |  
|  DoSSR   |   **62.69**       | **0.5115**           | 1             |
|  DoSSR   |  **69.42**       | **0.5781**           | 5             |

- **(1) Comparison to ColdDiff:** Results of ColdDiff is significantly worse than other methods. The main reason can be summerized as two aspects. First, applying bluring kernels to latent features is not equivalent to applying them to raw images, while ColdDiff is originally designed for the latter. Second, the blurring kernel can only be designed in a  hand-craft manner and may not represent the real-world degradation, so when tested with real LR images there would be a domain gap. The limilation of hand-crafted degradation is also well-known in many previous image restoration works. In contrast, other methods, including our DoSSR, does not assume a fixed degradation, so the performance is much better.

- **(2) Comparison to FlowIE:** Because FlowIE emphasizes single step evaluation, we follow its setting and compare DoSSR with 1-setp evaluation with it.  It can be see that FlowIE is also not satisfactory and largely underperforms our DoSSR (-11.87\%  MUSIQ). This is mainly attributed to FlowIE's bad adaptation from the DDPM pretrained T2I model, since the learning objective of flow matching differs from score matching. By contrast, our design makes full use of the DDPM pretrained weights so performs much better.

- **(3) Comparison to ResShift:** ResShift also underperforms ours by a large margin. The reason is similar to why FlowIE underperforms ours: The formulation of ResShift does not consider adaptation from the pretrained diffusion prior. We suggest reviewers to see A.7 and C.1 in our Appendix for more detailed discussions.

To summarize, our formulation differs from other alternatives especially in terms of **better leverage and adaptation from DDPM pretrained T2I models**. Also, based on this formulation, we derive a fast SDE solver. These two designs leads to a very efficient solution for diffusion-based image SR, thus can be seen as our two main contributions over existing works.

Refereces:

[1] Bansal, Arpit, et al. "Cold diffusion: Inverting arbitrary image transforms without noise." Advances in Neural Information Processing Systems 36 (2024).

[2] Zhu, Yixuan, et al. "FlowIE: Efficient Image Enhancement via Rectified Flow." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Yue, Zongsheng, Jianyi Wang, and Chen Change Loy. "Resshift: Efficient diffusion model for image super-resolution by residual shifting." Advances in Neural Information Processing Systems 36 (2024).

[4] Zhu, Yixuan, et al. "FlowIE: Efficient Image Enhancement via Rectified Flow." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Meta-review: The rebuttal has well addressed the raised issues. All the reviewers recommend to accept this paper. The area chairs agree with this recommendation.
Decision: Accept (poster)