Solving Diffusion ODEs with Optimal Boundary Conditions for Better Image Super-Resolution
Authors: Yiyang Ma
Huan Yang
Wenhan Yang
Jianlong Fu
Jiaying Liu
Venue: ICLR 2024 poster (2024)
Keywords: diffusion models
diffusion ODE
image super-resolution
Primary area: generative models
TL;DR: We propose a method of steadily sampling higher-quality SR images from an existing diffusion-based image SR model.

Abstract: Diffusion models, as a kind of powerful generative model, have given impressive results on image super-resolution (SR) tasks. However, due to the randomness introduced in the reverse process of diffusion models, the performances of diffusion-based SR models are fluctuating at every time of sampling, especially for samplers with few resampled steps. This inherent randomness of diffusion models results in ineffectiveness and instability, making it challenging for users to guarantee the quality of SR results. However, our work takes this randomness as an opportunity: fully analyzing and leveraging it leads to the construction of an effective plug-and-play sampling method that owns the potential to benefit a series of diffusion-based SR methods. More in detail, we propose to steadily sample high-quality SR images from pre-trained diffusion-based SR models by solving diffusion ordinary differential equations (diffusion ODEs) with optimal boundary conditions (BCs) and analyze the characteristics between the choices of BCs and their corresponding SR results. Our analysis shows the route to obtain an approximately optimal BC via an efficient exploration in the whole space. The quality of SR results sampled by the proposed method with fewer steps outperforms the quality of results sampled by current methods with randomness from the same pre-trained diffusion-based SR model, which means that our sampling method ''boosts'' current diffusion-based SR models without any additional training.

Published as a conference paper at ICLR 2024
SOLVING DIFFUSION ODES WITH OPTIMAL BOUNDARY CONDITIONS FOR BETTER IMAGE
Yiyang Ma', Huan Yang', Wenhan Yang', Jianlong Fu², Jiaying Liu
Wangxuan Institute of Computer Technology, Peking University, Miicossoft Research, Pengcheng Laboratory
[myy12769, (eriiiaiinnggkuueee.... edu
(huayan, jianffemicrosoft..o com, Pyangwh@pcl.ac.cr
ABSTRACT
Diffusion models, as kind of powerful generative model, have given impressive results on the image super-resolution (SR) tasks. However, due to the random- ness introduced in the reverse process of diffusion models, the performances of diffusion- based SR models fluctuate at every time of sampling, especially for the samplers with few resampled steps. This inherent randomness of diffusion mod- els results in ineffectiveness and instability, making it challenging for users to guarantee the quality of SR results. However, our work takes this randomness an opportumity: fully analyzing and leveraging fective plug-and-play sampling method that has the potential to benefit a series of diffusion-based SR methods. More in detail, propose to steadily sample high-quality SR images from pre-trained diffusion-based SR models by solving diffusion ordinary differential equations (diffussion ODEs) with optimal boundary conditions (BCs) and analyze the characteristics between the choices of BCs and their corresponding SR results. Our analysis shows the route to obtain an approx- imately optimal BC via an efficient exploration in the whole space. The quality of SR results sampled by the proposed method with fewer steps outperforms the quality of results sampled by current methods with randomness from the same pre-trained diffusion based SR model, which means that our sampling method "boosts" current diffusion-basee based SR models without any additional training.
the construction of an ef-
INTRODUCTION
Diffusion models (Ho et al. 2020 have drawn great research attention within the domain of com- puter vision because their great capacity for image generation. Therefore, it is intuitive leverage such powerful models to tackle the demanding task of image super-resolution (SR). The diffusion- based image SR task is modeled as generating high-quality images by diffusion models conditioned on corresponding low-resolution images (Saharia et al. 2022c Li et al. 2022 [Shan et al.) 2023 Sahak et al., (023). However, the reverse process (i.e., generating process) of diffusion models, including randomness (Ho et al., 2020; [Song et al. 222 b). leads to unstable performances of the diffusion-base SR methods. In other words, the users cannot guarantee the quality of SR results they lack a principled approach and can only rely random sampling from diffusion-based mod- els. The previous methods did not consider or explore the issue of randomness. Although multiple random samplings methods can lead to reasonable SR images using well-trained diffusion-baan based SR models. However, we cannot guarantee the quality of one-time sampling, and the sampled results on average still fall short of optimal quality, with significant performance gaps. Thus, it is critical to pursue a stable sampling method that generates SR images from pre-trained diffusion models with guaranteed good performances.
Most current diffusion-based SR works (Saharia et al. 2022c Li et al.
2022 Shang et al. 2023 Sa-
hak et 2023 Wang et al. 2023 focus on the model design instead of the sampling method. The
" *Corresooning author.

---

Published as a conference paper at ICLR 2024
0.1497
0.1071
0.1376
0.1268
0.1468
LPIPS
yo (o(ool
SR samples of
Ma M01
Approximately Optimal Boundary Condition
p(xy| (
LRR image 8/
Boundary Canditions
LR image 992
Xg
SR sumples of 82
Po (Noluz)
0,1425
LPIPS
Figure Given a well-trained diffusion-based SR model, by solving diffusion ODEs, we can sample reasonable SR results with different BCs Xy as the figure shows. However, there is instability in the performances of each BC XT. We manage to find an approximately optimal BC Xy which can be projected to the sample Xo with nearly the highest probability density by the solution ho (XT, to diffusion ODE. Based on our analysis in the See. 3.2 XT shared by different LR images yı- The method of finding XT refers to the Sec. 3.3 [Zom in for best view]
most commonly used sampling method for diffusion-based SR works is a resampled DDPM sampler with 100 steps (DDPM-100) instead of the original DDPM sampler with 1000 steps of the training noise schedule (DDPM-1000) due to its significantly reduced time cost, despite the tradc-off in SR image quality. It is first introduced by SR3 (Saharia et al. 2022c) from WaveGrad (Chen et al, 2021) Later works following SR3 use DDPM- samplers sample from a Gaussian distribution with learned parameters at each step, resulting in in- stability. (Song al. (2021b) demonstrate that such discrete-time DDPM samplers can be regarded as solving diffusion stochastic differential equations (diffission SDEs) and further give ordinary dif- ferential equations which share the same marginal probability densities as diffusion SDEs. Such ordinary differential equations are referred to as diffusion ODEs. Different from diffusion SDEs, given boundary condition (BC) Xy, one can solve the diffusion ODEs via ODE samplers (e.g., DDIM (Song et al. 2021a), DPM Solver (Lu et al. 2022)) to get an exact solution Xp. Nevertheless, the BCs XT ~ No, also come with randomness and lead to the instability issue in sampling SR images. Hence, it is highly desirable to obtain principled way for estimating the optimal BC x to steadily offer sampled SR images with high quality.
default setting. These discrete- time DDPM
In this paper, we analyze the characteristics of the optimal BC x of diffusion ODEs of SR models and propose an approach to approximate the optimal BC ë by exploring the whole space with the criterion of reference set containing R HR-LR LR image pairs R (z), which is small subset of the training dataset. Then, we can steadily generate high-quality SR images by solving the diffasion ODEEs of the trained ceffuusion-based SR model with the above derived approximately optimal BC XT. We establish that the optimal boundary condition xx, utilized to solve the diffu- sion ODEs in diffusion-based SR models is independent of the LR image inputs. Thus, we only need to prepare the approximately optimal BC Xy once to sample SR images of other unseen LR images. The experiment demonstrates that this simple independence assumption empirically offers impressive performance in a plug-and and-play manner. The main idea is shown in Fig.
We evaluate our method on both bicubic-SR and real-SR degradation settings. For bicubic-SR, we train a vanilla diffusion- baasusdoo--bsedd SR model which simply concatenates LR images with noisy images x, as the architecture proposed SR3 (Saharia al. 2022c). For real-SR, we apply our method to StableSR (Wang et al. (003). which finetunes pre-trained Stable Diffusion (Rombach et al.) 2022) on real-SR data. Experiments show that the quality of SR images sampled by few-step diffusion ODE samplers with our explored BC Äπ significantly outperforms the quality of results sampled by existing methods owning the same architecture. Our method is not restricted to any specific architecture of diffusion-based SR models. As the models we utilize in bicubic-SR and real-SR are quite different, the versatility of our method can demonstrated by experiments. Therefore, any diffusion-based SR model can leverage the proposed method to steadily sample high-quality
2

---

Published as a conference paper at ICLR 2024
SR images with only few steps, and achieve improved performance. In this way, method can "boost" existing diffusion duffusion-based SR models in the hlug-and-play manner.
RELATED WORK
2.1 IMAGE SUPER-RESOLUTION
Image super-resolution has drawn great research interest in recent years (Dong et al. 2014 Kim et al. 2016 Tong et al.] 2017: Lim et al. 2017 Ledig et al. 2017 Wang et al. 2018 8 Zhang et al.) 2018b Liang et al. (021). As pioneer work of deep-learning based SR methods, SRCNN (Dong et al. 2014) builds 3-layer convolutional neural network to map LR patches to SR patches with the criterion of MSE between SR patches and patches, getting better PSNR than traditional meth- ods. SeResNet (Ledig et al.] (017) introduces residual connections into SR networks, achieving impressive performances. RCAN (Zhang et al.) 2018b) uses channcl-attention mechanism to learn local-correlation which is crucial to the SR task. SWINIR (Liang et al., 2021) leverages vision trans- formers (Dosovitskiy et al. 2021 Liu et al.
2021 to build backbones of SR neural networks and
outperforms CNN-based NNs.
However, PSNR between SR images and HR images has a gap with the visual quality of SR images. Using generative models can synthesize more perceptually pleasant results. Thus, SRGAN (Ledig jet al. 2017) introduces GANs (Goodfellow et al., 2014) to SR tasks. Furthermore, Menon et al. (hha))CChanccc
(2020) Yang al. (202T); Chan al. (2021) incorporate pre-trained GANs from specific domains
into their SR frameworks, leveraging the generative capabilities of these GANs. PixelSR (Dahl al. 2017) uses auto-regressive models to generate SR images pixel-by-pixel. SRFlow (Lugmayr
et al. 2020) models SR tasks by normalizing flow-based models (Kingma & Dhariwal 2018). SR3
(Saharia al.) 2022c) first uses diffusion models (Ho et al.] [0200] Song et al. 2021b) ) to gener-
ate SR images conditioned
corresponding LR images. DDRM (Kawar et al. 2022) designs
training-free algorithm to guide pre-trained diffusion models to generate high-quality images which
are consistent with the LR images. StableSR leverages pre-trained Stable Diffusion (Rombach et al.,
2022 as a generative prior.
2.2 DIFFUSION MODELS
In recent years, diffusion models (Ho et al. 2020; Song et al.) 2021b), type of generative model, have achieved impressive results across various research domains, including image generation (Dhariwal & Nichol | & Dhariwal 2021). text-to-image generation (Nichol et al. 2022
Ramesh et al.] 2022 Saharia et al. 2022b). multi-modal generation (Ruan et al. [023 Ma et al, first proposed by (Shhl-Dickstein et al. (2015) and sim- plified as DDPM by Ho et al. (2020) which can be trained as several simple denoising models. IMPMOvedDDPM (Nihhol & Dhariwal 202I) proposes to learn the variance of each reverse step and APM (Bao et al. 2022) claims that such variances have analytic forms which not need to be learned. Song al. (202Tb) extend the diffusion models with discrete Markovian chains continu- ous differential equations. Ho et al. (2022) propose to train diffusion models by "velocity", getting more efficiency. [Rombach et al. (2022) build diffusion models on latent spaces instead of image spaces, reducing the training and inferring cost.
2023) and on. Ditfusion models
In terms of applying diffusion models, GLIDE (Nichol et al. 2022 first proposes to build diffu- sion model to generate images from descriptive texts. DALL 2 (Ramesh et al.) (022) and Imagen (Saharia et al. 2022b) design better architecture and use more computing resources, achieving better performances. Palette (Saharia et al. (022a)) first applies diffusion models to image-to-image trans- lation tasks. DreamBooth (Ruiz et al. (023) finetunes pre- are-trained text-to- sext-to-image diffusion models to achieve the goal of subject-driven generation. audios and videos at the same time. [Singer et al. (2023) create novel videos from texts without text-to-vides data. These works prove that diffusion models have strong generative abilities.
(Ruan al. 2023 generates aligned
3

---

Published as a conference paper at ICLR 2024
3 SAMPLING SR IMAGES WITH OPTIMAL BCs OF Diffusion ODES
We first review diffusion models and their continuous differential equations, then analyze the optimal BCs xi used by diffission ODEs sample SR images from diffusion- SR models, last depict the method of approximating the optimal BCs &T in Eqn. 20 with the criterion of a reference set containing R image pairs. With the approximately optimal XT, can sample high-quality SR images from diffusion- SR models by solving diffusion ODEs steadily.
3.1 DIFFUSION MODELS, Diffission SDES AND Diffusion ODES
Diffusion models (Ho et al. 2020, Song et al., 2021b). a kind of generative model, first map samples from an unknown distribution (e.g., the natural image distribution) to samples from well-known distribution (e.g., the standard Gaussian distribution) by gradually adding noise, and then attempt to revert such process via denoising step by step. The first process is called forward process. Taking Xo as a sample of the unknown distribution X, T as the number of addsen step, the state x,, I E [0, T) of forward process satisfies
((xr|xo)= (xx; (():)(()xx)))) o
= (x(x;; 0,1),
(1)
where a(t). (t) are differentiable functions of t defined by hyper-parameters. Furthermore, Kingma et al. (2021) prove that the transition distribution g(xx(xo) can be given by the following stochastic differential equation (SDE) at any & [0, T]:
dx.- f(t)x,dt + g(t)dw.,
where w; is standard Wiener process, and ((t),(()) are given by
((1) loog al) (t) do² d allg)(()) o² (t).
(3)
dt
dt
dt
The reverse process attempts to learn parameterized distribution (x(x) to fit the real data distribu- tion q [g) by using a trained noise- priee-predictio model e (x,, f) to gradually generate Xy from Xy (Ho
al. 2020). Lu et al. 2022) prove that the reverse process can be done by solving the following parameterized SDE (diffusion SDE) with numerical solvers:
(2( ott) co t)]dt + g(t)d ~ ((O,I),
dx; = [f(1)x++
(4)
where 6g (x,,,)) 1) is trainable noise-prediction neural network and w, is another standard Wiener process in the reverse time. The original DDPM (Ho et al.) 2020) sampler used by current diffusion- based SR models is discrete-time solver of diffusion SDE. When discretizing diffusion SDEs, the step sizes are limited because the Wiener process w, contains randomness. Thus, the resampled DDPM-100 sampler which is mentioned before with larger step sizes performs not satisfying.
Moreover, [ong et al.] 2021b) give an ordinary differential equation (ODE) which has the same marginal distribution of diffusion SDE:
dx,
= f(t)x; + (2(1) co (x,x, (O, 20(()
(5)
dt
Such ODE called diffusion ODE. Because diffusion ODEs have no randomness, one can get an exact solution Xn given a BC by solving the disfussio ODEs with corresponding numerical
solvers like DDIM (Song et al. 2021a) or DPM-Solver (Lu al. 2022). Thus, we can use a
parameterized projection:
(6) represent the solution of 5 We extend the diffusion models conditional ones po (xolc) by providing conditions e when training the noise- prese-prediction model eg (xx,, e, t). By randomly dropping the conditions during the training process, the model can be jointly conditional and unconditional
Xo = h=<x)x xT (O,1))
(Ho & Salimans 2021) We define the projections:
Xn (xT, c)2 Xo = (x +),
are the solution to conditional diffusion ODE and the solution to unconditional diffusion ODE the same diffusion model respectively, where denotes the blank condition which is dropped.
4

---

Published as a conference paper at ICLR 2024
3.2 ANALYZING OPTIMAL BCS x3. OF Diffusion ODES FOR DIFFUSION BBSED SR MODELS
For image SR tasks, steady SR results mean deterministic samples of the learned conditional distri- bution ps X2) c), where the conditions are LR images y. In other words, we should only sample once from the distribution. The parameterized distribution po (xnly) learned by well-trained diffu- sion model a fitting to the data probability distribution ((xoly) and the training data pairs (z; yi) samples and conditions the distribution g(xoly), where z, denotes the corresponding HR im- age of y;. From the perspective of max-likelihood, the (Z;, yi) pairs should be located at the point with the biggest probability distribution ((xul81))
2 = arg max ((x)) V.).
(8)
x
So, the optimal sample of (xoly) should satisfy:
XD = arg max pe (xoly).
(9)
When we solve diffusion ODEs to sample from the diffusion model (xxoy)), we actually sample Xy NN, I) and project xy final samples Xo via the projection in Eqn. 7 By leveraging the law of total probability, we can replace the variable from Xy to Xy, getting the likelihood of xy:
po(xTly)= Σ po (Xo xxx pp po
(xT
(10)
yec
where C is the theoretically universal of all LR images and y indicates all the LR images. The proof of Eqn. 10 refers to the Sec. of the appendix. For unconditional sampling, we have:
Po (xT) = pe (x(x)) (oo ( (x).,)) = pe (hg (XT, o)).
(11)
By substituting Eqn.
10 into Eqn. optimal BCs and samples should satisfy:
Xi arg max pe (ho((xx, y)), x5 = he (x(. ~7011
(12)
Based on Bayesian rule, have:
Po (xooly) Po(Xo.y)
(x)))x
p(y) po (xx)).
(13)
p(y)
By replacing the variable from Xo to XT in Eqn. conditional distribution is:
13 with Eqn. 1O and Eqn. 11
the parameterized
po(ho(XT.y)) Po (o Φ)) po (ho(xx,, ()).
(14)
p(y)
In Eqn. 14 p(y) is the prior probability distribution of LR images which is a uniform distribution, and po (xr, XT, Φ)) is not related to the LR image y. po (x )) an implicit classifier, in- dicating the probability of image is the corresponding LR image of an unconditionally generated image (XT, Φ). For a well-trained model, such probability also approximately uniform, because the distribution of unconditionally generated images will be approximate to the real distribution of choosing images in the dataset, which is uniform. Thus, pe (he (xr, y)) approximately indepen- dent to the specific LR images y, which indicates:
X
arg max po (ooh(XT: 2 arg max (o Vy, EC xx--A(0,I)
(15)
x²-A((0,))
We design experiment in the Sec. E of the appendix to validate derivation of the approximate independence (ho to different y. Hitherto, we have stated that the optimal BC x is approximately general for different LR images y. In the next subsection, depict how to approx- imate x with the criterion of a reference set containing HR-LR image pairs R = ((z,, R which is subset of the training dataset.
5

---

Published as a conference paper at ICLR 2024
3.3 APPROXIMATING OPTIMAL BCs Xy OF Diffusion ODES FOR DIFFUSION-BASED SR MODELS
As we have discussed before, well-trained model po (xoly) is fitting of (xxnly). Thus, we can take yxxoly) to substitute (x(o))y) in Eqn. 15 getting an approximation XT of x7.
Xy arg max (xr.
(16)
xT=)((o.))
Besides, we have the max-likelihood Eqn. 8 of (xolyy):
2 arg max (xxx v)) = ha( arg max (()) y:)), xr~>V(0.I)
(17)
X
Considering the characteristics of natural images, the distribution (xxoly) is a continuous distribu- tion. So, there exists a neighbour around z; where (xxoy::) is monotonic. Furthermore, the closer Xg gets to 2;; the bigger g(xo yr) is. By taking M(:;)) as the function which measures the distance of two images, the Xr can be approximated by:
XT
arg max ((og(Xr. y:)) Ro arg x7-A)(0.))
min mig x(-N
2;).
(18)
Because the monotonicity of (x) is limited in small neighbour, R HR-LR image pairs R = ((z,, y:)) to calculate Xy to achieve better approximation:
set containing
Xy ~ arg min N A=1
(1)
(19)
Considering the perceptual characteristics of images, we take LPIPS (Zhang et al. 2018a) as the implementation of (.,.). Because the projection he is the solution diffusion ODE, it is difficult to give an analytical result of Eqn. T9 We use the idea of the Monte Carol method to estimate &y.
We randomly sample K xT (NO.I)
I). calculate Equ. 19 and choose the best one:
Se arg min
Σ LPIPS(ho(XT 2;),
(20)
xT
where is the set of randomly sampled K xT I). Last, given unseen LR images y, the corresponding SR images can be generated by:
No = ho(xxry))
(21)
EXPERIMENTS
In order to demonstrate the effectiveness the proposed sampling method, we apply our method on two diffusion-based SR models. For bicubic-SR, we train vanilla model following SR3 (Saharia jet al. 2022c) as the baseline. For real-SR, we utilize StableSR (Wang et al.) (023) with w 0.5 without any color fixing and DiAAR (Xia et al.] (023a] as the baseline. It is noted that DiffIR officially employs a 4-step-DDPM sampler without noise which we call "D-4" as shown in Tab. 2
4.1 IMPLEMENTATION DETAILS
Datasets. For bicubic-Soi SR, we train the model on the widely dataset DF2k (Agustsson & Tim- ofte 2017: Lim et al. 2017) which containing 3,450 high-resolution images. We train a 64x -> 256x 256 model. The training and architecture details of the bicubie-SR model refer to the Sec. B
of the appendix. For real-SR, directly leverage the official pre-trained model of StableSR (Wang ctt al. 2023) with w = 0.5 without any color fixing.
To test the performances of bicubic-SR, use 3 different datasets containing DIV2k-test (Agusts- son & Timofte 2017). Urban100 (Huang et al. 2015). 1100 (Martin et al. 2001). For DIV2k-test and Urban
randomly crop 1,000 256 256 patches as HR images and downscale them to 64x64 patches by bicubic kernel as corresponding LR patches. For B 1000, we randomly extract 200 patches as the image resolutions in this dataset are not large compared with those in other datasets.

---

Published as conference paper at ICLR 2024
Table Qualitative results bicubic-SR on test datasets. "X7" denotes "approximately optimal boundary condition" calculated by the proposed method. The metrics of the bottom rows all sampled with the same SR3 model trained by us. "DPMS" denotes DPM-Solver (Lu et al. 2022). Numbers of PSNR are calculated on RGB channels. Red numbers denote the best performances and blue numbers denote the second best performances.
DIV2k-
Urban LPIPS
BSD100
Model (& sampling method)
LPIPS +
PSNR
PSNR
LPIPS
PSNR t
ESRGAN
0.1082
28.18
0.1226
23.04
0.1579
23.65
RankSRGAN
0.1171
27.98
0.1403
23.16
0.1714
23.80
SRDiff
0.1286
28.96
0.1391
23.88
0.2046
24.17
DDPM-1000
0.1075
28.75
0.1165
24.33
0.1555
23.86
DDPM-250
0.1142
28.95
0.1181
24.41
0.1621
24.00
DDPM-100
0.1257
29.16
0.1232
24.51
0.1703
24.15
DPMS-20
0.1653
27.25
0.1413
23.46
0.2037
22.79
SR3
DDIM-50
0.1483
28.55
0.1333
24.16
0.1823
23.75
DDIM-100
0.1571
28.16
0.1335
24.05
0.1950
23.55
DPMS- 20-2000-2 + Xy
0.1210
27.45
0.1179
23.57
0.1687
22.81
DDIM-50 + XT
0.1053
28.65
0.1164
24.26
0.1552
23.99
DDIM-100 + Ny
0.1032
28.48
0.1136
24.12
0.1505
23.67
Qualitative results of real-SR on test datasets. "XT"" denotes "approximately optimal boundary condition" calculated by the proposed method. The metrics of the bottom rows all sampled with the same StableSR model (Wang et al. (2023). Numbers of PSNR are calculated on RGB channels. Red numbers denote the best performances and blue numbers denote the second best performances.
Table 2:
DIV2k-test LPIPS
RealSR LPIPS +
Model (& sampling method)
DISTS 4
PSNR 1
DISTS |
PSNR t
RealSR BSRGAN
0.3051
0.5148
22.52
0.2532
0.3673
26.30
0.2253
0.3416
22.13
0.2057
0.2582
25.52
DASR
0.2340
0.3444
22.02
0.2113
0.3014
26.32
Real-ESRGAN
0,2108
0.3109
22.36
0.2020
0,2511
25.12
KDSR-GAN
0.2022
0.2840
22.92
0.2006
0.2425
26.09
DDPM 100
0.2010
0.3189
19.42
0,2210
0.3065
21.37
StableSR
DDIM-50
0.2217
0.3629
18.82
0,2336
0.3536
21.24
DDIM-50 XT
0.2046
0.3169
19.55
0,2164
0.2999
22.13
D-4
0.1773
0,2360
22.94
0.2076
0.2604
25.33
DiffIR
D-4 ËT
0.1772
0.2357
22.95
0.1993
0.2419
25.82
To test the performances of real-SR, we utilize datasets including DIV2k-test (Agustsson Timofte, 2017) and RealSR (Cai et al., 2019). For DIV2k-test, employ the x4 degradation process pro- posed by Wang et al. (2021). 1). synthesizing 1,000 128x 128 LR patches. For RealSR, we randomly 512x 512 LR-HR pairs.
crop 1,000 128x 128
Compared methods and metrics. This paper proposes method of sampling from diffusion-based SR models, so, the main baselines are current sampling methods used by other diffusion-basee SR models on the same model. For bicubic-SR, we leverage several resampled DDPM samplers and diffusion ODE samplers. It is noted that report the performances of DDPM-1000 (Ho et al. 2020 upper bounds of previous sampling methods, which serves as evidence of our model's capability. For real-SR, we employ resampled DDPM-200 following the official setting of StableSR (Wang et al. 2023) and DDIM-50 (Song et al. 2021a) as the baseline of diffusion ODE solver. We utilize PSNR on RGB channels and T.PIS (Zhang al. (018a) as evaluation metries. For real-SR, further adopt DISTS (Ding et al. 2020 demonstrate the generality of the proposed method on diverse perceptual metrics.
Besides, report the performances of other SOTA SR methods. For bicubic-SR, show the
performances of SRDiff (Li al. 2022) 2022) and GAN- methods including ESRGAN (Wang et al. 2018) and RankSRGAN (Zhang et al. 2019). For real-SR, show the performances of RealSR (Ji al. 2020). BSRGAN (Zhang et al. 2021). 1. -SaGAN (Wang et al. (02T). DASR (Liang et al.
2022 and KDSR-GAN
Xia al. 2023b We the open-source codes and pre-trained models

---

Published as a conference paper at ICLR 2024
Bicubic
ESRGAN
DDPM-250
OurS
GT
SRDiff
RSRGAN
DDPM- 100
DDIM-50 50
Bicubic
ESRGAN
DDPM- 250
Ours
GT
SRDiff
RSRGAN
DDPM-100
DDIM-50
Biaubic
ESRGAN
DDPM-255
Ours
GT
SRDiff
RSRGAN
DDPM-100
DDIM-50
Figure 2: Qualitative comparisons of bicubic-SR results obtained by different methods. "RSRGAN" denotes RankSRGAN (Zhang et al. 2019). All images the right of the black line are sampled from the same vanilla diffusion-baat based SR model trained by us. [Zoom in for best view]
of these methods without any modification. The details the source codes of the compared methods refer to the Sec. of the appendix. To the best of our knowledge, the diffusion-based models with XT have achieved superior performances compared to GAN-based SR models. This highlights the effectiveness and efficiency of our method in surpassing the capabilities of GAN-based SR models.
Settings of calculating žт and diffuusio ODE solvers. As we have discussed in the Sec. 3.3 use reference set R { which contains HR-LR image pairs and set K. which contains randomly sampled Xy to calculate the approximately optimal BC Xy. In practice, the and K set to 300 and 1,000 respectively for both bicubic-SE SR and real-SR. The reference sets synthesized from DIV-2k training set by utilizing the degradation used in the training process. The use DDIM (Song et al. 2021a) on real-SR and further adopt DPM-Solver (Lu et al.) (022) on bicubic-SR to demonstrate that our method can be generally employed different types of diffusion ODE solvers.
discussion the effect of R and K refers the Sec. 4.3 For diffusion ODE solvers,
4.2 QUANTITATIVE AND QUALITATIVE RESULTS
The performances of Sicubic-SR and real-SR on testing datasets are shown in Tab. 1 and Tab. 2 re- spectively. For bicubic-SR, the performance of DDPM- 1000 shows the capacity of the model, while the commonly-used sampling methods including DDPM-250, DDPM-100 trade off sample quality for faster sampling speed. can be seen that the performance of the proposed sampling method with diffusion ODE solver of DDIM- 100 outperforms all other sampling methods of the same diffusion- based SR model. Remarkably, our method surpasses the previous upper-bound DDPM-1000 which much slower. For real-SR, our method surpasses the official sampling method of StableSR (Wang
8

---

Published as a conference paper at ICLR 2024
al. 2023). DDPM-200, with a faster sampling speed, unleashing the capability of StableSR better. Such results demonstrate that we can steadily generate high-quality SR images from the pre-trained diffusion-basse SR models by the proposed method. Visual comparisons of bicubic-SR images of different methods are shown Fig. 2 More visual results can be found in the Sec. Flf the appendix.
4.3 ABLATION STUDIES
we use a reference set = {(z, y:) and a set of randomly sampled Xr IC to estimate the approximately optimal BC Xy. The scales of the two sets will affect the quality of the estimated Xy. The larger R and K. are, the better estimation of XT is. Thus, we perform ablation studies of the scale of the two sets on the task of bicubic-SR.
As we have discussed in the Sec. 3.3
For the ablation on R. we keep K = 200. We build subsets R, containing image pairs and set i to 1, 2, 4, 8, 16. For each i, we build 8 R. with different random image pairs. With the criterion of each R2 we choose the corresponding XT and test them on subset DIV2K test set containing 100 patches with DDIM-50. The mean values and standard deviation values of LPIPS of the SR results with estimated Xy at each are shown in Fig.
For the ablation on K, we keep R 20. We randomly sample Xy to build sets K;; and set to 10, 20, 40, 80, 160. For each i, we build 8 KG; with different Xy. We estimate Xy from each IC; and test them the same subset of DIV2k test set used in the ablation studies on R with DDIM-50. The mean values and standard deviation values of LPIPS of the SR results with estimated Xy at each i also shown in Fig.
can be seen that the performances become better and steadier as R and K increase.
approuimately optimal
random optimal
random
442
042
14
value of
value
Figure 3: Ablation on values of R and K. Shadows denote the standard deviation, the red dotted lines denote LPIPS of SR samples of the subset by DDIM-50 with randomly sampled xy. indicating the lower-bound performance, and the green dotted lines denote LPIPS of SR results of the subset by DDIM-50 with X7, indicating the upper-bound of performance.
5 CONCLUSION AND FUTURE WORK
In this work, we propose to steadily sample high-quality SR images from crffiusionbbsedd SR models by solving diffusion ODEs with approximately optimal BCs ky. We describe the process of finding these optimal boundary conditions. Experiments show that the proposed sampling method outper- limited
forms commonly-used sampling methods for suffusiio-basee SR models. Our method to specific architectures of diffusion- based SR models, and does not require additional training. This flexibility allows our method to effectively enhance the sampling performance of pre-trained fiffusion-based SR models without any constraints in plug-and-pla manner.
The calculated approximately optimal BC žT has the same dimension as LR images y, which can not be directly applied to LR images with other shapes. We will explore designing algorithms which combine the Xy with resolution- arbbitrry sampling methods (Zhang et al., 2023) to achieve the goal method on LR images with different resolutions. Besides, we only discuss the žy
of applying in the tasks of image ruser-iesol In the future, we will explore further application in other low-level tasks, including image colorization, low-light enhancement, etc.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work was supported in part by the National Natural Science Foundation of China under Grant 62332010, and in part by the Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology).
REFERENCES
Eirikur Agustsson and Radu Timofte. NTIRE 2017 challenge on single image super-resolution: Dataset and study. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition Workshops, 2017.
Fan Bao, Chongxuan Li, Jun Zhu, and Zhang, Analytic-DPM: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. In Proc. Int'l Int'l Conf. Learning Representations, 2022.
Jianrui Cai, Hui Zeng, Hongwei Yong, Zisheng Cao, and Lei Zhang. Toward real-world single image super-resolution: A new benchmark and new model. In Proc. IEEE Int. Conf. Comput. Vis. 2019.
Kelvin CK Chan, Xintao Wang, Xiangyu Xu, Jinwei Gu, and Chen Change Loy. GLEAN: Gen- erative latent bank for large-factor image super-resolution. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2021.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. WaveG- rad: Estimating gradients for waveform generation. In Proc. Int'l Conf. Learning Representations, 2021.
Ryan Dahl, Mohammad Norouzi, and Jonathon Shlens. Pixel recursive super resolution. In Proc. IEEE Int. Conf. Comput. Vis. 2017.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In Proc. Annu. Conf. Neural Inf. Process. Systems, 2021.
Keyan Ding, Kede Ma, Shiqi Wang, and Eero Simoncelli. Image quality assessment: Unifying structure and texture similarity. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2020.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. In Proc. IEEE European Conf. Computer Vision, 2014.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x 16 words: Transformers for image recognition at scale. In Proc. Int'l Conf. Learn- ing Representations, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proc. Annu. Conf. Neural Inf. Process. Systems, 2014.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In Proc. Annu. Conf. Neural Inf. Process. Systems Workshops, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proc. Annu. Conf. Neural Inf. Process. Systems, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022.
Jia- Bia-Bin Huang. Abhishek Singh, and Narendra Ahuja. Single image super-resolution from trans- formed self-exemplars. In Proc. IEEE Int') Conf. Computer Vision and Pattern Recognition, 2015.
10

---

Published as a conference paper at ICLR 2024
Xiaozhong Ji, Yun Cao, Ying Tai, Chengjie Wang. Jilin Li, and Feiyue Huang. Real-worts super- resolution via kernel estimation and noise injection. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition Workshops, 2020.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Proc. Annu. Conf. Neural Inf. Process. Systems, 2022.
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2016.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Proc. Annu. Conf. Neural Inf. Process. Systems, 2021.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Proc. Int'l Conf. Learning Representations, 2015.
Durk P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Proc. Annu. Conf. Neural Inf. Process. Systems, 2018.
Christian Ledig, Lucas Theis, Ferenc Huszár. Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, al. Photo-realistic single image super-resolution using a generative adversarial network. In Proc. IEEE Int'l Conf. Com- puter Vision and Pattern Recognition, 2017.
Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. SRDiff: Single image super-resolution with diffusion probabilistic models. Neurocomput- ing, 2022.
Jie Liang Hui Zeng. and Lei Zhang. Efficient and Eegradation-adapiii network for real-world image super-resolution. Proc. IEEE European Conf. Computer Vision, 2022.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. SwinIR: Image restoration using swin transformer. In Proc. IEEE Int. Conf. Comput. Vis. Workshops 2021.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep resid- ual networks for single image super-resolution. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition Workshops, 2017.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proc. IEEE Int. Conf. Comput. Vis. 2021.
Cheng Lu, Yuhao Zhou, Fan Bao. Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In Proc. Annu. Conf. Neural Inf. Process. Systems, 2022.
Andreas Lugmayr, Martin Danelljan, Lue Van Gool, and Radu Timofte. SRFlow: Learning the super-resolution space with normalizing flow. In Proc. IEEE European Conf. Computer Vision, 2020.
Yiyang Ma, Huan Yang, Wenjing Wang, Jianlong Fu, and Jiaying Liu. Unified multi-modal latent diffusion for joint subject and text conditional image generation. arXiv preprint arXiv:2303.09319. 2023.
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proc. IEEE Int. Conf. Comput. Vis. 2001.
Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. PULSE: Self- supervised photo upsampling via latent space exploration of generative models. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2020.

---

Published as a conference paper at ICLR 2024
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Proc. Int'l Conf. Machine Learning, 2022.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proc. Int'l Conf. Machine Learning, 2021.
PyTorch-CCntributors PyTorch. https://pytorch..
2017.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2022.
Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. LAM-Diffusion Learning multi-modal diffusion models for joint audio and video generation. In Proc. IEFE Int'l Conf. Computer Vision and Puttern Recognition, 2023.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfrr Aber- man. DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proc. IEEE Int'l Conf. Computer Vision and Puttern Recognition, 2023.
Hshmat Sahak, Daniel Watson, Chitwan Saharia, and David Fleet. Denoising diffusion probabilistic models for robust image super-resolution the wild. arXiv preprint arXiv:2302.077 020.00042277444 2023.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. Proc. ACM SIG- GRAPH, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Proc. Annu. Conf. Neural Inf. Process. Systems, 2022b.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2022c.
Shuyao Shang, Zhengyang Shan, Guangxing Liu, and Jinglin Zhang. ResDiff: Combining cnn and diffusion model for image super-resolution. arXiv preprint 2rXhv:2303.08714 2023.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a- Video: Text-to-vided generation without text-video data. Proc. Int'l Conf. Leurning Representations, 2023.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In Proc. Ini'l Conf. Machine Learning, 2015.
Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. In
Proc. Ini'l Conf. Learning Representations, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proc. Int' I Conf. Learning Representations, 2021b.
Tong Tong, Gen Li, Xiejie Liu, and Qinquan Gao. Image super-resolution using dense skip connec- tions. In Proc. IEEE Int. Conf. Comput. Vis. 2017.
Jianyi Wang, Zongsheng Yue, Shangehen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution. arXiv preprint 0050v 2023.
12

---

Published as a conference paper at ICLR 2024
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. ESRGAN: Enhanced super-resolution generative adversarial networks. In Proc. IEEE European Conf. Computer Vision Workshops, 2018.
Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-ESRGAN: Training real-world blind super-resolution with pure synthetic data. In Proc. IEEE Inl. Conf. Comput. Vis. Workshops 2021.
Bin Xia, Yulun Zhang, Shiyin Wang, Yitong Wang, Xinglong Wu, Yapeng Tian, Wenming Yang, and Luc Van Gool. DifARR: Efficient diffusion model for image restoration. In Proc. IEEE Int. Conf. Comput. Vis. 2023a.
Bin Xia, Yulun Zhang, Yitong Wang. Yapeng Tian, Wenming Yang. Radu Timofte, and Luc Van Gool. Knowledge distillation based degradation estimation for blind super-resolution. In Proc. Int'l Conf. Learning Representations, 2023b.
Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang. GAN prior embedded network for blind face restoration in the wild. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2021.
Jiahui Zhang, Shijian Lu, Fangneng Zhan, and Yingchen Yu. Blind image super-resolution via contrastive representation learning. arXiv preprint arXiv:2107.00708. 2021.
Qinsheng Zhang, Jiaming Song. Xun Huang, Yongxin Chen, and Ming- Yu Liu. Diffcollage: Parallel generation of large content with diffusion models. In Proc. IEEE Int'l Conf. Computer Vision and Puttern Recognition, 2023.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proc. IEEE Int'l Conf. Computer Vision and Puttern Recognition, 2018a.
Wenlong Zhang, Yihao Liu, Chao Dong, and Yu Qiao. RankSRGAN: Generative adversarial net- works with ranker for image super-resolution. In Proc. IEEE Int. Conf. Comput. Vis. 2019.
Yulun Zhang, Kumpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention networks. In Proc. IEEE European Conf. Computer Vision, 2018b.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A PROOF TO EQN. 10
Proposition. The likelihood of xT which is obtained by replacing the variable from xo to xT the likelihood of xo satisfies:
px(xlly)= = -Po(ho(Xr,)))
Proof. First, we apply the law of total probability to all the choices of y E C, getting:
px(Tll Σ Epddbolyy//aootttttss))) SEC
(A.1)
where C is the theoretically universal set of all LR images and y indicate all the LR images. If y X (x,, y)ly) would indicate the probability of the generated image (x being the corresponding SR image of another LR image which is almost O. Thus, y can only be equal to y. Thus, have:
{0,5 yy
(x(hg(xT.))l))
(A.2)
furthermore,
Σ
(A.3)
YЕC
Because y is the reverse function of he(xr, which is deterministic, we have:
Σ
[oo Pa(ho ()T((x))))) YЕC
(A.4)
Hence, the proof is completed.
B FULL IMPLEMENTATION DETAILS OF THE SR3 BASELINE
The implementation details of the SR3 baseline trained by us contain two parts: details of the diffusion-based SR model (xol)) and details of the noise-prediction network used by the dif- fusion model e (xt, yt We provide them separately in this section, ensuring the reproducibility of our results.
B.1 IMPLEMENTATION DETAILS OF THE DIFFUSION MODEL.
We use the original diffusion model introduced by Ho et al. (2020) which only predicts the noise in noisy state without predicting the variances. Thus, the model can be simply trained through the mean square error (MSE) loss between the predicted noise and the real noise. The training loss is:
Er.xo.ele (x y, 2)1
(B.1)
where denotes LR images and c, y, is the noise prediction network which is particularly depicted in the Sec. [.2 The noise schedule is the same as Ho et al. al. (2020). which sets 1000 and the forward process variances to constants increasing linearly from 31 10 to 3T 0.02. During the reverse process of DDPM, set the variance o; to 1- 3, which performs much better than a; 3r in resampled few-step sampling, following Nichol & Dhariwal (2021). Following Dhariwal & Nichol (2021), we use a resampled schedule for a few-step sampling. For DDPM-250, we use the schedule of 90, 60, 60, 20, 20, which is the same the best schedule for image generation tasks found by Dhariwal & Nichol (202I). For DDPM-100, we use the schedule 43, 20, 15, 10, 10, which is not exhaustively swept.
14

---

Published as a conference paper at ICLR 2024
B.2 IMPLEMENTATION DETAILS OF THE NOISE-PREDICTION NETWORK
Following most of the current diffusion models Ho et al. 2020 Saharia et al. 2022c Li et al. 2022
Shang al. 2023 Ruan al. 2023 Ma et al. 2023; Ramesh et al., 2022 Saharia al. 2022b Rombach et al. (022) used in several aspects, we use Net as the backbone of our noise-prediction
network. Following SR3 (Saharia al. 2022c), the LR images y are first upsampled by the bicubic kernel to the same size to noise states and then simply concatenated to noise states x, along the channel dimension. The bicubic kernel we used both in downsampling and upsampling is introduced by torchvision (PyTorch- Contributors, (017) with anti-alias. The architecture of our UNet is similar to the upsampler built by Dhariwal & Nichol (2021) with a small number of parameters. The detailed architecture is shown in Tab. 3 We first train the model for 2M iterations with a batch size of 16, then train the model for another IM iterations with a batch size of 64, ensuring the convergence of model. We use Adam optimizer (Kingma& & Ba (2155) during the whole training process and use mixed precision to accelerate training. The total training cost is about 2000 Tesla V100 GPU GPU-hours.
Table 3: Detailed architecture of our UNet used for the diffusion- SR model.
UNet 64 + 256
Model size
36M
Channels
92
Depth
2
Channels multiple
1,1,2,2,3
Heads
4
Attention resolution
32,16
BigGAN up/downsample
V
Dropout
0.0
Batch size
16 - 64
Iterations
2M 1M
Learning rate
le 4
C THE SOURCES OF THE COMPARED METHODS
The sources of the compared methods including SRDiff (Li et al. 2022). ESRGAN (Wang et al.
2018 RankSRGAN (Zhang al., 2019). RealSR (Ji et al. (020). BSRGAN (Zhang et al. [021] and DASR (Liang et al. 2022) are shown in Tab. 4 It is noted that the model of RealSR (Ji et al.
2020 we employ is "DF2K-JPEG".
Table 4: The sources of the compared methods.
Degradation
Method
URL
SRDiff
httpes///githbbb.mm///aaaat////
Bicubic
ESRGAN RankSRGAN
RealSR
Real
BSRGAN
DASR
http:///gthabb.oom//cssaaaaa///
D BOOSTING MID-TRAINING MODEL.
In the main paper, we analyze all the characteristics of x; and propose method of approximating Ry assuming the fiffusion-based SR model has been well-trained (i.e., pe (x(x)ly) is close fit to gy)))))) However, we find that the proposed method can also boost mid-trane mid-training models. We use the baseline SR3 model of bicubic-SR. The model is trained for only 500k iterations with batch
15

---

Published as a conference paper at ICLR 2024
Table 5: Performances of the mid-training model with only 500k training iterations. Red numbers denote the best performances among the mid-traing training model and blue numbers denote the second best performances among the mid-training model.
Sampling
DeV2k-test
Urban 100
BSD100
Model
method
LPIPS
PSNR
LPIPS
PSNR 1
LPIPS 4
PSNR 1
DDPM-1000
0.1075
28.75
0.1165
24.33
0.1555
23.86
DDPM-250
0.1142
28.95
0.1181
24.41
0.1621
24.00
well- trained
DDPM-100
0.1257
29.16
0.1232
24.51
0.1703
24.15
DDIM-50
0.1483
28.55
0.1333
24.16
0.1823
23.75
DDIM-50 Xr
0.1053
28.65
0.1164
24.26
0.1552
23,99
DDPM-1000
0.2403
18.57
0.1663
19.34
0.2269
18.77
DDPM-250
0.2361
18.65
0.1734
19.05
0.2249
18.81
mid-
DDPM-100
0,2315
18.71
0.1640
19.30
0.2140
18,98
training
DDIM-50
0.4536
17.10
0.3098
17.62
0.4040
17.84
DDIM-50 X
0.2209
19.15
0.1618
19.97
0.2514
20.49
size of 16, costing 200 Tesla V100 GPU-hours. The performances are shown in Tab. We suspect that the reason for the boosting of the mid-training model is although the mid-training model not close fit to g y) yet, it has learned the extreme points of (xo (y). Thus, the assumptions corresponding to extreme points approximately hold (i.e., Eqn. Eqn. 17). So, we still can extract Yr based on Eqn. 20 and use it as approximately optimal BC other LR images y, getting better performances. We observe that DDIM-50 performs much worse than other sampling methods when applied the mid-training conclusion of applying these sampling methods in diffusion- & Dhariwal 2021). However, our method can still boost the DDIM-50 (i.e., the diffusion ODE sol used in the paper) with the approximately optimal BC Xy, reaching comparable performances with DDPM-based sampling methods.
frffusiio-bbeedd SR model. Such phenomenon is in conflict with the
image generation models (Nichol
Table 6: Pearson's coefficients between 10 LPIPS sequences of 100 bicubic-SR images for each LR image generated by the baseline SR3 model.
LR-1
LR-2
LR-3
LR-4
LR-5
LR-6
LR-7
LR-8
LR-9
LR-10
LR-1
1.000
0.754
0.840
0.798
0.811
0.751
0.902
0.837
0.877
0.765
LR-2
0.754
1.000
0.811
0.789
0.832
0.702
0.831
0.775
0.812
0.717
LR-3
0,840
0.811
1.000
0.732
0.799
0.654
0.841
0.799
0.836
0.745
LR-4
0.798
0.789
0.732
1.000
0.756
0.699
0.855
0.801
0.793
0.732
LR-5
0.811
0.832
0.799
0.756
1.000
0.632
0.811
0.792
0.856
0.789
LR-6
0.751
0.702
0.654
0.699
0.632
1,000
0.721
0.734
0.611
0.704
LR-7
0.902
0.831
0.841
0.855
0.811
0.721
1.000
0.754
0.787
0.725
LR-8
0.837
0.775
0.799
0.801
0.792
0.734
0.754
1.000
0.813
0.786
LR-9
0.877
0.812
0.836
0.793
0.856
0.611
0.787
0.813
1.000
0.801
LR-10
0.765
0.717
0.745
0.732
0.789
0.704
0.725
0.786
0.801
1.000
E VALIDATION ON THE INDEPENDENCE OF po po TO
As we have stated in the Sec. 3.2 Po (x y)) is not related to the specific LR images y. In this section, we design an experiment show the related evidence. As we have mentioned in has the same shape as
the Sec. 3.3 we assume distance measurement function M (ho(Xr.y) (ha((XT,3 y)) and we use q(hg(XT, to approximate pa (ha((x. y)). So, given different LR im- ages y. if pe (bo((x,, same shape. Thus, we validate the shapes of M(h (xx(xxx Y:). 2;) of different y;. We randomly sam- ple 10 Mecubic-LR image pairs and 100 xy, then generate 100 SR images by the baseline SR3 model of each LR image and calculate their LPIPS, getting 10 LPIPS sequences. To evaluate the shapes of the 10 LPIPS sequences, calculate the Pearson correlation coefficients of every two sequences
are independent, the functions M (ho(XT, (x), 2;) of X7 should have the
16

---

Published as a conference paper at ICLR 2024
GT
SR with Xr1
SR with
Biotic
Figure 4: SR results with shared xy. Results with Xf1 all have excessive artifacts and results with XT2 are all over-smooth. Results with shared xy share visual features. [Zoom in for best view]
and form a matrix shown in Tab. 6 can be seen that the coefficients are all high, indicating the strong correlation between different LPIPS sequences. To visualize the correlation between SR sults different LR images y., we further exhibit several SR images sharing the same Xy in Fig. [ be seen that SR images of different LR images with the same X have similar visual features. SR results with XT seem over-sharp and contain excessive artifacts while SR results with XT2 seem over-smooth. All of them are reasonable but satisfying enough, indicating the necessity of finding an approximately optimal BC Xr.
should be noticed that this experiment only validates that the consistency of shapes of M (X Y;),, z;), which is an derivation of the independence of po(ho(Xr, to y, instead of the independence itself.
MORE VISUAL RESULTS
In this section, we show more visual results of bicubic-SR compared with ESRGAN (Wang al. 2018) (which is the representative of GaNebased methods) in Fig Fig. and Fig. demonstrating superiority of our method in perceptual quality.
Ground Truth
Bicttic
ESRGAN
Figure 5: Further visual comparisons. [Zoom in for best view]
17

---

Published as a conference paper at ICLR 2024
Ground Truth
Bicubic
ESRGAN
OurS
Figure 6: Further visual comparisons. [Zoom in for best view]
18

---

Published as a conference paper at ICLR 2024
Ground Truth
Bicubic
ESRGAN
OwIs
lt
LOIEE 3F
Regua 1.000
Figure 7: Further visual comparisons. [Zoom in for best view]
19


Reviews:
[{"review_id": "UwxeRQboaW", "reviewer": "Reviewer_4gay", "summary": "This paper focuses on the randomness in the inverse process of the diffusion model applied to super-resolution tasks, which makes it difficult to ensure the quality of SR results. By solving diffusion ordinary differential equations with optimal boundary conditions, the authors propose an efficient plug-and-play method that enables diffusion models to stably sample high-quality SR images with fewer sampling steps.\n\nPost rebuttal:\nI have read the rebuttal and would like to raise my score a little bit.", "questions": "Further supplement and improve the experiment, please refer to the Weaknesses for details.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe proposed method achieves good visualization results with fewer steps in SR tasks. \n2.\tThe proposed method has good flexibility and can be applied to multiple diffusion models", "weaknesses": "1.\tThe experiments are insufficient. Although the proposed method has achieved good results on LPIPS, the PSNR values on multiple test sets are very low. The author did not discuss it in depth and did not show the results of SSIM.\n2.\tThe paper stated that the proposed method has fewer parameters and is more efficient than the GAN method, but did not show the corresponding comparison results. Such as comparison of specific parameters or running time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kzi9c0ptKp", "reviewer": "Reviewer_8EEt", "summary": "The paper analyzes the characters of boundary conditions (BCs) in the diffusion-ODE sampling process of diffusion-based Super-Resolution (SR) models and finds out that the optimal BC is shared by different LR images approximately. Based on the analysis, the paper further proposes a method of obtaining an approximately optimal BC based on a reference LR-HR set. The derivation of the paper is mathematically complete. Experiments on the tasks of both bicubic-SR and real-SR demonstrate the superiority of the proposed approximately optimal BC.", "questions": "1. The paper only discusses the context of SR (and other low-level tasks in the Sec. 5). But it seems that the theory does not limit the relationship between guidance and generated results. Do the authors think that the method can be leveraged in more general generation tasks such like text-to-image generation?\n2. It seems that the improvement on StableSR (in the task of real-SR) is less than the improvement on SR3 (in the task of bicubic-SR). Could the authors discuss the reasons for this phenomenon?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The motivation of analyzing the BCs of diffusion ODEs is intuitive. Different BCs would apparently lead to different SR results. It is vital to find the rule of how BCs affect the results and to propose a method to get a better BC.\n2. The analysis of the paper is mathematically complete. The paper proposes the concept of optimal BC and proves that such an optimal BC is common to different LR images. The conclusion is seemingly solid.\n3. The experiments are sufficient enough to support the theory. The paper claims that the method of obtaining the approximately optimal BS is not related to the degradation model. Experiments on the tasks of bicubic-SR and real-SR demonstrate the assertion. Further ablation studies show the influence of reference set and the set of BCs.", "weaknesses": "1. The proposed method leverages LPIPS as the implementation of distance measurement function M(·,·). Can we leverage pixel-level metrics like negative PSNR as M? The authors should give more discussions.\n2. The paper assumes that the model is well-trained. However, even a “well-trained” model cannot fit the real data distribution precisely. Thus, does such an assumption cause potential inaccuracy of the conclusion?\n3. The paper claims that p_θ (y|h_θ (x_T,ϕ)) is approximately uniform. However, what ensures that the model does not have biases when leveraging the “blank token”, which is essentially a placeholder token (that doesn't actually exist)?\n4. It seems that the process of calculating the approximately optimal BC is time-consuming. How long does it take?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QIsnQomkd0", "reviewer": "Reviewer_34bX", "summary": "This paper propose to steadily sample high-quality SR images from pre-trained diffusion-based SR models by solving diffusion ordinary differential equations with optimal boundary conditions.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper seems well-written", "weaknesses": "1 The methods compared in Table 2 are all outdated. It is necessary to compare them with some state-of-the-art real-world super-resolution tasks [1,2].\n\n2 You need to compare with the state-of-the-art diffusion SR method [3], which also appears to have performed bicubic-SR and real SR tasks.\n\n3 From the figures in the appendix, it seems that the visual improvement is not significant.\n\n4 Please provide a comparison of the computational complexity and runtime for all the methods mentioned in the paper to show your effectiveness.\n\n5 Tables 1 and 2 show that the PSNR is not particularly high, indicating that the network's fidelity is not good. Super-resolution tasks not only seek visual improvement but also place great importance on fidelity. Compared methods have better fidelity. Therefore, I suggest the authors work on improving both PSNR (fidelity) and LPIPS, as this would provide stronger evidence of the effectiveness of your method.\n\n\n[1] Real-esrgan: Training real-world blind super-resolution with pure synthetic data\n\n[1] Knowledge Distillation based Degradation Estimation for Blind Super-Resolution\n\n[2] Diffir: Efficient diffusion model for image restoration", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)