One-Step Effective Diffusion Network for Real-World  Image Super-Resolution
Authors: Rongyuan Wu
Lingchen Sun
Zhiyuan Ma
Lei Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: super-resolution; diffusion model; low-level vision
Primary area: machine_vision

Abstract: The pre-trained text-to-image diffusion models have been increasingly employed to tackle the real-world image super-resolution (Real-ISR) problem due to their powerful generative image priors. Most of the existing methods start from random noise to reconstruct the high-quality (HQ) image under the guidance of the given low-quality (LQ) image. While promising results have been achieved, such Real-ISR methods require multiple diffusion steps to reproduce the HQ image, increasing the computational cost. Meanwhile, the random noise introduces uncertainty in the output, which is unfriendly to image restoration tasks. To address these issues, we propose a one-step effective diffusion network, namely OSEDiff, for the Real-ISR problem. 
We argue that the LQ image contains rich information to restore its HQ counterpart, and hence the given LQ image can be directly taken as the starting point for diffusion, eliminating the uncertainty introduced by random noise sampling. We finetune the pre-trained diffusion network with trainable layers to adapt it to complex image degradations. To ensure that the one-step diffusion model could yield HQ Real-ISR output, we apply variational score distillation in the latent space to conduct KL-divergence regularization. As a result, our OSEDiff model can efficiently and effectively generate HQ images in just one diffusion step. 
Our experiments demonstrate that OSEDiff achieves comparable or even better Real-ISR results, in terms of both objective metrics and subjective evaluations, than previous diffusion model-based Real-ISR methods that require dozens or hundreds of steps. The source codes are released at https://github.com/cswry/OSEDiff.

One-Step Effective Diffusion Network for Real-World Image Super-Resolution
Rongyuan Wu Lingchen Sun 1.2." Zhiyuan Ma Lei Zhang Z 1,2,!
'The Hong Kong Polytechnic University
OPPO Research Institute
(rong-yuan wu, ling-chen.sun, zm2354.ma| (aaemaaaaaa poluuec,,,ollu.kkk solzhang@comp.peer comp-polyu.edu.ht
Equal contribution
'Corresponding author
Abstract
The pre-trained text-to-image diffusion models have been increasingly employed to tackle the real-world image super-resolution (Real-ISR) problem due to their powerful generative image priors. Most of the existing methods start from random noise to reconstruct the high-quality (HQ) image under the guidance of the given low-quality (I-Q) image. While promising results have been achieved, such Real- ISR methods require multiple diffusion steps to reproduce the HQ image, increasing the computational cost. Meanwhile, the random noise introduces uncertainty the output, which is unfriendly image restoration tasks. To address these issues, we propose a one-step effective diffusion network, namely OSEDiff, for the Real- ISR problem. We argue that the LQ image contains rich information restore its HQ counterpart, and hence the given LQ image can be directly taken as the starting point for diffusion, eliminating the uncertainty introduced by random noise sampling. We finetune the pre-trained diffusion network with trainable layers to adapt it complex image degradations. To ensure that the one-step diffusion model could yield HQ Real-ISK output, we apply variational score distillation the latent space conduct KL.-divergence regularization. As result, OSEDiff model can efficiently and effectively generate HQ images in just one diffusion step. Our experiments demonstrate that OSEDiff achieves comparable or even better Real-ISR results, in terms of both objective metrics and subjective evaluations, than previous diffusion model-based Real-ISR methods that require dozens or hundreds of steps. The source codes are released at https://gthub..com//www/////
1 Introduction
Image super-resolution (ISR) 13 66 29 65
6
46 27 61 is a classical yet still active research 24
problem, which aims restore a high-quality (HQ) image from its low-quality (L.Q) observation suffering from degradations of noise, blur and low-resolution, etc. While one line of ISR research 29 65] simplifies the degradation process from HQ to LQ images as bicubic downsampling (or downsampling after Gaussian blur) and focus on the study network architecture design, the trained models can hardly be generalized to real-world LQ images, whose degradations are often unknown and much more complex. Therefore, another increasingly popular line of ISR research the so-called real-world ISR (Real-ISR) [6] 45 problem, which aims to reproduce perceptually realistic HQ images from the LQ images captured in real-world applications.
13 66
There are two major issues training Real-ISR model. One is how to build the LQ-HQ training image pairs, and another is how to ensure the naturalness of restored images, i.e., how to ensure that the restored images follow the distribution of HQ natural images. For the first issue, some researchers
This work is supported by the PolyU-OPPO Joint Innovation Lab.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

LPPS
DIGTS
SCIM
Ch
Borser
SieSit
Deftsii-50 50
RD
Statrish-200
PSAA
Seed8-50
PKSD-20
PKSD
Batter
MUSI
CLPIAA
OSEDI!
MANIO
(A) Perfecmancc compartion
Efficiency compartion ison
Figure I: Performance and efficiency comparison among SD-based Real-ISR methods. (a). Perfor- mance comparison on the DrealSR benchmark [51]. Metrics like LPIPS and NIQE, where smaller scores indicate better image quality, are inverted and normalized for display. OSEDiff achieves leading scores on most metrics with only one diffusion step. (b). Model efficiency comparison. The inference time is tested on an A100 GPU with 512 512 imput image size. OSEDiff has the fewest trainable parameters and is over 100 times faster than StableSR [4
have proposed to collect real-world LQ-HQ image pairs using long-short camera focal lenses 51]. However, this is very costly and can only cover certain types of real-world image degradations. Another more economical way is to simulate the real-world LQ-HQ image pairs by using complex image degradation pipelines. The representative works include BSRGAN [6]] and Real-ESRGAN [4). where a random shuffling of basic degradation operators and a high-order degradation model are respectively used to generate LQ-HQ image pairs.
With the given training data, how to train a robust Real-ISR model to output perceptually natural images with high quality becomes key issue. Simply learning mapping network between LQ-HQ paired data with pixel-wise losses can lead to over-smoothed results 24 46). It is crucial to integrate natural image priors into the learning process to reproduce HQ images. A few methods have been proposed to this end. The perceptual loss [8] explores the texture, color, and structural priors in a pre-trained model such as VGG-16 38 and AlexNet 23 The generative adversarial networks (GANs) [4] alternatively train generator and discriminator, and they have been adopted for Real-ISR tasks (24 46 45
|7 (33. The generator network aims to synthesize HQ images, while the discriminator network aims to distinguish whether the synthesized image is realistic or not. While great successes have been achieved, especially in the restoration of specific classes of images such as face images [56] [4]. GAN-based Real-ISR tends to generate unpleasant details due to the unstable adversarial training and the difficulties in discriminating the image space of diverse natural scenes.
61
The recently developed generative diffusion models (DM) [39 [6]. especially the large-scale pre- trained lext-to-image (T2I) models [37 36], have demonstrated remarkable performance in various downstream tasks. Having been trained on billions of image-text pairs, the pre-trained T2I models possess powerful natural image priors, which can be well exploited to improve the naturalness and perceptual quality of Real-ISR outputs. Some methods 57 31 52 40 59] have been developed to employ the pre-trained T2I model for solving the Real-ISR problem. While having shown impressive results in generating richer and more realistic image details than GAN-based methods, the existing SD-based methods have several problems to be further addressed. First, these methods typically take random Gaussian noise as the start point of the diffusion process. Though the LQ images are used as the control signal with a ControlNet module (63), these methods introduce unwanted randomness in the output HQ images [40]. Second, the restored HQ images are usually obtained by tens or even hundreds of diffusion steps, making the Real-ISR process computationally expensive. Though some one-step diffusion based Real-ISR methods 48 have been recently proposed, they fail in achieving high-quality details compared to multi-step methods.
To address the aforementioned issues, we propose a One-Step Effective Diffusion network, OSEDiff' in short, for the Real-ISR problem. The UNet backbone pre-trained SD models has strong capability to transfer the input data into another domain, while the given LQ image actually has rich information restore its HQ counterpart. Therefore, we propose to directly feed the IQ images into the pre-trained SD model without introducing any random noise. Mcanwhile, integrate trainable

---

LoRA layers [17] into the pre-trained UNet, and finetune the SD model to adapt it to the Real-ISR task. On the other hand, to ensure that the one-step model can still produce HQ natural images as the multi-step models, we utilize variational score distillation (VSD) 44
58 10 for KI KI--divergence
regularization, This operation effectively leverages the powerful natural image priors of pre-trained SD models and aligns the distribution of generated images with natural image priors. As illustrated in Fig. | our extensive experiments demonstrate that OSEDiff achieves comparable or superior performance measures to state-of-the-art SD-based Real-ISR models, while it significantly reduces the number of inference steps from N to I and has the fewest trainable parameters, leading to more than x 100 speedup in inference time over previous methods such as StableSR [4
2 Related Work
Starting from SRCNN [3]. deep tearning-based methods have become prevalent for ISR. A variety of methods have been proposed (30] 66
6 7 to improve the accuracy of ISR reconstruction. However, most of these methods assume simple and known degradations such as bicubic downsampling, limiting their applications to real-world images with complex and unknown degradations. In recent years, researches have been exploring the potentials of generative models, including GAN [14] and diffusion networks [16], for solving the Real-ISR problem.
67
9
5
29
65
GAN-based Real-ISR. The use of GAN for photo-realistic ISR be traced back to SRGAN 24 where the image degradation is assumed to be bicubic downsampling. Later on, researchers found that GAN has the potential perform real-world image restoration with more complex degradations 45.. Specifically, by using randomly shuffled degradation and high-order degradation to generate more realistic LQ-HQ training pairs, BSRGAN [T] and Real-ESRGAN 45] demonstrate promising
Real-ISR results, which trigger many following works 4 27 28 5
DASR [2]] designs tiny
network predict the degradation parameters to handle degradations of various levels. SwinIR
[29 switches the generator from CNNs to stronger transformers. further enhancing the performance of Reall-ISR However, the adversarial training process in GAN is unstable and its diseriminator is limited in telling the quality of diverse natural image contents. Therefore, GAN-base Real-ISR methods often suffer from unnatural visual artifacts. Some works such as L.L. 27 and DeSRA (53 can suppress much the artifacts, yet they are difficult to generate more natural details.
Diffusion-based Real-ISR. Some early attempts (21 20 47 employ the denoising diffusion proba- bilistic models (DDPMs) 16 39 [ to address the ISR problem by assuming simple and known degradations (e.g.. bicubic downsampling). These methods are training-free by modifying the reverse transition of pre-trained DDPMs using gradient descent, but they cannot be applied to complex unknown degradations. Recent researches [4] 57 40 59] have leveraged stronger pre-trained T2I models, such as Stable Diffusion (SD) [. to tackle the Real-ISR problem. In general, they introduce an adapter [ to fine-tune the SD model to reconstruct the HQ image with the I.Q image as the control signal. StableSR 42 finetunes a time-aware encoder and employs feature warping to balance fidelity and perceptual quality. PASD [57] extracts both low-level and high-level features from the LQ image and inputs them to the pre-trained SD model with a pixel-aware cross attention module. To further enhance the semantic-aware ability of the Real-ISR model, SeeSR [52] introduces degradation-robus! tag-style text prompts and utilizes soft prompts to guide the diffusion process. To mitigate the potential risks of diffusion uncertainty, CCSR [40 leverages a truncated diffusion process to recover structures and finetunes the VAE decoder by adversarial training to enhance details. SUPIR (59] leverages the powerful generation capability of SDXL model and the strong captioning to synthesize rich image details.
32 capability of LLaVA
The above mentioned methods, however, require tens or even hundreds of steps to complete the diffusion process, resulting in unfriendly latency. SinSR shortens ResShift [60] to single-step inference by consistency preserving distillation. Nevertheless, the non-distrbution--asee distillation loss tends to obtain smooth results, and the model capacity of SinSR and ResShift are much smaller than the SD models to address Real-ISR problems.
Methodology
3.1 Problem Modeling
Real-ISR is to estimate an HQ image ग from the given LQ image ær, This task can be convention- ally modeled as an optimization problem: t. = argming, æ (Lattaa (*(H)) .r.)) + >hreg (# where & is the degradation function, Ldata is the data term to measure the fidelity of the optimization
3

---

Pronst Extraetor
Text
Gradient
Trainable
Model Initialization
Frozen
x2
2,
Encoder
VAF
Dillusion
Finetumed
Network
Regularizer
E
Finelune
random degrndation
Pretmineed
Add
Reg algaiiie
De
X
Figure 2: The training framework of OSEDiff. The LQ image is passed through trainable encoder Eg, a LoRA finetuned diffusion network Eo and frozen decoder Dg to obtain the desired HQ image. In addition, text prompts extracted from the LQ image and input to the diffusion network to stimulate its generation capacity. Mcanwhile, the output the diffusion network Eş will sent to two regularizer networks (a frozen pre-trained one and fine-tuned onc). where variational score distillation performed latent space to ensure that the output of eg follows HQ natural image distribution. The regularization loss will be sockppropggated to update Eg and Co. Once training finished, only Eg. Eg and Do will be used in inference.
output, Greg is the regularization term exploit the prior information of natural images, and scalar is the balance parameter. Many conventional ISR methods [3 by assuming simple and known degradation models and employing hand-crafted natural image prior models (i.e., image sparsity based prior [54])
29 65 restore the desired HQ image
However, the performance of such optimization-basse methods is largely hindered by two factors. First, the degradation function is often unknown and hard to model real-world scenarios. Second, the hand-crafted regularization terms Lrog are hard effectively model the complex natural image priors. With the development of deep-learning techniques, it has become prevalent to learn a neural network Go, which is parameterized 8, from training dataset of (x pairs to map the LQ image to an HQ image. The network training can be described as the following learning problem:
ft argmin, (x) Laatt (Go(æ1), æH) + Xhreg (Go(
(1)
where Laata and freg are the loss functions. dasta enforces that the network output क. Go(az) approach to the ground-truth æ much as possible, which can be quantified by metrics such as L1 norm, Le norm and LPIPS [64]. Using only the Ldsta loss to train the network Gg from scratch may over-fit the training dataset. In this work, we propose finetune pre-trained generative network, more specifically the SD (6) network, to improve the generalization capability of Gø. In addition, the regularization loss freg is critical to improve the generalization capability of Gg. as well as enhance the naturalness of output HQ images h Suppose that we have the distribution of real-world HQ images, denoted by p) the KL divergence [8] is an ideal choice to serve the loss function of Lregi that is, the distribution of restored HQ images, denoted by ( should be identical p( much as possible. The regularization loss can be defined as:
Ling DKT. (go (m )lp((=)))
(2)
mostly instantiate the above objective via adversarial training [14. which involves learning a discriminator to differentiate between the generated HQ image λ and the real HQ image RH, and updating the generator Ga to make 2 and æ indistinguishable. However, discriminators are often trained from scratch alongside the generator. They may not be able to acquire full distribution of HQ images and lack enough discriminative power, resulting sub-optimal Real-ISR performance.
Existing works 24 46

---

The recently developed T2I diffusion models such as SD [36 offer new options for us to formulate the loss Lirg. These models, trained on billions of image-text pairs, can effectively depict the natural image distribution in latent space. Some score distillation methods have been reported to employ SD to optimize images by using the KL-ddivergence as the objective [49 25
particular, variational score distillation (VSD) 49, 58, [0] induces such Kl.-divergence based objective from particle- based variational optimization to align the distributions represented by two diffusion models. Based on the above discussions, we propose to instantiate the learning objective in Eq. ① by designing an efficient and effective one-step diffusion network. In specific, we finetune the pre-trained SD with LoRA [17] as our Real-ISR backbone network and employ VSD as regularizer to align the distribution of network outputs with natural HQ images. The details are provided in the next section.
3.2
One-Step Effective Diffusion Network
Framework Overview. As discussed in Sec. I the existing SDD-ased Real-ISR methods 22
perform multiple timesteps to estimate the HQ image with random noise as the starting point and the LQ image as control signal. These approaches are resource-intensive and will inherently introduce randomness. Based on our formulation in Sec. [.1] we propose one-step effective diffusion (OSEDiff) network for Real-ISR, whose training framework is shown in Fig. 2 Our generator Go to be trained composed of trainable encoder Eg. finetuned diffusion network Co and frozen decoder Dg. To ensure the generalization capability of Go. the output of the diffusion network Eg will be sent to two regularizer networks, where VSD loss is performed in latent space. The regularization loss are back-propagated to update Eg and ég. Once training is finished, only the generator Go will be used inference. In the following, we will delve into the detailed architecture design of OSEDiff, as well as its associated training losses.
57
31
52 40]
Let's denote by Eg, Eg and D3 the VAE encoder, latent diffusion network and VAE decoder of pretrained SD model, where & denotes the model parameters. Inspired by the recent success of LoRA [17 in finetuning SD downstream tasks 34 351, we adopt LoRA to fine-tune the pre-trained SD in the Real-ISR task to obtain the desired generator Gg.
Network Architecture Design.
As shown in the left part of Fig- 2 to maintain SD's original generation capacity, we introduce trainable LoRA [17 layers to the encoder E,, and the diffusion network cs. finetuning them into Eg and eg with our training data. For the decoder, we fix its parameters and directly set Da Da- This is to ensure that the output space of the diffusion network remains consistent with the regularizers.
Recall that the diffusion model diffuses the input latent feature through 2; = a; Bic, where ae; 3e are scalars that are dependent to diffusion timestep E {1,-.... ,T} [6]. With a neural network that can predict the noise in 2,, denoted as é, the denoised latent can be obtained as & = =.- which is expected to be cleaner and more photo-realistic than 2,. Moreover, SD is a text-conditioned generation model. By extracting the text embeddings (36), denoted by Cy- from the given text description y, the noise prediction can be performed as é = t,,
We adapt the above rext-to-image denoising process to the Real-ISR task, and formulate the LQ-to-HQ latent transformation Fg as a text-conditioned image-to-image denoising process as:
2 = Fo(z 11 - Bree(zz;TToo
aT
where we conduct only one-step denoising on the LQ latent zt, without introducing any noise, at the T'-t diffusion timestep. The denoising output स expected be more photo-realistic than zt. As for the text embeddings, we apply some text prompt extractor, such as the DAPE (52), to LQ input Lg, and obtain cy Y.)L). Finally, the whole LQ-to-HQ image synthesis can be written as:
z = Go( & Dg(FF ((z))))
As mentioned in Sec. 3.1 S.1 to improve the performance for a Real-ISR model, it is required to supervise the generator training with both the data term Lasta and regularization term Creg. As shown in the right part of Fig. 2 we propose to adapt VSD [4] as the regularization term. Apart from utilizing the SD model as the pre-trained regularizer cg. VSD also introduces a finetuned regularizer, i.e., latent diffusion module finetuned on the distribution go (æn) ) of generated images with LoRA. We denote this finetuned diffusion module as ey.
5

---

Training Loss. We train the generator Go with the data loss Ldata and regularization loss Lreg. We Cdata as the weighted sum of MSE loss and LPIPS loss:
ddata (L), æH) MSE (Go( æH) CLLLPIPS (G)(Ø1),
(5)
where A1 is weighting scalar. As for Lieg. adopt the VSD loss via:
Lreg (=L) = LVSD (Go(zL), Cy) = LVSD (Go(æ1), M(LL)).
(6)
Given any trainable image-shape feature æ. its latent code x (x) and encoded text prompt condition Cy. VSD optimizes to make it consistent with the text prompt via:
dz
VæLVsD (x, cy)) Er,e w(t) (es(zeit, E (z2, 1i cy))
(7)
dx
where the expectation of the gradient conducted over all diffusion timesteps E (I,'., , and NO, I). Therefore, the overall training objective for the generator Gø is:
C (G)( æH) = Ldata (G æH) + X2Creg (Go(#1))
(8)
where X2 is a weighting sealar. Besides, as required by VSD, the finetuned regularizer es should also be trainable, and its training objective is:
Caift
CMSE (Eær (042 + Bie; L, .e).
(9)
Note that the above Laitt loss is only applied to update ezy The whole algorithm to illustrate the training pipeline can be found in the Appendix.
VSD in Latent Space. The original VSD computes the gradients in the image space. When it is used to train an SD-based generator network, there will be repetitive latent decoding/encodiig in computing Greg- This is costly and makes the regularization less effective. Considering the fact that well-trained VAE should satisfy Eg = (Ds R z, we can approximately let ( = & In this case, we eliminate the redundant latent encoding/decoding in computing the regularization loss, as we follow DMD (88) to optimize the distribution loss in the latent state space rather than in the noise space. The gradient of the regularization loss w.r.t. the network parameter & in the latent space is:
VeLvsD(Go(zL).c)) = Lv cy)
00
02 dā h
Er, = (2N) +he w(()) (Ea 66 (
Dā.
00
- (1(1) (c6(22; (Zrit,,
02
00
(10)
Experiments
4.1 Experimental Settings
employed different training datasets, making unified training standards for Real-ISR difficult to establish. For simplicity, we adopt SeeSR's setup (22) and train OSEDiff using the L.DIR (26) dataset and the first 10K face images from
Training and Testing Datasets. Prior works 42 57 31 52
FFHQ 19 The degradation pipeline of Real-ESRGAN [45] is used to synthesize LQ-HQ training
pairs. We evaluate OSEDiff and compare it with competing methods using the test set provided by StableSR [42], including both synthetic and real- world data. The synthetic data includes 3000 images of size 512 x 512, whose GT are randomly cropped from DIV2K-Val [2) and degraded using the Real-ESRGAN pipeline 45 The real-world data include LQ-HQ pairs from RealSR and
DRealSR [5T] with sizes of 128 128 and 512 x 512, respectively.
Compared Methods. We compare OSEDiff with state-of-the-an DM-based Real-ISR methods, including StableSR [4). ResShift [60), PASD 57 DiffBIR [31] SeeSR [52 and SinSR [48] Among them, StableSR, PASD, DiffBIR, and SeeSR are all based on the pre-trained SD model. ResShift trains DM from scratch in the pixel domain, while SinSR is one-step model distilled

---

Table 1: Quantitative comparison with oftte-of-thearrr methods on both synthetic and real-world benchmarks. 's' denotes the number of diffusion reverse steps in the method. The best and second best results of each metric are highlighted in red and blue, respectively.
Datasets
Methods
PSNRT
SSIMT
L.PPS.
DISTSL
FID
NIQRI
MUSIQU
MANIQAT
CLIPIQAT
StableSR-4200
23.26
0.5726
0.3113
0.2048
24.44
4.7581
65.92
0.6192
0.6771
DiffBRR-s50
23.64
0.5647
0.3524
0.2128
30.72
4.7042
65.81
0.6210
0.6704
SeeSR-s50
23 68
0.6043
0.3194
0.1968
25.90
4,8102
68.67
0.6240
PASD=s20
23.14
0.5505
0.3571
0.2207
29.20
4.3617
68.95
0.6483
0.6788
DIV2K-Vil Val
ResShift-s15
24.65
0.6181
0.3349
0.2213
36.11
6.8212
61.09
0,5454
0.6071
SinSR-s1
24.41
0.6018
0.3240
0.2066
35.57
6.0159
62.82
0.5386
0.6471
OSEDifE-s1
23.72
0.6108
0.2941
0.1976
26,32
4,7097
67.97
0.6148
0.6683
StableSR-s200
28.03
0.7536
0.3284
0,2269
148.98
6.5239
58.51
0,5601
0.6356
DifTBRR-s5
0.6571
0.4557
0.2748
166.79
6.3124
61.07
0,5930
0.6395
SecSR-<50
28.17
0.7691
0,3189
0.2315
147.39
6.3967
64,93
0.6042
0.6804
PASD-s20
27.36
0.7073
0,3760
0.2531
156.13
5.5474
64.87
0.6169
0.6808
DrealSR
ResShift-s15
28.46
0.7673
0,4006
0,2656
172.26
8.1249
50,60
0,4586
0.5342
SinSR-s1
28.36
0.7315
0,3665
0,2485
170.57
6.9907
55,33
0.4884
0.6383
OSEDifI-sI
0.7835
0.2968
0.2165
135.30
64.65
0.5899
0.6963
StableSR-s200
24.70
0.7085
0,3018
0.2288
128.51
5.9122
65,78
0.6221
0.6178
DitBBR--50
24.75
0.6567
0,3636
0.2312
128.99
5,5346
64,98
0.6246
0.6463
SeeSR-c50
0.2223
125.55
0.6442
PASD-s20
25.21
0.6798
0,3380
0.2260
124.29
5.4137
68,75
0.6487
0.6620
RealSR
ResShift-s15
26.31
0.7421
0,3460
0.2498
141.71
7.2635
58.43
0,5285
0.5444
SinSR-s1
26.28
0.7347
0.3188
0.2353
135.93
6,2872
60,80
0,5385
0,6122
OSEDifE-sl
25.15
0.7341
0.2921
0.2128
123,49
5.6476
69.09
0.6326
0.6693
Table 2: Complexity comparison among different methods. All methods are tested with an input image of size 512 x 512, and the inference time is measured on an A100 GPU.
StableSR
DifIBIR
SeeSR
PASD
ResShift
SinSR
OSEDiff
Inference Step
200
50
50
20
Inference Time (s)
11.50
2.72
4.30
2,80
0.71
0.13
0.11
MACs (G)
79940
24234
65857
29125
5491
2649
2265
Total Param (M)
1410
1717
2524
1900
119
119
1775
# Trainahle Param (M)
150.0
380.0
749.9
625.0
118.6
118.6
8.5
from ResShift. Note that we do not compare with the recent method SUPIR 59
because it tends to
generate rich yet excessive details, which are however unfaithful the input image.
For those GAN-based Real-ISR methods, including BSRGAN 61 Real-ESRGAN and FeMaSR we put their results into the Appendix.
45 LDI. 27
Evaluation Metrics. To provide a comprehensive and holistic assessment on the performance of different methods, we employ range of full-reference and no-reference metrics. PSNR and SSIM 50 (calculated on the Y channel in YCbCr space) are reference-based fidelity measures, while eeference-basee perceptual quality measures. FID [13] evaluates the distance of distributions between GT and restored images. NIQE [62,, MANIQA-pipaa [55], MUSIQ [2), and CLIPIQA 41 no-reference image quality measures. We also conduct a study,
LPIPS [64], DISTS [12]
which is presented in the Appendix.
Implementation Details. We train OSEDiff with the Adam W optimizer 333 at learning rate of 5e-5. The entire training process took approximately day on 4 NVIDIA A100 GPUs with a batch size of 16. The rank of LoRA in the VAE Encoder, diffusion network, and finetuned regularizer is set to 4. For the text prompt extractor, although advanced multimodal language models [32] can provide detailed text descriptions, they come at high inference cost. We adopt the degradation-awart prompt extraction (DAPE) module in SeeSR [52] to extract text prompts. The SD 2.1-base is used as the pre-trained T2I model. The weighting scalars x and X are set to 2 and 1, respectively.
4.2 Comparison with State-of-the-Arts
Quantitative Comparisons. The quantitative comparisons among the competing methods on the three datasets are presented in Table We can have the following observations. (1) First, OSEDiff exhibits clear advantages over competing methods in full-reference perceptual quality metrics LPIPS and DISTS, distribution alignment metric FID, and semantic quality metric CLIPIQA, especially
7

---

Zoomed LQ
Stable5R-2200
DXFBBU--500
SecSR-150
ResShiff-s15
SinSR-st
OSEDiff-sl
Zoomed 1.42
StableS 2206
1XfiBIR-20
Sec8R-
PASD-220
ResShull-115
Su8Rk-l
OSEDiE-l
Figure 3: Qualitative comparisons of different Real-ISR methods. Please zoom in for a better view.
on the two real-world datasets DrealSR and RealSR. (2) Second, SeeSR and PASD show better no-reference metrics like NIQE, MUSIQ and MANIQA. This is because these multi-step methods can produce rich image details in the diffusion process, which are preferred by no-reference metrics. (3) Third, ResShift and its distilled version SinSR show better full-reference fidelity metrics such as PSNR. This is mainly because they train a DM from scratch specifically for the restoration purpose, instead of exploring the pre-trained T2I model such as SD. However, ResShift and SinSR show poor perceptual quality metrics than other methods.
Qualitative Comparisons. Fig. 3 presents visual comparisons of different Real-ISR methods. As illustrated the first example, ResShift and SinSR severely blur the facial details due to the lack of pre-trained image priors. StableSR, DiffBIR and SeeSR reconstruct more facial details by exploring the image prior in pre-trained SD model. PASD generates excessive yet unnatural details. Though OSEDiff performs only one step forward propagation, it reproduces realistic and superior facial details to other methods. Similar conclusion can be drawn from the second example. StableSR and DifIBR are limited in generating rich textures due the lack of text prompts. PASD suffers from incorrect semantic generation because its prompt extraction module is not robust to degradation. While SeeSR utilizes degradation-awarc semantic cues stimulate image generation priors, the generated leaf veins are unnatural, which may be influenced by its random noise sampling. In contrast, OSEDiff can generate detailed and natural leaf veins. More visualization comparisons and the results of subjective user study can be found in the Appendix.
Complexity Comparisons. We further compare the complexity of competing DM-based Real-ISR models in Table including the number of inference steps, inference time, and trainable parameters. All methods tested A 100 GPU with input image of size 512 512. OSEDiff has the fewest trainable parameters, and the trained LoRA layers can be merged into the original SD to further reduce the computational cost. By using only one forward pass, OSEDiff has significant advantage in inference time over multi-step methods. Specifically, OSEDiff demonstrates substantial speed advantage, being approximately 105 times faster than StableSR, 39 times faster than SeeSR, and 6 times faster than ResShift. When compared to the single-step step method SinSR, OSEDiff not only achieves faster inference but also delivers significantly higher output quality. terms of complexity, OSEDiff requires the lowest MACs at just 2265G, as it operates with only a single diffusion step. In contrast, methods like StableSR, which require 200 steps, incur substantially higher MACs (e.g., 79940G). Regarding trainable parameters, OSEDiff is highly parameter-efficient, requiring only 8.5M parameters (LoRA layers), compared models such as SeeSR, which necessitates 749.9M parameters. This highlights the efficiency of OSEDiff during the training process.

---

Table Comparison of different losses on the RealSR benchmark.
PSNR
I.PIPS]
MUSIQQ
CLIPIQAT
w/o VSD loss
25.42
0.2934
65.23
0,5876
GAN loss
25.00
0.2760
67.29
0.6254
VSD loss in image domain
25.05
0.2759
67.90
0.6256
OSEDif
25.15
0.2921
69.09
0.6693
Table 4: Comparison of different text prompt extractors on the DrealSR benchmark.
Prompt Extraction
Methods
LPPISS]
DISTS
FID
MUSIQ1
MANIQAT
CLIMOAT
Prompi Exuaction Time
0.7910
0.2896
0.2080
126.59
62.13
0.5782
0.6599
DAPF
0.7835
0.2968
0.2165
135.30
640022
0,5899
0.693
LLVA--1.5
27.72
0.7735
0,3109
0.2249
149.45
6,4199
65.70
0,6038
0,7033
3.53
Table 5: Comparison of LoRA in VAE en- coder with different ranks.
Table 6: Comparison of LoRA in UNet with different ranks.
Rank
PSNRT
DISTS
MUSIQI
NIQE
Rank
PSNR
DISTS
MUSIQT
NIQEJ
2
25.28
0.2154
68.89
5,7171
25.15
0.2128
69.09
5,6479
4
25.15
0.2128
69.09
5,6479
24.86
0.2134
68.37
5.8184
8
24.87
0.2115
68.39
5.8184
Table 7: Ablation studies on finetuning the VAE encoder and decoder on the RealSR benchmark.
Train VAE Encoder
Train VAE Decoder
PSNRT
DISTS)
LPIPS].
CLIPIQAT
MUSIQT
NIQE]
(1)
x x
x
25.27
0.1966
0.2656
0,5303
58.99
6,5496
(2)
V
25.30
0.2049
0.2829
0,5604
65.83
6.6291
(3)
V
V
25.59
0.2141
0.3017
0,5778
65,92
6,9845
OSEDiff
V
X
25.15
0.2128
0.2921
0.6693
69.09
5,6479
4.3 Ablation Study
Effectiveness of VSD Loss. To validate the effectiveness of our VSD loss latent space, we perform ablation studies by removing the VSD loss, replacing it with the GAN loss used in [35], and applying VSD loss the image domain. The results on the RealSR test set are shown Table 3 We can see that without using the VSD loss, the perceptual quality metrics are significantly degraded because it hard to ensure good visual quality using only MSE loss and even LPIPS loss 64 Using GAN loss and VSD loss in the image domain can improve the performance, but the results are not good as applying VSD loss in the latent domain. Our proposed OSEDiff can effectively align the distribution of Real-ISR outputs by performing VSD regularization in the latent domain.
Comparison on Text Prompt Extractors. We then conduct experiments to evaluate the effect of different text prompt extractors on the Real-ISR results. We test three options. The first option does not employ text prompts. The second option uses the DAPE module in SeeSR [52] extract degradation-awar aware tag-style prompts, as we used in our main experiments. The third option uses LLaVA-v1.5 1.5 as used in SUPIR (59). We retrain the models based on different prompt extraction methods. The ablation results are shown in Table
32 to extract long text descriptions after removing the degradation of input LQ images,
One can see that without using text prompts as inputs, those full-reference metrics such PSNR, SSIM, LPIPS, DISTS and even FID can improve, while those no-reference metries such MUSIQ. MANIQA and CLIPIQA become worse. By using DAPE and LLaVA to extract text prompts, the generation capability of the pre-trained T2I SD model can triggered, resulting richer synthesized details, which however can reduce the full-reference indices. A visual example is shown in Figure
We see that while LLaVA extracts significantly longer text prompts than DAPE, they produce a similar amount of visual details. However, it is worth mentioning that the MILM model LLaVA is very costly, increasing the inference time of DAPE by 170 times. Considering the cost-effectiveness, we ultimately choose DAPE as the text prompt extractor in OSEDiff.

---

1.Q Insae
HO Image (Nall prompt)
Imnge (Prowpp from DAP))
HQ Iiage (Prowep Liaswvll.))
Pranept from Llase vii.5
Prangt from DAPE
and pengeiin stteding flock whitee fradhurs charly viuibe, cneating sbrkking
poesilly rocks
pengivin positioned conder marig
Tioak, boock proguie, sesie, stard!
soane, with lotal huryend the oide biekground. The porgliis opecars do be looking at somethhin off to the siil, poasilly absarving iig its nurroesaing ingi or searching for food!"
Figure 4: The impact of different prompt extraction methods. Please zoom in for a better view.
Setting of LoRA Rank. When finetuning the VAE encoder and the UNet, we need to set the rank of LoRA layers. Here we evaluate the effect of different LoRA ranks on the Real-ISR performance by using the RealSR benchmark [3]. The results are shown in Tables Sables5Jandfe and respectively. As shown in T if too small LoRA rank, such as 2, is set for the VAE encoder, the training will be unstable and cannot converge. On the other hand, if a higher LoRA rank, such as 8, is used for the VAE encoder, it may overfit in estimating image degradation, losing some image details in the output, as evidenced by the PSNR, DISTS, MUSIQ and NIQE indices. We find that setting the rank to 4 can achieve balanced result for the VAE encoder. Similar conclusions
be made for the setting of LoRA rank on UNet. As can be seen from Table 6 a rank of 4 strikes a good balance. Therefore, we set the rank 4 for both the VAE encoder and UNet LoRA layers.
The Finetuning on the VAE Encoder and Decoder. We conducted ablation studies to examine the impact of finetuning the VAE encoder and decoder, as shown Table In the first row, where neither the VAE encoder nor decoder finetuned, the results show poor perception performance. Comparing with OSEDiff, where only the VAE encoder is finetuned, we observe significant improvements in perceptual quality (e.g., MUSIQ improves from 58.99 to 69.09). This demonstrates that finetuning the VAE encoder is important for removing degradation and enhancing overall performance. When comparing the third row, where both the VAE encoder and decoder are finetuned, with OSEDiff, where only the encoder is trained and the decoder is fixed, note that OSEDiff also achieves better perceptual quality (CLIPIQ improves from 0.5778 to 0.6693). This indicates that fixing the VAE decoder is important to ensure that the UNet output remains in the original VAE latent space, which helps minimizing the VSD loss more effectively. Thus, finetuning the VAE encoder is important to remove degradation, while fixing the VAE decoder helps maintaining stability in the latent space, leading to better perceptual quality.
5 Conclusion and Limitation
We proposed OSEDiff, a one-step effective diffusion network for Real-ISR, by utilizing the pre- trained text-to-image model as both the generator and the regularizer in training. Unlike traditional multi-step diffusion models, OSEDiff directly took the given LQ image as the starting point for diffusion, eliminating the uncertainty associated with random noise. By fine-tuning the pre-trained diffusion network with trainable LoRA layers, OSEDiff can well adapt to the complex real-world image degradations. Meanwhile, we performed the variational score distillation in the latent space to ensure that the model's predicted scores align with those of multi-step pre-trained models, enabling OSEDiff to efficiently produce HQ images in one diffusion step. Our experiments showed that OSEDiff achieved comparable or superior Real-ISR outcomes to previous multi-step diffusion- based methods in both objective metrics and subjective assessments. We believe exploration can facilitate the practical application of pre- pre-trained T2I models to Real-ISR tasks.
There are some limitations of OSEDiff. First, the details generation capability of OSEDiff can be further improved. Second, like other SD-based methods, OSEDiff is limited in reconstructing fine-scale structures such as small scene texts. We will investigate these problems in further work.
10

---

References
[1] Stability.ai. httpa//hhhaap////aaaa/////aa//aa
[2] Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super- Dataset and study. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pages 126-135, 2017.
[3] Jianrui Cai, Hui Zeng, Hongwei Yong. Zisheng Cao, and Lei Zhang. Toward real real-world single image super-resolution A new benchmark and a new model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3086-3095, 2019.
[4] Chaofeng Chen, Xinyu Shi, Yipeng Qin, Xiaoming Li, Xiaoguang Han, Tao Yang, and Shihui Guo. Real-world blind super-resolution via feature matching with implicit high resolution priors. In Proceedings of the 30th ACM International Conference on Multimedia, pages 1329 1338, 2022.
Hanting Chen, Yunhe Wang. Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12299-12310, 2021.
[6] Xiangyu Chen, Xintao Wang. Jiantao Zhou, Yu Qiao, and Chao Dong. Activating more pixels in image super-resolution transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 22367-22377 2023.
[7] Zheng Chen, Yulun Zhang. Jinjin Gu, Linghe Kong, Xiaokang Yang, and Fisher Yu. Dual aggregation transformer for image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12312-12321, 2023.
[8] Imre Csiszár. I-divergence geometry probability distributions and minimization problems. The annals of probubility, pages 146-158, 1975.
[9] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang. Second-order attention network for single image resouuteooluion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1065-11074 2019.
[10] Trung Dao, Thuan Hoang Nguyen, Thanh Lc, Duc Vu, Khoi Nguyen, Cuong Pham, and Anh Tran. Swift- brush v2: Make your one-step diffusion model better than its teacher. arXiv preprint 2xXx::2408..1700 2024.
[11] Prafulla Dhariwal and Alexander Nichol, Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780- 8794, 2021.
[12] Keyan Ding, Kede Ma, Shiqi Wang, and Ecro Simoncelli. Image quality assessment: Unifying structure and texture similarity. IEEE transactions on pattern analysis and machine intelligence, 44(5):2567-258 2020.
[13] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13. pages 184-199. Springer, 2014.
[14] lan Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[15] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[16] Jonathan Ho, Ajay Jain, and Pieter Abbecl. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 6851, 2020.
[17] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation large language models, arXiy preprint arXiv:2106. 09685. 2021.
[18] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 11 14, pages 694-711. Springer, 2016.
11

---

[19] Tero Karras, Samuli Laine. and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and patiemn recognition, pages 2019.
[20] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Demoising diffusion restoration models. Advances in Neural Information Processing Systems, 35:23593-23606 2022.
[21] Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochastically. Advances in Neural Information Processing Systems, 34:21757-21769, 2021.
[22] Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5148-5157, 2021.
[23] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[24] Christian Ledig, Lucas Theis, Ferene Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. mott-realisttic single image super- resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4681 14990, 2017.
[25] Kyungmin Lec, Kihyuk Sohn, and Jinwoo Shin. Dreamflow: High-quality text-to-3d generation by approximating probability flow. arXiv preprint arXiv:2403. 14966, 2024.
[26] Yawei Li, Kai Zhang. Jingyun Liang, Jiezhang Cao, Ce Liu, Rui Gong, Yulun Zhang, Hao Tang, Yun Liu. Denis Demandolx, et al. Lsdir: A large scale dataset for image restoration. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 1775-1787, 2023.
[27] Jie Liang. Hui Zeng, and Lei Zhang. Details or artifacts: A locally discriminative learning approach to realistic image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5657-5666, 2022.
[28] Jie Liang, Hui Zeng, and Lei Zhang. Efficient and degradation-adaptive network for weal-world image super-resolution. In European Conference on Computer Vision, pages 574-591. Springer, 2022.
[29] Jingyun Liang, Jiczhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference computer vision, pages 1833-1844, 2021.
[30] Bee Lim, Sanghyun Son, Heewon Kim. Seungjun Nah. and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 134-144,77 2017.
[31] Xinqi Lin, Jingwen Hc, Ziyan Chen, Zhaoyang Lyu, Ben Fei, Bo Dai, Wanli Ouyang, Yu Qiao, and Chao Dong. Diffbir: Towards blind image restoration with generative diffusion prior. arXiv preprint 20w.2300..1070, 2023.
[32] Haotian Liu. Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[33] Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.01101. 2017.
[34] Simian Luo. Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao. Lom-lora: A universal stable-diffusion acceleration module. arXiv preprint arXiv:2311.05556 2023.
[35] Gaurav Parmar, Tacsung Park, Srinivasa Narasimhan, and Jun- Yan Zhu. One-step image translation with text-to-image models. arXiv preprint mrxivv24403122336 2024.
[36] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEFF/CVF conference on computer vision and puttern recognition, pages 10684-10695, 2022.
[37] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- to image diffusion models with deep language understanding. Advances in Neural Information Processing 2022.
Systems,
12

---

[38] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint urXiv:1409.1556, 2014.
[39] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 1rX4v:2011.13456, 2020.
[40] Lingchen Sun, Rongyuan Wu, Zhengqiang Zhang, Hongwei Yong, and Lei Zhang. Improving the stability of diffusion models for content consistent resoutnesol arXiv preprint arXiv:2401.0053 2023.
[41] Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 2555-2563, 2023.
[42] Jianyi Wang. Zongsheng Yue, Shangehen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution International Journal of Computer Vision, pages 1-21, 2024.
[43] Peihao Wang. Dejia Xu. Zhiwen Fan. Dilin Wang. Sreyas Mohan, Forrest landola, Rakesh Ranjan, Yilei Li, Qiang Liu, Zhangyang Wang, et al. Taming mode collapse in score distillation for text-to-3d generation. arXiv preprint arXiv:2401.0000
2023.
[44] Xintao Wang. Yu Li. Honglun Zhang, and Ying Shan. Towards worldworld blind face restoration with generative facial prior. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 9168-9178, 2021.
[45] Xintao Wang. Liagginn Xie, Chao Dong. and Ying Shan. Real Real-esrgan: Training real- cal-world blind super- resolution with pure synthetic data. In Proceedings of the IEEE/CVF international conference on computer vision. pages 1905-1914, 2021.
[46] Xintao Wang. Ke Yu. Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong. Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops, pages 0-0, 2018.
[47] Yinhuai Wang. Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint arXiv:2212.00490 2022.
[48] Yufei Wang, Wenhan Yang. Xinyuan Chen, Yaohui Wang. Lanqing Guo, Lap-Pui Chau, Ziwei Liu, Yu Qiao, Alex C Kot, and Bihan Wen. Sinsr: Diffusion-based image super-resolution a single step. arXiv preprint arXiv:2311.14731 2023.
[49] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su. and Jun Zhu. Prolificdreamer: High-fide fightfidelitt and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024.
[50] Zhou Wang. Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4))600-6/) 612, 2004.
[51] Pengxu Wei, Ziwei Xie, Hanman Lu, Zongyuan Zhan, Qixiang Ye, Wangmeng Zuo, and Liang Lin. Component divide-and-conque for real- world image resoutieso In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part VIII 16, pages 101-117. Springer, 2020.
[52] Rongyuan Wu, Tao Yang. Lingehen Sun. Zhengqiang Zhang, Shuai Li, and Lei Zhang. Seesr: Towards semantics-an aware real-world image super-resolution. arXiv preprint vxXiv2331..661/6 2023.
[53] Liangbin Xie, Xintao Wang, Xiangyu Chen, Gen Li, Ying Shan, Jiantao Zhou, and Chao Dong. Desra: Detect and delete the artifacts of gan-basce real- world super-resolution models. 2023.
[54] Jianchao Yang, John Wright, Thomas S Huang, and Yi Ma. Image super-resolution via sparse representation. IEEE transactions on image processing, 19(1 11::261-2287777 2010.
[55] Sidi Yang, Tianhe Wu, Shuwei Shi, Shanshan Lao. Yuan Gong, Mingdeng Cao, Jiahao Wang, and Yujiu Yang. Maniqa: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1191-1200, 2022.
attention network for no-reference image quality assessment. In
[56] Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang. Gan prior embedded network for blind face restoration in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 672-681, 2021.
13

---

[57] Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang. Pixel-aware stable diffusion for realistic image super-resolution and personalized stylization. arXiv preprini arXiv:2308. 14469, 2023.
[58] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6613-6623, 2024.
[59] Fanghua Yu, Jinjin Gu, Zheyuan Li, Jinfan Hu, Xiangtao Kong. Xintao Wang. Jingwen He, Yu Qiao, and Chao Dong. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild. arXiv preprint ar Xiv:2400.. 136277 2024.
[60] Zongsheng Yue, Jianyi Wang, and Chen Change Loy. Resshift: Efficient diffuusion model image super-resolution by residual shifting, arXiv preprint arXiv:2307.123444 2023.
[61] Kai Zhang, Jingyun Liang, Luc Van Gool, Radu Timofte. Designing a practical degradation model for deep blind image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2021.
[62] Lin Zhang, Lei Zhang, and Alan C Bovik, A feature-enriched completely blind image quality evaluator. IEEE Transactions on Image Processing, 24(8):2579-255 25991, 2015.
[63] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3836-3847, 2023.
[64] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 386-595, 2018.
[65] Xindong Zhang, Hui Zeng. Shi Guo, and Lei Zhang. Efficient long-range range attention network for image super-resolution. European Conference on Computer Vision, pages 649 667. Springer, 2022.
[66] Yulun Zhang. Kunpeng Li. Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super-resolution using very deep residual channel attention networks. Proceedings of European conference on computer vision (ECCV), pages 286-301, 2018.
[67] Yulun Zhang, Yapeng Tian, Kong, Bineng Zhong, and Yun Fu. Residual dense network image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2472-2481, 2018.
14

---

A Appendix
In the appendix, we provide the following materials:
Comparison with GAN-based methods (referring to Section 4.1 in the main paper).
Results of user study (referring to Section 4.1 in the main paper).
More real-world visual comparisons under scaling factor 4x (referring to Section 4.2 the main paper).
Training algorithm of OSEDiff (referring to Section 3.2 in the main paper).
A.1
Comparison with GAN-based Methods
We compare OSEDiff with four representative GAN-based Real-ISR methods, including BSRGAN [1]. Real-ESRGAN 45 LDL [27] and FeMaSR [4. The results are shown in Table 8 It is not surprise that GAN-based methods have better fidelity measures such as PSNR and SSIM than OSEDiff. However, OSEDiff has much better perceptual qualify metrics. We also provide visual comparisons in Figure Compared to GAN-based methods, OSEDiff able to generate realistic and reasonable details, such squirrel hair, textures of petals, buildings, and leaves.
A.2 User Study
To further validate the effectiveness of our proposed OSEDiff method, conducted user study by using 20 real-world LQ images. An LQ image and its HQ counterparts generated by different Real-ISR methods were presented to volunteers, who were asked select the best HQ result. The volunteers were instructed to consider two factors when making their decisions: the image perceptual quality and and its content (including structure and texture) consistency with the LQ input, with each factor contributing equally to the final selection.
We randomly selected 20 real-world LQ images from the ReaILR200 dataset 152 Figure 6 shows the thumbnails used in the user study, cropped into squares for a convenient layout. We generated the HQ outputs of them by using the DM-based Real-ISR methods StableSR [2]. DifiBIR [1,, SeeSR [52), PASD [57], ResShift [60]. SinSR [48], and OSEDiff. A number of 15 volunteers were invited to participate in the evaluation. The results shown in Figure 6 b). We see that OSEDiff ranks the
second, just lagging slightly behind SeeSR. However, should be noted that OSEDiff runs over 10 times faster than SeeSR by performing only one-step diffusion.
A.3 More Visual Comparisons
F
provides providess visual comparisons between OSEDiff and other DM-based methods. One see that OSEDiff achieves comparable to or even better results than the multi-step diffusion methods in scenarios such portraits, flower patterns, buildings, animal fur, and letters.
Table 8: Quantitative comparison with GAN-based methods on both synthetic and real-world benchmarks. The best results of each metric are highlighted red.
Datunets
Methods
PSNRT
SSIM1
LPIPS
DISTSL
FID_
NIQEL
MUSIQT
MANIQAT
CLIPIQAT
BSRGAN
24.58
0.6269
0.3351
0.2275
44.23
4.7518
61.20
0.5071
0.5247
Bcal-ESRGAN
37.64
4.6786
DIV2K-Val Val
LDL
23.83
0.6344
0.3256
0.2227
42.29
4.8554
60.04
0.5350
0,5180
ReMASR
23.06
0.5887
0.3126
0.2057
35,87
4.7410
60.33
0.5074
0.5997
0.2883
6,5192
Real ESRGAN
28.64
0.8053
0.2847
0.2089
147.62
6.6928
$4.18
0.4907
0.4422
DrealSR
LDL.
0.2815
0.2132
155.53
7.1298
0.4914
0.4310
FrMARR
26.90
0.7572
0.3169
0.2235
157.78
5.9073
53.74
0.4420
0.5464
OSEDiT
27.92
0.7835
0.2968
0.2165
135.30
6.4912
64.66
0,5899
0,6963
BSRGAN
26.39
0.7654
0.2670
0.2121
141.28
5:6567
63.21
0.5399
0.5001
Real-ESRGAN
D.7616
0.2727
0.2863
135.18
5.8295
60.18
0.5487
0.4449
ReaISR
LDL
25.28
0.7567
0.2766
0.2121
142,71
6.0024
60.82
0.5485
0.147
RMMARR
$8.95
OSEDiET
25.15
0.7341
0.2921
0.2128
123.49
5.6476
69.09
0.6326
0.6693
15

---

A.4 Algorithm of OSEDifT
The pseudo-code of our OSEDiff training algorithm is summarized Algorithm 1. We follow [49 [8] and use classifier free guidance (cfg) when calculating The cfg value is set to 7.5, and the negative prompts we use are: "painting, oil painting, illustration, drawing, art, sketch, oil painting, cartoon, CG Style, 3D render, unreal engine, blurring, dirty, messy, worst quality, low quality, frames, watermark, signature, jpeg artifucts, deformed, lowres, over-smooth."
Algorithm 1 Training Scheme of OSEDiff
Training dataset S, pretrained SD parameterized by including VAE encoder Fig, latent diffusion network Fig. and VAF decoder <s. prompt extractor Y, training iteration N Initialize Go parameterized by 0, including
Input:
Es Eg with trainable LoRA
Eş with trainable LoRA
Dg t D3
Initialize Es f- Eg with trainable LoRA for i - to N do
Sample æt, from S /* Network forward
Cy - f(rL)
ZL + Es(xL)
Z + Falzzicy)
tH < Da(EH)
/* Compute data objective
VoLant [ æu) ALCIpis(#u, 8āg
/+ Compute regularization objective, following DMD 58
Sample from (0, I)
11
Sample from {20.--- ,80)
12
2, f- 2, + otE
13
Za + stopgrad F& (2);
14
zy < stopgrad Far (2,;
1/mean[[] 2z
VoLreg < [w<zz =0]] diu
* Compute regularizer funetuning objective
17
Sample from (0,
Sample from [1.....
19
2: ar stop dic
29
Laif + CMSE(C& (Zx;t,
/+ Network Parameter Update Update 0 with Ldsta + X2
21
22
Update ø' with Lai
end
Output: Generator Go including VAE encoder Es, latent diffusion network Eg VAE decoder eo
16

---

Zoomte LIQ
BSRGAN
Resl ESRGIN
LQ
LDL
PrMMsS
OSEPAT
Zounzl LQ
BSRGAN
Rea-ESRGAN
PeMasS
Zoomed LA)
BSKGAN
REI-USKAAN
LDL
TeMaSS
OSEDaT
Figure 5: Qualitative comparisons between OSEDiff and GAN-based Real-ISR methods. Please zoom in for a better view.
Staoksk
22%
BALOSIS
Seeiit
UPHRR
25%
SeiRt
87
voting results are volunteers. The corresponding percentages and numerical counts are displayed with the pie chart.
(0) The IQ images used in user study
Figure 6: The LQ images used in user study and the voting results.
17

---

Zoomed LX
StollesR-2200
1iffbll--00
Scc5X-<30
ResShul--15
SiuSR-s1
OSEDiT-=1
Zoonrd LQ
Stabl=5B-2200
DunBiR-500
SwuSR-500
ResSLiD-115
SiuSR-11
OSEDiT-e1
Zoomed LQ
StablSRR =200
DiIBIIR
SeeSR =50
Re:Stin--15
SuSSitss]
OSEDET<1
Zoomed LQ
SubleSR s200
DiTDIR
SeeSR =50
ResStinn-115
SuaSRtss
OSEDiT<a
YDA EEE 0235 EBCA
YDA 13 131-E 0235 FBOA
-E
YDA 131-E 0235 EBCA
YDA131-2
0235 EBČA
YDA131-4 -E 0235 FBČA
Zoonzed LQ
StableSR-2200
DiIBER-50
SeeSR-550
A
YDA131 0235 EBCA
YDA 13-131EE 0235 EBOA
YDA 13113-4 131-E 0235 EBOA
YDA131 0235 EBCA
ResStaifts15
SinSR-s1
OSEDit-e1
Figure 7: More visulization comparisons of different DM-based Real-ISR methods. Please zoom in for a better view.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in the final section (Conclusion and Limitation).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification:
Yes, the paper provides full set of assumptions and proof for each theoretical
result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, the paper fully discloses all necessary information to reproduce the main experimental results, supporting the main claims and conclusions.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
20

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: training and testing data are publicly available. We will release the codes and model if the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cotee//ddeeeebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, the paper provides all necessary training and test details, including data splits, hyperparameters, and optimizer type, to fully understand the results.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[No]
Justification:
We do not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We detail the type of compute workers, memory, and execution time in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Yes, the research in the paper fully conforms to the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we analyze the potential social impact in the final section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, clearly indicate the baseline methods and training/testing data used in the paper. Their licenses permit use within academic scope.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Codes and model will be released if the paper is accepted.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowd sourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "Two9Kcp53C", "reviewer": "Reviewer_jXQz", "summary": "The paper introduces OSEDiff, a one-step diffusion network for Real-World Image Super-Resolution. OSEDiff performs variational score distillation in the latent space to ensure predicted scores align with those of multi-step pre-trained models. ​ By fine-tuning the pre-trained diffusion network with trainable LoRA layers, OSEDiff can adapt to real-world image degradations. Experimental results show that OSEDiff achieves comparable or superior Real-ISR outcomes to previous methods. ​ The paper's contributions include the development of OSEDiff, which addresses the limitations of traditional multi-step diffusion models and enables efficient production of high-quality images in one diffusion step. ​", "questions": "Please refer to the weakness part", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed method outperforms the previous single-step SR method (SinSR) in terms of the most of metrics.\n2.\tIt is comparable to previous multi-step diffusion models but greatly reduces inference time.\n3.\tThe model has very few training parameters and can be trained with minimal resources, which is advantageous.\n4.\tFigure 3 shows that the qualitative comparison results are very promising.", "weaknesses": "1.\tHow much would the performance of OSEDiff change if multiple denoising steps were performed, such as 20 steps or 50 steps?\n2.\tThe design of the key module, the regularizer networks, is not clearly explained.\n3.\tIt is unclear how the LR image is used as a condition. Is it concatenated along the channel dimension in the latent space? However, the pre-trained SD’s VAE encoder downscales by a factor of eight, so for a 512x512 HR input, the latent output is only 64x64, which conflicts with the 128*128 LQ setting mentioned in the first paragraph of section 4.1.\n4.\tThe reason for fine-tuning the VAE encoder is not very clear. Please explain in the revision why it is necessary to fine-tune the VAE encoder jointly with the diffusion model. If this has already been discussed, please indicate where it is covered.\n5.\tDoes the model support input of LR images with different resolutions? For example, if the input is a 32x32 LR image, can feasible results be obtained?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Multiple inference steps for OSEDiff.**\n\nPlease kindly note that OSEDiff is specifically designed for one-step diffusion for Real-ISR. Unlike  previous multi-step methods (e.g., StableSR, PASD, SeeSR), which are all based on ControlNet by using noise as input and LQ image as control signal, OSEDiff directly uses LQ image as input (without random noise) and apply LoRA to finetune the diffusion network for producing HQ output. In its current network design, OSEDiff cannot be used for multiple denoising steps. Nonetheless, we believe that for the Real-ISR task, OSEDiff is much preferred as it achieves faithful and perceptually realistic results while significantly reducing the complexity. If we wish to extend OSEDiff to a multi-step framework for stronger generative capability, we could model multi-step diffusion in the residual space, similar to ResShift. We will explore this possibility in the future.  \n\n**Q2. Details about regularizer networks.**\n\nThanks for pointing out this problem. We will add more details on the regularizer networks in the revision. To model the distribution of natural and restored images, two diffusion models are needed to compute the KL-divergence via variational score distillation (VSD): one for the natural image distribution and another for the restored image distribution. Since the pre-trained SD model can effectively represent the natural image distribution, one regularizer network can be the frozen SD U-net. On the other hand, the U-net can be fine-tuned to model the distribution of restored images. Therefore, we fine-tune the SD U-net as another regularizer network in a similar way to that we fine-tune the SD as the generator network. However, they are trained differently: the generator is trained on $t=999$ to perform super-resolution, while the regularizer is trained on $t \\in (\\{0, \\dots,999\\})$ to align with the restored image distribution via VSD loss.\n\n**Q3. LQ as condition.**\n\nSorry for the confusion caused. Actually, different from previous methods such as StableSR, PASD and SeeSR, our proposed OSEDiff does not use the LQ image as a condition but directly as the input to the UNet. First, we upsample the LQ image to the target size. Then, we input it into the encoder to obtain the latent feature, which is passed into the UNet to obtain the refined feature. Finally, the refined feature goes through the decoder to output the restored HQ image. For example, for $\\times$4 SR on a 128$\\times$128 LQ image, we first use a bicubic interpolator to upsample it to 512$\\times$512. Then, we pass it to OSEDiff to get a 512$\\times$512 HQ image. We will clarify this in the revised manuscript.\n\n**Q4. Finetune VAE encoder.**\n\nWe apologize for any lack of clarity in our manuscript. The VAE encoder is set to be trainable to remove degradation. In other words, we fine-tune the encoder with LoRA so that it can also serve as a degradation removal network to pre-process the input LQ image. \n\nTo better illustrate the role of fine-tuning the encoder, we compare the performance of OSEDiff on the RealSR dataset with and without fine-tuning the VAE encoder. The results are shown in the table below. We can see that fine-tuning the encoder significantly improves the no-reference metrics. Although fine-tuning the encoder leads to slightly worse full-reference metrics such as PSNR and LPIPS, these metrics do not necessarily indicate better visual quality. In Figure 1 of the enclosed PDF file, we visualize the results of OSEDiff with and without encoder fine-tuning. One can see that fixing the VAE encoder may introduce some artifacts, which can be caused by the severe degradation in the LQ input. We will add the above discussions in the revision. \n\n**Table: Comparison of OSEDiff with and without fine-tuning the VAE encoder on the RealSR dataset.**\n|               | Finetune VAE Encoder | PSNR↑ | SSIM↑  | LPIPS↓ | CLIPIQA↑ | MUSIQ↑ | NIQE↓  |\n|---------------|:-----------------:|:-----:|:------:|:------:|:--------:|:------:|:------:|\n| OSEDiff       |        ✗          | 25.27 | 0.1966 | 0.2656 | 0.5303   | 58.99  | 6.5496 |\n| OSEDiff       |        ✓          | 25.15 | 0.2128 | 0.2921 | 0.6693   | 69.09  | 5.6479 |\n\n**Q5. LR inputs of different resolutions.**\n\nYes, OSEDiff can handle inputs of different resolutions in a similar way to other diffusion based super-resolution methods. First, the LR image (with the size of H$\\times$W) is upsampled to an HR image $Y_{bicubic}$ (with the size of rH$\\times$rW) using bicubic interpolation, where $r$ is the SR factor. If the short side of $Y_{bicubic}$ is less than 512, we resize its short side to 512 before feeding it to OSEDiff. Finally, the output is resized back to rH$\\times$rW to obtain $Y_{osediff}$. In other cases, $Y_{bicubic}$ (rH$\\times$rW) is directly fed into OSEDiff to get $Y_{osediff}$ (rH$\\times$rW). For the $\\times$4 SR task, if LR is 32$\\times$32, we first upsample LR to 512$\\times$512 using bicubic interpolation, enhance it to  512$\\times$512 by OSEDiff, and then resize the output to 256$\\times$256.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PaQYUz9OiX", "reviewer": "Reviewer_KDhM", "summary": "This paper presents a novel approach to real-world image super-resolution for a one-step effective diffusion network (OSEDiff). The proposed OSEDiff effectively eliminates the uncertainty introduced by random noise sampling in previous methods, achieving significant performance improvements across multiple benchmarks, demonstrating its potential for practical applications.", "questions": "No more questions.", "limitations": "The authors have discussed about the limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The focus on one-step diffusion in this paper is particularly noteworthy, as it addresses a critical challenge for the practical application of diffusion models in image super-resolution tasks.\n2. The performance gains and visually compelling results suggest the effectiveness of the proposed method for real-world image super-resolution.\n3. The paper is well-structured and easy to follow.", "weaknesses": "1. The core concept of this paper shares similarities with \"One-step Diffusion with Distribution Matching Distillation\". A comprehensive comparison with this work, including performance metrics, training efficiency (parameters and FLOPs), and potential limitations, would strengthen the paper's contribution.\n2. While Table 2 provides a comparison of trainable parameters, including the total number of model parameters and FLOPs during inference would offer a more complete picture of computational efficiency.\n3. There is a lack of discussion on the degradation in LQ images, such as how to ensure that text prompts remain accurate in low-quality scenarios.\n4. In Figure 7 (first and second scenarios), some generated details and textures appear to be creatively added rather than strictly reconstructed from the LQ input. Addressing potential limitations in fidelity due to this creative aspect would be valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Comparison with DMD.**\n\nThanks for the nice suggestion. While both OSEDiff and DMD draw upon the concept of variational distillation from ProlificDreamer, they differ significantly in several aspects. First, DMD is designed for text-to-image tasks, whereas OSEDiff is tailored for image restoration tasks, aiming to reconstruct an HQ image from its LQ counterpart. To achieve this goal, OSEDiff employs a trainable VAE encoder to remove degradation from LQ images, and it directly takes the LQ as input to the diffusion network to avoid the uncertainty caused by random noise sampling. Trainable LoRA layers are introduced to the diffusion UNet to adapt it to the restoration task. In contrast, DMD lacks these specific designs tailored for image restoration tasks. Furthermore, DMD requires full parameter fine-tuning of two SD models, while OSEDiff only needs to fine-tune a small number of LoRA parameters. This makes OSEDiff more memory-efficient and training-friendly. We will cite DMD and discuss its similarity and differences from OSEDiff in the revision.\n\n**Q2. Comparison of the total number of model parameters and FLOPs.**\n\nThanks for the suggestion. The table below compares the total number of model parameters and FLOPs of different methods.\n\nWe see that those pre-trained SD model based methods (i.e., StableSR, DiffBIR, SeeSR, PASD and OSEDiff) have a similar total number of parameters. SinSR employs the diffusion model trained in ResShift, which is much smaller than the pre-trained SD model. However, the generalization capability of ResShift and SinSR is much lower than that of the SD based methods.\n\nIn terms of FLOPs, the multi-step diffusion-based models have significantly higher FLOPs than single-step methods (e.g., SinSR and OSEDiff) because they run the SD UNet for multiple times. \nThe input resolution of SinSR's VAE decoder is twice of that of OSEDiff's VAE decoder, as SinSR uses an f4 VAE and OSEDiff uses an f8 VAE. Overall, OSEDiff and SinSR have similar FLOPs. We will add these analysis in the revision. \n\n**Table: Complexity comparison among different methods. All methods are tested with an input image of size 512×512 on an A100 GPU.**\n\n|                       | StableSR | DiffBIR | SeeSR | PASD   | ResShift | SinSR | OSEDiff |\n|-----------------------|:--------:|:-------:|:-----:|:------:|:--------:|:-----:|:-------:|\n| **# Total Param (M)** | 1410     | 1717    | 2524  | 1900   | 119      | 119   | 1775    |\n| **FLOPs (G)**         | 79940    | 24234   | 65857 | 29125  | 5491     | 2649  | 2265    |\n| **Inference Step**    | 200      | 50      | 50    | 20     | 15       | 1     | 1       |\n| **Inference Time (s)**| 11.50    | 2.72    | 4.30  | 2.80   | 0.71     | 0.16  | 0.35    |\n\n**Q3. Robust text prompts from LQ.**\n\nSorry that we did not make it clear. We adopt the DAPE module from SeeSR, which has proven to be robust to the degradation of LQ images for extracting tag-style text prompts. The DAPE module is trained to generate correct semantic prompts even when the input images are severely degraded.\n\n**Q4. Limitations in fidelity.**\n\nCompared with traditional image restoration methods, the recent methods that leverage pre-trained SD priors can produce perceptually much more realistic results. As a compromise, some generated details may not be faithful enough to the input LQ image. This is an inherent limitation of such generative prior based methods, as noted by this reviewer. \n\nCompared with other methods along this line, however, the proposed OSEDiff has achieved a much better balance between fidelity and creativity. First, it takes the LQ image as input without any random noise and performs only one-step diffusion. This significantly improves the stability of image synthesis using diffusion priors, reducing much the possibility of generating false details. On the other hand, OSEDiff utilizes VSD loss to regularize the generator network, ensuring that it can preserve the generative capacity of the pre-trained SD model. As can be seen from the experiments in the main paper, OSEDiff achieves state-of-the-art results in both fidelity and perceptual metrics among the SD based methods. In future research, for image scenes or regions that require high-fidelity, such as face and text, we can apply a higher weight of fidelity loss. For scenes like greenery and hair where the perceptual quality is more important, we can adaptively apply a higher weight of VSD loss to enhance expressiveness.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bfcswgcHPt", "reviewer": "Reviewer_px3F", "summary": "This paper introduces a diffusion-based real-world image super-resolution method, OSEDiff, which can efficiently generate high-quality images in just one diffusion step. Firstly, in order to eliminate the uncertainty introduced by random noise sampling, the authors propose to directly feed the low-quality images without any random noise into the pre-trained SD model which integrates trainable LoRA layers. Furthermore, the variational score distillation for KL-divergence regularization is utilized to align the distribution of generated images with natural image prior, ensuring that the one-step diffusion model could still generate high-quality output. OSEDiff achieves comparable performance to state-of-the-art SD-based real-world image super-resolution methods with fewer trainable parameters and inference time.", "questions": "(1) Could you please provide some visualization of the output of diffusion network which are sent to the two regularizer networks to further prove the effectiveness of the regularization loss in the latent space?\n\n(2) Could you please explain in detail why \"directly feeding the LQ images into the pre-trained SD model\" can eliminate the uncertainty of output and achieve better Real-ISR performance than the methods which uses inputs with random noise?\n\n(3) Could you please provide detailed network structure of VAE encoder and the diffusion network with trainable LoRA layers?\n\n(4) Could you please provide more qualitative comparisons of the method with/without text prompt extractor to further prove the effectiveness of text prompt or further elaborate on the importance of the text prompt to prove that it is essential?", "limitations": "(1) Briefly explain why \"directly feeding the LQ images into the pre-trained SD model\" can eliminate the uncertainty of output, and why it can achieve better Real-ISR performance than the methods which uses inputs with random noise.\n\n(2) Please provide some visualization of the output of diffusion network which are sent to the two regularizer networks.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper introduces a one-step effective diffusion network for real-world image super-resolution. The low-quality images without any random noise are utilized as the inputs of the pre-trained SD model and the VSD in latent space are utilized to align the distribution of generated images with natural image prior. The proposed method solves two problems in diffusion-based real-world image super-resolution and it achieves comparable performance to state-of-the-art SD-based real-world image super-resolution methods with fewer trainable parameters and inference time. The figures of this paper are simple and clear, with precise formula expressions, demonstrating a certain clarity.", "weaknesses": "(1) The motivation is unclear. In the introduction, the authors use a certain amount of space to mention two major issues in training a Real-ISR model, including the problem of how to build the LQ-HQ training image pairs, but this paper does not propose an innovative solution to this problem.\n\n(2) The explanation of the novelty and its effectiveness are insufficient. 1) Contribution 1, \"directly feeding the LQ images into the pre-trained SD model without introducing any random noise,\" claims to eliminate the uncertainty of output but lacks supporting evidence, raising doubts about its validity. It is recommended to provide visual evidence and analyze why this can eliminate the uncertainty and achieve better Real-ISR performance in detail. 2) Contribution 2, “utilizing variational score distillation for KL-divergence regularization,” it is recommended to provide some visualization of the output of diffusion network which are sent to the two regularizer networks to prove that it can align the distribution of generated images with natural image prior. \n\n(3) The metrics used need more explanation. Since CLIPIQA is text-related, although it has been proved to be effective and generalized in image quality assessment, it is not that convincing to directly apply it to evaluate the quality of real-world super-resolution images. More explanation can be provided.\n\n(4) Details of the network structure are lacking in the figure. In Figure 2, the detailed network structure of VAE encoder and the diffusion network with trainable LoRA layers are lacking. It is recommended to provide detailed network structure.\n\n(5) In “Ablation Study”, the super-resolution performance of the method with text prompt extractor is not competitive enough compared to removing text prompt. Although the qualitative comparisons show that the method with text prompt extractor can generate richer image details, the method without text prompt is much better for several common metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Unclear motivation.**\n\nThanks for the comments. The goal of this work is to develop an efficient and effective Real-ISR method by using the pre-trained SD prior. In the research and development of Real-ISR methods, how to construct LQ-HQ training pairs is a critical issue. Therefore, we spend a certain amount of space to introduce this problem so that readers can have a more complete understanding of Real-ISR. Based on this reviewer's comments, we will compress this part and make the introduction of LQ-HQ pair more concise.  \n\n**Q2. The benefits of reduced uncertainty.**\n\nThanks for your suggestion. Previous diffusion-based Real-ISR methods apply several denoising steps to generate HQ latents from noisy inputs. However, this process introduces unwanted randomness in the Real-ISR outputs, causing variations with different noise samples. OSEDiff uses LQ image as input to the diffusion network without random noise sampling, enabling a deterministic mapping from LQ to the HQ image. Meanwhile, OSEDiff leverages the VSD loss to enhance the generation capability of the LoRA finetuned network. \n\nAs suggested by this reviewer, in Figure 2 of the enclosed PDF file we provide visual comparison of the Real-ISR results of OSEDiff, StableSR and SeeSR. For StableSR and SeeSR, we show their results with two different noise samples. One can see clearly the difference between the two outputs caused by the randomness of noise. For example, the details in StableSR-1 are overly generated, while the result of StableSR-2 is smooth. In contrast, OSEDiff does not involve noise in the input, and it achieves stable Real-ISR result. We will add the visual evidence and analysis in the revision.\n\n**Q3. Visualization of network outputs.**\n\nThanks for the good suggestion. As suggested by this reviewer, we visualize the distributions of outputs of OSEDiff with and without VSD loss using t-SNE on the RealSR dataset. The distribution of pre-trained SD model is plotted as a reference. We use LLaVa1.5 to generate the captions for the RealSR dataset, and use them as prompts for pre-trained SD, with the inference steps set to 50. As shown in Figure 3 of the enclosed PDF file, the output distribution of OSEDiff with VSD loss is much closer to the pre-trained SD than the OSEDiff without VSD loss. This clearly validates that the VSD loss aligns the  distribution of diffusion network outputs with that of pre-trained SD. We will add this part in the revision.\n\n**Q4. Why use CLIPIQA for Real-ISR evaluation.**\n\nWe follow previous works (StableSR, DiffBIR, and SeeSR) to use CLIPIQA as an evaluation metric for fair comparison. The visual examples in Figures 1, 4, and 5 the enclosed PDF file demonstrate that higher CLIPIQA scores generally correspond to better visual quality of the images. However, we agree with this reviewer that highly reliable no-reference IQA remains an unsolved problem, especially for real-world images, and it needs more efforts from the community to find a more robust metric.\n\n**Q5. Details about trainable LoRA module.**\n\nThanks for the suggestion. We finetuned all layers except the normalization layers using LoRA. We empirically found that fine-tuning the normalization layers causes the model to crash, so we frozen them. We will add more detailed descriptions in the revision.\n\n**Q6. Null prompts vs. Tag prompts.**\n\nThough the results with null prompts score higher on some reference metrics (e.g., PSNR and DISTS) than those with tag-style prompts extracted by DAPE, their visual quality is inferior. This is mainly because the commonly used metrics such as PSNR cannot faithfully reflect the visual quality of images. As shown in Figure 4 of the enclosed PDF file, the results using tag-style prompts have richer leaf vein textures, clearer lines, and less noise. Therefore, we choose tag prompts extracted by DAPE. We will further clarify this in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "urhssYkSu2", "reviewer": "Reviewer_4vqL", "summary": "In this paper, authors propose a novel one-step effective diffusion network, termed as OSEDiff, for the Real-ISR. The proposed OSEDiff adopts LQ image as the input and directly output the final output with the help of the decoder of VAE, thus eliminate the uncertainty introduced by random noise sampling and achieve one-step generation of diffusion model. Besides, the variational score distillation is introduced in the latent space to conduct KL-divergence regularization to enhance the quality of final Real-ISR output. In the experimental part, the proposed method outperforms other baselines in both quantitative and qualitative comparisons.", "questions": "There are several concerns and suggestions here:\n1. I am wondering the comparisons between the proposed method and SinSR. Does the improvement of performance gain from the pre-trained SD? \n2. Experiments of the paper is expected to be improved. For instance, total parameters instead of the training parameters are expected to be compared in the Table 2, which is more closely related to the deployment.\n3. I am wondering the reason why the setting of \"w/o VSD loss\" may achieve better PSNR than proposed method. I would be better to provide more visual results of it to demonstrate the effectiveness of the VSD loss.", "limitations": "Could be referred to above.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "There are several strengths here:\n1. One-step diffusion model is attractive and intereting, which will significantly reduce the inference time.\n2. The paper is easy to read and understand.\n3. The proposed method show comparable results in the experimental part.", "weaknesses": "There are several weakness here:\n1. Experimental part could be further improved, which will be further demonstrated in the following section.\n2. The efficient and params of the proposed method is expected to be detailed demonstrated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Comparison with SinSR.**\n\nThe improvements of OSEDiff over SinSR mainly come from the pre-trained SD model and the VSD loss. The SD model, pre-trained on large-scale data, contains rich prior knowledge of natural images, enabling effective one-step generation. Additionally, we finetune the multi-step SD model into a single-step network using LoRA and VSD loss. It ensures the one step generation capacity of OSEDiff for restoration tasks while avoiding the lengthy inference time of multiple steps.\n\n**Q2. Total params rather than training params.**\n\nThanks for the suggestion. The table below show the total number of parameters, FLOPs and the inference time of the competing methods. Please refer to our responses to Q2 of Reviewer KDhM for more discussions. We will add this table and the associated discussions in the revision. \n\n**Table: Complexity comparison among different methods. All methods are tested with an input image of size 512×512 on an A100 GPU.**\n\n|                       | StableSR | DiffBIR | SeeSR | PASD   | ResShift | SinSR | OSEDiff |\n|-----------------------|:--------:|:-------:|:-----:|:------:|:--------:|:-----:|:-------:|\n| **# Total Param (M)** | 1410     | 1717    | 2524  | 1900   | 119      | 119   | 1775    |\n| **FLOPs (G)**         | 79940    | 24234   | 65857 | 29125  | 5491     | 2649  | 2265    |\n| **Inference Step**    | 200      | 50      | 50    | 20     | 15       | 1     | 1       |\n| **Inference Time (s)**| 11.50    | 2.72    | 4.30  | 2.80   | 0.71     | 0.16  | 0.35    |\n\n**Q3. Why is the PSNR lower when using VSD Loss compared to not using it, and more visualization results.**\n\nIn the objective function of a general learning-based image restoration framework (refer to Eq. (1) in the main paper), there are two major components: a fidelity term and a regularization term. The fidelity term is usually measured and constrained by $L_2$ or $L_1$ norms, which is friendly to the PSNR metric, while the regularization term is designed based on the employed prior knowledge of natural images. In our work, the VSD loss, which aligns the distribution of network outputs to that of SD prior distribution (i.e., natural image prior distribution), is used as the regularization term. With VSD, the fidelity will be traded-off a little but the perceptual quality will be much enhanced (due to the alignment with natural image prior distribution). Without VSD, the network will focus on optimizing the fidelity term so that the PSNR metric can be improved; however, the perceptual quality will be decreased. Visual examples can be found in Figure 5 of the enclosed PDF file. Without the VSD loss, the PSNR of the restoration results is higher, but they lack semantic details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers, Area Chairs, and Program Chairs:

We are grateful for the constructive comments and valuable feedback from the reviewers. We appreciate the reviewers' recognition on the novelty of our method (Reviewers KDhM and 4vqL), its superior performance (Reviewers jXQz and px3F), efficient training (Reviewer jXQz), and its clarity (Reviewers KDhM, px3F, and 4vqL).

To address the reviewers' concerns, we have provided more analysis about inference setting, VAE encoder finetuning, the distribution of output, comparison with DMD and SinSR, and the total parameters and FLOPs. We have also attached a PDF file for more visual comparisons. Please find our itemized responses to all reviewer’s comments below. We will really appreciate it if Reviewer px3F can kindly reconsider the decision, provided that the main concerns are well addressed.

Best regards,

Authors of Paper 4246

Meta-review: This manuscript proposes a single-step image super-resolution model based on diffusion models, utilizing DMD loss for distillation, which results in faster generation compared to multi-step methods. All reviewers have provided positive or neutral scores, so the final decision for this manuscript is acceptance.
 
However, the manuscript has issues with insufficient citation of prior work. The proposed method, which employs LoRA finetuning, heavily relies on 'One-Step Image Translation with Text-to-Image Models' [1], which demonstrates the use of Stable Diffusion as a base model with LoRA finetuning, yet this work is not cited in the manuscript. The proposed loss function is essentially the DMD loss, but DMD [2] is not cited either. Additionally, contrary to what was mentioned in the authors' rebuttal, the use of LoRA for efficient finetuning is already provided in the official DMD code. Moreover, the manuscript bears significant similarity to SwiftBrush [3], but this work is not cited. It is strongly recommended that the authors cite the aforementioned prior works and include a discussion of them in the revised version of the manuscript.

Furthermore, there are concerns regarding the fairness of the experimental setup. While the proposed method and SeeSR[4] were trained on the high-quality LSDIR [5] dataset, other comparison methods used different datasets. This discrepancy in training data, particularly given the superior quality of LSDIR, leads to an unfair advantage and potentially unreliable comparison results. It is strongly recommended that the authors additionally provide experimental results using the same training datasets as the other methods. 

[1] Parmar, Gaurav, et al. "One-step image translation with text-to-image models." arXiv preprint arXiv:2403.12036.

[2] Yin, Tianwei, et al. "One-step diffusion with distribution matching distillation." CVPR. 2024.

[3] Nguyen, Thuan Hoang, and Anh Tran. "Swiftbrush: One-step text-to-image diffusion model with variational score distillation." CVPR 2024.

[4] Wu, Rongyuan, et al. "Seesr: Towards semantics-aware real-world image super-resolution."CVPR 2024.

[5] Li, Yawei, et al. "Lsdir: A large scale dataset for image restoration." CVPR 2023.
Decision: Accept (poster)