One-Step Effective Diffusion Network for Real-World  Image Super-Resolution

Authors: rongyuan  wu
lingchen  sun
zhiyuan  ma
lei  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 04:06:53

Abstract: The pre-trained text-to-image diffusion models have been increasingly employed to tackle the real-world image super-resolution (Real-ISR) problem due to their powerful generative image priors. Most of the existing methods start from random noise to reconstruct the high-quality (HQ) image under the guidance of the given low-quality (LQ) image. While promising results have been achieved, such Real-ISR methods require multiple diffusion steps to reproduce the HQ image, increasing the computational cost. Meanwhile, the random noise introduces uncertainty in the output, which is unfriendly to image restoration tasks. To address these issues, we propose a one-step effective diffusion network, namely OSEDiff, for the Real-ISR problem. 
We argue that the LQ image contains rich information to restore its HQ counterpart, and hence the given LQ image can be directly taken as the starting point for diffusion, eliminating the uncertainty introduced by random noise sampling. We finetune the pre-trained diffusion network with trainable layers to adapt it to complex image degradations. To ensure that the one-step diffusion model could yield HQ Real-ISR output, we apply variational score distillation in the latent space to conduct KL-divergence regularization. As a result, our OSEDiff model can efficiently and effectively generate HQ images in just one diffusion step. 
Our experiments demonstrate that OSEDiff achieves comparable or even better Real-ISR results, in terms of both objective metrics and subjective evaluations, than previous diffusion model-based Real-ISR methods that require dozens or hundreds of steps. The source codes are released at https://github.com/cswry/OSEDiff.

Main paper: 
 introduction: 
 Image super-resolution (ISR) [13, 66, 29, 65, 6, 24, 46, 27, 61] is a classical yet still active research problem, which aims to restore a high-quality (HQ) image from its low-quality (LQ) observation suffering from degradations of noise, blur and low-resolution, etc. While one line of ISR research [13, 66, 29, 65, 6] simplifies the degradation process from HQ to LQ images as bicubic downsampling (or downsampling after Gaussian blur) and focus on the study on network architecture design, the trained models can hardly be generalized to real-world LQ images, whose degradations are often unknown and much more complex. Therefore, another increasingly popular line of ISR research is the so-called real-world ISR (Real-ISR) [61, 45] problem, which aims to reproduce perceptually realistic HQ images from the LQ images captured in real-world applications.There are two major issues in training a Real-ISR model. One is how to build the LQ-HQ training image pairs, and another is how to ensure the naturalness of restored images, i.e., how to ensure that the restored images follow the distribution of HQ natural images. For the first issue, some researchers [51] . Metrics like LPIPS and NIQE, where smaller scores indicate better image quality, are inverted and normalized for display. OSEDiff achieves leading scores on most metrics with only one diffusion step. (b). Model efficiency comparison. The inference time is tested on an A100 GPU with 512 × 512 input image size. OSEDiff has the fewest trainable parameters and is over 100 times faster than StableSR [42] .have proposed to collect real-world LQ-HQ image pairs using long-short camera focal lenses [3, 51] . However, this is very costly and can only cover certain types of real-world image degradations. Another more economical way is to simulate the real-world LQ-HQ image pairs by using complex image degradation pipelines. The representative works include BSRGAN [61] and Real-ESRGAN [45] , where a random shuffling of basic degradation operators and a high-order degradation model are respectively used to generate LQ-HQ image pairs. With the given training data, how to train a robust Real-ISR model to output perceptually natural images with high quality becomes a key issue. Simply learning a mapping network between LQ-HQ paired data with pixel-wise losses can lead to over-smoothed results [24, 46] . It is crucial to integrate natural image priors into the learning process to reproduce HQ images. A few methods have been proposed to this end. The perceptual loss [18] explores the texture, color, and structural priors in a pre-trained model such as VGG-16 [38] and AlexNet [23] . The generative adversarial networks (GANs) [14] alternatively train a generator and a discriminator, and they have been adopted for Real-ISR tasks [24, 46, 45, 61, 27, 53] . The generator network aims to synthesize HQ images, while the discriminator network aims to distinguish whether the synthesized image is realistic or not. While great successes have been achieved, especially in the restoration of specific classes of images such as face images [56, 44] , GAN-based Real-ISR tends to generate unpleasant details due to the unstable adversarial training and the difficulties in discriminating the image space of diverse natural scenes.The recently developed generative diffusion models (DM) [39, 16] , especially the large-scale pretrained text-to-image (T2I) models [37, 36] , have demonstrated remarkable performance in various downstream tasks. Having been trained on billions of image-text pairs, the pre-trained T2I models possess powerful natural image priors, which can be well exploited to improve the naturalness and perceptual quality of Real-ISR outputs. Some methods [42, 57, 31, 52, 40, 59] have been developed to employ the pre-trained T2I model for solving the Real-ISR problem. While having shown impressive results in generating richer and more realistic image details than GAN-based methods, the existing SD-based methods have several problems to be further addressed. First, these methods typically take random Gaussian noise as the start point of the diffusion process. Though the LQ images are used as the control signal with a ControlNet module [63] , these methods introduce unwanted randomness in the output HQ images [40] . Second, the restored HQ images are usually obtained by tens or even hundreds of diffusion steps, making the Real-ISR process computationally expensive. Though some one-step diffusion based Real-ISR methods [48] have been recently proposed, they fail in achieving high-quality details compared to multi-step methods.To address the aforementioned issues, we propose a One-Step Effective Diffusion network, OSEDiff in short, for the Real-ISR problem. The UNet backbone in pre-trained SD models has strong capability to transfer the input data into another domain, while the given LQ image actually has rich information to restore its HQ counterpart. Therefore, we propose to directly feed the LQ images into the pre-trained SD model without introducing any random noise. Meanwhile, we integrate trainable LoRA layers [17] into the pre-trained UNet, and finetune the SD model to adapt it to the Real-ISR task. On the other hand, to ensure that the one-step model can still produce HQ natural images as the multi-step models, we utilize variational score distillation (VSD) [49, 58, 10] for KL-divergence regularization. This operation effectively leverages the powerful natural image priors of pre-trained SD models and aligns the distribution of generated images with natural image priors. As illustrated in Fig. 1 , our extensive experiments demonstrate that OSEDiff achieves comparable or superior performance measures to state-of-the-art SD-based Real-ISR models, while it significantly reduces the number of inference steps from N to 1 and has the fewest trainable parameters, leading to more than ×100 speedup in inference time over previous methods such as StableSR [42] . 
related work: 
 Starting from SRCNN [13] , deep learning-based methods have become prevalent for ISR. A variety of methods have been proposed [30, 66, 67, 9, 5, 29, 65, 6, 7] to improve the accuracy of ISR reconstruction. However, most of these methods assume simple and known degradations such as bicubic downsampling, limiting their applications to real-world images with complex and unknown degradations. In recent years, researches have been exploring the potentials of generative models, including GAN [14] and diffusion networks [16] , for solving the Real-ISR problem.GAN-based Real-ISR. The use of GAN for photo-realistic ISR can be traced back to SRGAN [24] , where the image degradation is assumed to be bicubic downsampling. Later on, researchers found that GAN has the potential to perform real-world image restoration with more complex degradations [61, 45] . Specifically, by using randomly shuffled degradation and high-order degradation to generate more realistic LQ-HQ training pairs, BSRGAN [61] and Real-ESRGAN [45] demonstrate promising Real-ISR results, which trigger many following works [4, 27, 28, 53] . DASR [28] designs a tiny network to predict the degradation parameters to handle degradations of various levels. SwinIR [29] switches the generator from CNNs to stronger transformers, further enhancing the performance of Real-ISR. However, the adversarial training process in GAN is unstable and its discriminator is limited in telling the quality of diverse natural image contents. Therefore, GAN-based Real-ISR methods often suffer from unnatural visual artifacts. Some works such as LDL [27] and DeSRA [53] can suppress much the artifacts, yet they are difficult to generate more natural details.Diffusion-based Real-ISR. Some early attempts [21, 20, 47] employ the denoising diffusion probabilistic models (DDPMs) [16, 39, 11] to address the ISR problem by assuming simple and known degradations (e.g., bicubic downsampling). These methods are training-free by modifying the reverse transition of pre-trained DDPMs using gradient descent, but they cannot be applied to complex unknown degradations. Recent researches [42, 57, 31, 40, 59 ] have leveraged stronger pre-trained T2I models, such as Stable Diffusion (SD) [1] , to tackle the Real-ISR problem. In general, they introduce an adapter [63] to fine-tune the SD model to reconstruct the HQ image with the LQ image as the control signal. StableSR [42] finetunes a time-aware encoder and employs feature warping to balance fidelity and perceptual quality. PASD [57] extracts both low-level and high-level features from the LQ image and inputs them to the pre-trained SD model with a pixel-aware cross attention module. To further enhance the semantic-aware ability of the Real-ISR model, SeeSR [52] introduces degradation-robust tag-style text prompts and utilizes soft prompts to guide the diffusion process. To mitigate the potential risks of diffusion uncertainty, CCSR [40] leverages a truncated diffusion process to recover structures and finetunes the VAE decoder by adversarial training to enhance details. SUPIR [59] leverages the powerful generation capability of SDXL model and the strong captioning capability of LLaVA [32] to synthesize rich image details.The above mentioned methods, however, require tens or even hundreds of steps to complete the diffusion process, resulting in unfriendly latency. SinSR shortens ResShift [60] to a single-step inference by consistency preserving distillation. Nevertheless, the non-distrbution-based distillation loss tends to obtain smooth results, and the model capacity of SinSR and ResShift are much smaller than the SD models to address Real-ISR problems. 
problem modeling: 
 Real-ISR is to estimate an HQ image xH from the given LQ image x L . This task can be conventionally modeled as an optimization problem: xH = argmin x H (L data (Φ(x H ), x L ) + λL reg (x H )), where Φ is the degradation function, L data is the data term to measure the fidelity of the optimization In addition, text prompts are extracted from the LQ image and input to the diffusion network to stimulate its generation capacity. Meanwhile, the output of the diffusion network ϵ θ will be sent to two regularizer networks (a frozen pre-trained one and a fine-tuned one), where variational score distillation is performed in latent space to ensure that the output of ϵ θ follows HQ natural image distribution. The regularization loss will be back-propagated to update E θ and ϵ θ . Once training is finished, only E θ , ϵ θ and D θ will be used in inference.output, L reg is the regularization term to exploit the prior information of natural images, and scalar λ is the balance parameter. Many conventional ISR methods [13, 29, 65] restore the desired HQ image by assuming simple and known degradation models and employing hand-crafted natural image prior models (i.e., image sparsity based prior [54] ).However, the performance of such optimization-based methods is largely hindered by two factors. First, the degradation function Φ is often unknown and hard to model in real-world scenarios. Second, the hand-crafted regularization terms L reg are hard to effectively model the complex natural image priors. With the development of deep-learning techniques, it has become prevalent to learn a neural network G θ , which is parameterized by θ, from a training dataset S of (x L , x H ) pairs to map the LQ image to an HQ image. The network training can be described as the following learning problem:EQUATIONwhere L data and L reg are the loss functions. L data enforces that the network output xH = G θ (x L ) can approach to the ground-truth x H as much as possible, which can be quantified by metrics such as L 1 norm, L 2 norm and LPIPS [64] . Using only the L data loss to train the network G θ from scratch may over-fit the training dataset. In this work, we propose to finetune a pre-trained generative network, more specifically the SD [36] network, to improve the generalization capability of G θ . In addition, the regularization loss L reg is critical to improve the generalization capability of G θ , as well as enhance the naturalness of output HQ images xH . Suppose that we have the distribution of real-world HQ images, denoted by p(x H ), the KL divergence [8] is an ideal choice to serve as the loss function of L reg ; that is, the distribution of restored HQ images, denoted by q θ (x H ), should be identical to p(x H ) as much as possible. The regularization loss can be defined as:EQUATIONExisting works [24, 46] mostly instantiate the above objective via adversarial training [14] , which involves learning a discriminator to differentiate between the generated HQ image xH and the real HQ image x H , and updating the generator G θ to make xH and x H indistinguishable. However, the discriminators are often trained from scratch alongside the generator. They may not be able to acquire the full distribution of HQ images and lack enough discriminative power, resulting in sub-optimal Real-ISR performance.The recently developed T2I diffusion models such as SD [36] offer new options for us to formulate the loss L reg . These models, trained on billions of image-text pairs, can effectively depict the natural image distribution in latent space. Some score distillation methods have been reported to employ SD to optimize images by using the KL-divergence as the objective [49, 25, 43] . In particular, variational score distillation (VSD) [49, 58, 10] induces such a KL-divergence based objective from particlebased variational optimization to align the distributions represented by two diffusion models. Based on the above discussions, we propose to instantiate the learning objective in Eq. ( 1) by designing an efficient and effective one-step diffusion network. In specific, we finetune the pre-trained SD with LoRA [17] as our Real-ISR backbone network and employ VSD as our regularizer to align the distribution of network outputs with natural HQ images. The details are provided in the next section.Framework Overview. As discussed in Sec. 1, the existing SD-based Real-ISR methods [42, 57, 31, 52, 40] perform multiple timesteps to estimate the HQ image with random noise as the starting point and the LQ image as control signal. These approaches are resource-intensive and will inherently introduce randomness. Based on our formulation in Sec. 3.1, we propose a one-step effective diffusion (OSEDiff) network for Real-ISR, whose training framework is shown in Fig. 2 .Our generator G θ to be trained is composed of a trainable encoder E θ , a finetuned diffusion network ϵ θ and a frozen decoder D θ . To ensure the generalization capability of G θ , the output of the diffusion network ϵ θ will be sent to two regularizer networks, where VSD loss is performed in latent space.The regularization loss are back-propagated to update E θ and ϵ θ . Once training is finished, only the generator G θ will be used in inference. In the following, we will delve into the detailed architecture design of OSEDiff, as well as its associated training losses.Network Architecture Design. Let's denote by E ϕ , ϵ ϕ and D ϕ the VAE encoder, latent diffusion network and VAE decoder of a pretrained SD model, where ϕ denotes the model parameters. Inspired by the recent success of LoRA [17] in finetuning SD to downstream tasks [34, 35] , we adopt LoRA to fine-tune the pre-trained SD in the Real-ISR task to obtain the desired generator G θ .As shown in the left part of Fig. 2 , to maintain SD's original generation capacity, we introduce trainable LoRA [17] layers to the encoder E ϕ and the diffusion network ϵ ϕ , finetuning them into E θ and ϵ θ with our training data. For the decoder, we fix its parameters and directly set D θ = D ϕ . This is to ensure that the output space of the diffusion network remains consistent with the regularizers.Recall that the diffusion model diffuses the input latent feature z through z t = α t z + β t ϵ, where α t , β t are scalars that are dependent to diffusion timestep t ∈ {1, • • • , T } [16] . With a neural network that can predict the noise in z t , denoted as ε, the denoised latent can be obtained as ẑ0 = zt-βtε αt , which is expected to be cleaner and more photo-realistic than z t . Moreover, SD is a text-conditioned generation model. By extracting the text embeddings [36] , denoted by c y , from the given text description y, the noise prediction can be performed as ε = ϵ θ (z t ; t, c y ).We adapt the above text-to-image denoising process to the Real-ISR task, and formulate the LQ-to-HQ latent transformation F θ as a text-conditioned image-to-image denoising process as:EQUATIONwhere we conduct only one-step denoising on the LQ latent z L , without introducing any noise, at the T -th diffusion timestep. The denoising output ẑH is expected be more photo-realistic than z L . As for the text embeddings, we apply some text prompt extractor, such as the DAPE [52] , to LQ input x L , and obtain c y = Y(x L ). Finally, the whole LQ-to-HQ image synthesis can be written as:EQUATIONAs mentioned in Sec. 3.1, to improve the performance for a Real-ISR model, it is required to supervise the generator training with both the data term L data and regularization term L reg . As shown in the right part of Fig. 2 , we propose to adapt VSD [49] as the regularization term. Apart from utilizing the SD model as the pre-trained regularizer ϵ ϕ , VSD also introduces a finetuned regularizer, i.e., a latent diffusion module finetuned on the distribution q θ (x H ) of generated images with LoRA. We denote this finetuned diffusion module as ϵ ϕ ′ .Training Loss. We train the generator G θ with the data loss L data and regularization loss L reg . We set L data as the weighted sum of MSE loss and LPIPS loss:EQUATIONwhere λ 1 is a weighting scalar. As for L reg , we adopt the VSD loss via:EQUATIONGiven any trainable image-shape feature x, its latent code z = E ϕ (x) and encoded text prompt condition c y , VSD optimizes x to make it consistent with the text prompt y via:EQUATIONwhere the expectation of the gradient is conducted over all diffusion timesteps t ∈ {1, • • • , T } and ϵ ∼ N (0, I). Therefore, the overall training objective for the generator G θ is:EQUATIONwhere λ 2 is a weighting scalar. Besides, as required by VSD, the finetuned regularizer ϵ ϕ ′ should also be trainable, and its training objective is:EQUATIONNote that the above L diff loss is only applied to update ϵ ϕ ′ . The whole algorithm to illustrate the training pipeline can be found in the Appendix.VSD in Latent Space. The original VSD computes the gradients in the image space. When it is used to train an SD-based generator network, there will be repetitive latent decoding/encoding in computing L reg . This is costly and makes the regularization less effective. Considering the fact that a well-trained VAE should satisfyE ϕ (x) = E ϕ (D ϕ (z)) ≈ z, we can approximately let E ϕ (x H ) = ẑH .In this case, we can eliminate the redundant latent encoding/decoding in computing the regularization loss, as we follow DMD [58] to optimize the distribution loss in the latent state space rather than in the noise space. The gradient of the regularization loss w.r.t. the network parameter θ in the latent space is:EQUATION4 Experiments 
experimental settings: 
 Training and Testing Datasets. Prior works [42, 57, 31, 52] employed different training datasets, making unified training standards for Real-ISR difficult to establish. For simplicity, we adopt SeeSR's setup [52] and train OSEDiff using the LSDIR [26] dataset and the first 10K face images from FFHQ [19] . The degradation pipeline of Real-ESRGAN [45] is used to synthesize LQ-HQ training pairs. We evaluate OSEDiff and compare it with competing methods using the test set provided by StableSR [42] , including both synthetic and real-world data. The synthetic data includes 3000 images of size 512 × 512, whose GT are randomly cropped from DIV2K-Val [2] and degraded using the Real-ESRGAN pipeline [45] . The real-world data include LQ-HQ pairs from RealSR [3] and DRealSR [51] , with sizes of 128 × 128 and 512 × 512, respectively.Compared Methods. We compare OSEDiff with state-of-the-art DM-based Real-ISR methods, including StableSR [42] , ResShift [60] , PASD [57] , DiffBIR [31] , SeeSR [52] and SinSR [48] . Among them, StableSR, PASD, DiffBIR, and SeeSR are all based on the pre-trained SD model. ResShift trains a DM from scratch in the pixel domain, while SinSR is a one-step model distilled from ResShift. Note that we do not compare with the recent method SUPIR [59] because it tends to generate rich yet excessive details, which are however unfaithful to the input image.For those GAN-based Real-ISR methods, including BSRGAN [61] , Real-ESRGAN [45] , LDL [27] ,and FeMaSR [4] , we put their results into the Appendix.Evaluation Metrics. To provide a comprehensive and holistic assessment on the performance of different methods, we employ a range of full-reference and no-reference metrics. PSNR and SSIM [50] (calculated on the Y channel in YCbCr space) are reference-based fidelity measures, while LPIPS [64] , DISTS [12] are reference-based perceptual quality measures. FID [15] evaluates the distance of distributions between GT and restored images. NIQE [62] , MANIQA-pipal [55] , MUSIQ [22] , and CLIPIQA [41] are no-reference image quality measures. We also conduct a user study, which is presented in the Appendix.Implementation Details. We train OSEDiff with the AdamW optimizer [33] at a learning rate of 5e-5. The entire training process took approximately 1 day on 4 NVIDIA A100 GPUs with a batch size of 16. The rank of LoRA in the VAE Encoder, diffusion network, and finetuned regularizer is set to 4. For the text prompt extractor, although advanced multimodal language models [32] can provide detailed text descriptions, they come at a high inference cost. We adopt the degradation-aware prompt extraction (DAPE) module in SeeSR [52] to extract text prompts. The SD 2.1-base is used as the pre-trained T2I model. The weighting scalars λ 1 and λ 2 are set to 2 and 1, respectively.Quantitative Comparisons. The quantitative comparisons among the competing methods on the three datasets are presented in Table 1 . We can have the following observations. (1) First, OSEDiff exhibits clear advantages over competing methods in full-reference perceptual quality metrics LPIPS and DISTS, distribution alignment metric FID, and semantic quality metric CLIPIQA, especially Qualitative Comparisons. Fig. 3 Effectiveness of VSD Loss. To validate the effectiveness of our VSD loss in latent space, we perform ablation studies by removing the VSD loss, replacing it with the GAN loss used in [35] , and applying VSD loss in the image domain. The results on the RealSR test set are shown in Table 3 . We can see that without using the VSD loss, the perceptual quality metrics are significantly degraded because it is hard to ensure good visual quality using only MSE loss and even LPIPS loss [64] . Using GAN loss and VSD loss in the image domain can improve the performance, but the results are not as good as applying VSD loss in the latent domain. Our proposed OSEDiff can effectively align the distribution of Real-ISR outputs by performing VSD regularization in the latent domain.Comparison on Text Prompt Extractors. We then conduct experiments to evaluate the effect of different text prompt extractors on the Real-ISR results. We test three options. The first option does not employ text prompts. The second option uses the DAPE module in SeeSR [52] to extract degradation-aware tag-style prompts, as we used in our main experiments. The third option uses LLaVA-v1.5 [32] to extract long text descriptions after removing the degradation of input LQ images, as used in SUPIR [59] . We retrain the models based on different prompt extraction methods. The ablation results are shown in Table 4 .One can see that without using text prompts as inputs, those full-reference metrics such PSNR, SSIM, LPIPS, DISTS and even FID can improve, while those no-reference metrics such as MUSIQ, MANIQA and CLIPIQA become worse. By using DAPE and LLaVA to extract text prompts, the generation capability of the pre-trained T2I SD model can be triggered, resulting in richer synthesized details, which however can reduce the full-reference indices. A visual example is shown in Figure 4 . We see that while LLaVA extracts significantly longer text prompts than DAPE, they produce a similar amount of visual details. However, it is worth mentioning that the MLLM model LLaVA is very costly, increasing the inference time of DAPE by 170 times. Considering the cost-effectiveness, we ultimately choose DAPE as the text prompt extractor in OSEDiff. Setting of LoRA Rank. When finetuning the VAE encoder and the UNet, we need to set the rank of LoRA layers. Here we evaluate the effect of different LoRA ranks on the Real-ISR performance by using the RealSR benchmark [3] . The results are shown in Tables 5 and 6 , respectively. As shown in Table 5 , if a too small LoRA rank, such as 2, is set for the VAE encoder, the training will be unstable and cannot converge. On the other hand, if a higher LoRA rank, such as 8, is used for the VAE encoder, it may overfit in estimating image degradation, losing some image details in the output, as evidenced by the PSNR, DISTS, MUSIQ and NIQE indices. We find that setting the rank to 4 can achieve a balanced result for the VAE encoder. Similar conclusions can be made for the setting of LoRA rank on UNet. As can be seen from Table 6 , a rank of 4 strikes a good balance. Therefore, we set the rank as 4 for both the VAE encoder and UNet LoRA layers.The Finetuning on the VAE Encoder and Decoder. We conducted ablation studies to examine the impact of finetuning the VAE encoder and decoder, as shown Table 7 . In the first row, where neither the VAE encoder nor decoder is finetuned, the results show poor perception performance. Comparing with OSEDiff, where only the VAE encoder is finetuned, we observe significant improvements in perceptual quality (e.g., MUSIQ improves from 58.99 to 69.09). This demonstrates that finetuning the VAE encoder is important for removing degradation and enhancing overall performance. When comparing the third row, where both the VAE encoder and decoder are finetuned, with OSEDiff, where only the encoder is trained and the decoder is fixed, we note that OSEDiff also achieves better perceptual quality (CLIPIQ improves from 0.5778 to 0.6693). This indicates that fixing the VAE decoder is important to ensure that the UNet output remains in the original VAE latent space, which helps minimizing the VSD loss more effectively. Thus, finetuning the VAE encoder is important to remove degradation, while fixing the VAE decoder helps maintaining stability in the latent space, leading to better perceptual quality. 
conclusion and limitation: 
 We proposed OSEDiff, a one-step effective diffusion network for Real-ISR, by utilizing the pretrained text-to-image model as both the generator and the regularizer in training. Unlike traditional multi-step diffusion models, OSEDiff directly took the given LQ image as the starting point for diffusion, eliminating the uncertainty associated with random noise. By fine-tuning the pre-trained diffusion network with trainable LoRA layers, OSEDiff can well adapt to the complex real-world image degradations. Meanwhile, we performed the variational score distillation in the latent space to ensure that the model's predicted scores align with those of multi-step pre-trained models, enabling OSEDiff to efficiently produce HQ images in one diffusion step. Our experiments showed that OSEDiff achieved comparable or superior Real-ISR outcomes to previous multi-step diffusion-based methods in both objective metrics and subjective assessments. We believe our exploration can facilitate the practical application of pre-trained T2I models to Real-ISR tasks.There are some limitations of OSEDiff. First, the details generation capability of OSEDiff can be further improved. Second, like other SD-based methods, OSEDiff is limited in reconstructing fine-scale structures such as small scene texts. We will investigate these problems in further work. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Yes, we analyze the potential social impact in the final section.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: Codes and model will be released if the paper is accepted.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowd sourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In the appendix, we provide the following materials:• Comparison with GAN-based methods (referring to Section 4.1 in the main paper).• Results of user study (referring to Section 4.1 in the main paper).• More real-world visual comparisons under scaling factor 4× (referring to Section 4.2 in the main paper).• Training algorithm of OSEDiff (referring to Section 3.2 in the main paper).We compare OSEDiff with four representative GAN-based Real-ISR methods, including BSRGAN [61] , Real-ESRGAN [45] , LDL [27] and FeMaSR [4] . The results are shown in Table 8 . It is not a surprise that GAN-based methods have better fidelity measures such as PSNR and SSIM than OSEDiff. However, OSEDiff has much better perceptual qualify metrics. We also provide visual comparisons in Figure 5 . Compared to GAN-based methods, OSEDiff is able to generate realistic and reasonable details, such as squirrel hair, textures of petals, buildings, and leaves.To further validate the effectiveness of our proposed OSEDiff method, we conducted a user study by using 20 real-world LQ images. An LQ image and its HQ counterparts generated by different Real-ISR methods were presented to volunteers, who were asked to select the best HQ result. The volunteers were instructed to consider two factors when making their decisions: the image perceptual quality and and its content (including structure and texture) consistency with the LQ input, with each factor contributing equally to the final selection.We randomly selected 20 real-world LQ images from the RealLR200 dataset [52] . Figure 6 (a) shows the thumbnails used in the user study, cropped into squares for a convenient layout. We generated the HQ outputs of them by using the DM-based Real-ISR methods StableSR [42] , DiffBIR [31] , SeeSR [52] , PASD [57] , ResShift [60] , SinSR [48] , and OSEDiff. A number of 15 volunteers were invited to participate in the evaluation. The results are shown in Figure 6 (b). We see that OSEDiff ranks the second, just lagging slightly behind SeeSR. However, it should be noted that OSEDiff runs over 10 times faster than SeeSR by performing only one-step diffusion.Figure 7 provides more visual comparisons between OSEDiff and other DM-based methods. One can see that OSEDiff achieves comparable to or even better results than the multi-step diffusion methods in scenarios such as portraits, flower patterns, buildings, animal fur, and letters. The pseudo-code of our OSEDiff training algorithm is summarized as Algorithm 1. We follow [49, 58] and use classifier-free guidance (cfg) when calculating z ϕ .The cfg value is set to 7.5, and the negative prompts we use are: "painting, oil painting, illustration, drawing, art, sketch, oil painting, cartoon, CG Style, 3D render, unreal engine, blurring, dirty, messy, worst quality, low quality, frames, watermark, signature, jpeg artifacts, deformed, lowres, over-smooth." Justification: We detail the type of compute workers, memory, and execution time in the experiment section.AlgorithmGuidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: Yes, the research in the paper fully conforms to the NeurIPS Code of Ethics in every respect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This manuscript proposes a single-step image super-resolution model based on diffusion models, utilizing DMD loss for distillation, which results in faster generation compared to multi-step methods. All reviewers have provided positive or neutral scores, so the final decision for this manuscript is acceptance.
 
However, the manuscript has issues with insufficient citation of prior work. The proposed method, which employs LoRA finetuning, heavily relies on 'One-Step Image Translation with Text-to-Image Models' [1], which demonstrates the use of Stable Diffusion as a base model with LoRA finetuning, yet this work is not cited in the manuscript. The proposed loss function is essentially the DMD loss, but DMD [2] is not cited either. Additionally, contrary to what was mentioned in the authors' rebuttal, the use of LoRA for efficient finetuning is already provided in the official DMD code. Moreover, the manuscript bears significant similarity to SwiftBrush [3], but this work is not cited. It is strongly recommended that the authors cite the aforementioned prior works and include a discussion of them in the revised version of the manuscript.

Furthermore, there are concerns regarding the fairness of the experimental setup. While the proposed method and SeeSR[4] were trained on the high-quality LSDIR [5] dataset, other comparison methods used different datasets. This discrepancy in training data, particularly given the superior quality of LSDIR, leads to an unfair advantage and potentially unreliable comparison results. It is strongly recommended that the authors additionally provide experimental results using the same training datasets as the other methods. 

[1] Parmar, Gaurav, et al. "One-step image translation with text-to-image models." arXiv preprint arXiv:2403.12036.

[2] Yin, Tianwei, et al. "One-step diffusion with distribution matching distillation." CVPR. 2024.

[3] Nguyen, Thuan Hoang, and Anh Tran. "Swiftbrush: One-step text-to-image diffusion model with variational score distillation." CVPR 2024.

[4] Wu, Rongyuan, et al. "Seesr: Towards semantics-aware real-world image super-resolution."CVPR 2024.

[5] Li, Yawei, et al. "Lsdir: A large scale dataset for image restoration." CVPR 2023.

Reviews: review: paper_summary: paper_summary: The paper introduces OSEDiff, a one-step diffusion network for Real-World Image Super-Resolution. OSEDiff performs variational score distillation in the latent space to ensure predicted scores align with those of multi-step pre-trained models. ​ By fine-tuning the pre-trained diffusion network with trainable LoRA layers, OSEDiff can adapt to real-world image degradations. Experimental results show that OSEDiff achieves comparable or superior Real-ISR outcomes to previous methods. ​ The paper's contributions include the development of OSEDiff, which addresses the limitations of traditional multi-step diffusion models and enables efficient production of high-quality images in one diffusion step. ​
strength_weakness: strength_weakness: 1.	How much would the performance of OSEDiff change if multiple denoising steps were performed, such as 20 steps or 50 steps?
2.	The design of the key module, the regularizer networks, is not clearly explained.
3.	It is unclear how the LR image is used as a condition. Is it concatenated along the channel dimension in the latent space? However, the pre-trained SD’s VAE encoder downscales by a factor of eight, so for a 512x512 HR input, the latent output is only 64x64, which conflicts with the 128*128 LQ setting mentioned in the first paragraph of section 4.1.
4.	The reason for fine-tuning the VAE encoder is not very clear. Please explain in the revision why it is necessary to fine-tune the VAE encoder jointly with the diffusion model. If this has already been discussed, please indicate where it is covered.
5.	Does the model support input of LR images with different resolutions? For example, if the input is a 32x32 LR image, can feasible results be obtained?
questions: questions: Please refer to the weakness part
limitations: limitations: NA
paper_summary: paper_summary: This paper presents a novel approach to real-world image super-resolution for a one-step effective diffusion network (OSEDiff). The proposed OSEDiff effectively eliminates the uncertainty introduced by random noise sampling in previous methods, achieving significant performance improvements across multiple benchmarks, demonstrating its potential for practical applications.
strength_weakness: strength_weakness: 1. The core concept of this paper shares similarities with "One-step Diffusion with Distribution Matching Distillation". A comprehensive comparison with this work, including performance metrics, training efficiency (parameters and FLOPs), and potential limitations, would strengthen the paper's contribution.
2. While Table 2 provides a comparison of trainable parameters, including the total number of model parameters and FLOPs during inference would offer a more complete picture of computational efficiency.
3. There is a lack of discussion on the degradation in LQ images, such as how to ensure that text prompts remain accurate in low-quality scenarios.
4. In Figure 7 (first and second scenarios), some generated details and textures appear to be creatively added rather than strictly reconstructed from the LQ input. Addressing potential limitations in fidelity due to this creative aspect would be valuable.
questions: questions: No more questions.
limitations: limitations: The authors have discussed about the limitations.
paper_summary: paper_summary: This paper introduces a diffusion-based real-world image super-resolution method, OSEDiff, which can efficiently generate high-quality images in just one diffusion step. Firstly, in order to eliminate the uncertainty introduced by random noise sampling, the authors propose to directly feed the low-quality images without any random noise into the pre-trained SD model which integrates trainable LoRA layers. Furthermore, the variational score distillation for KL-divergence regularization is utilized to align the distribution of generated images with natural image prior, ensuring that the one-step diffusion model could still generate high-quality output. OSEDiff achieves comparable performance to state-of-the-art SD-based real-world image super-resolution methods with fewer trainable parameters and inference time.
strength_weakness: strength_weakness: (1) The motivation is unclear. In the introduction, the authors use a certain amount of space to mention two major issues in training a Real-ISR model, including the problem of how to build the LQ-HQ training image pairs, but this paper does not propose an innovative solution to this problem.

(2) The explanation of the novelty and its effectiveness are insufficient. 1) Contribution 1, "directly feeding the LQ images into the pre-trained SD model without introducing any random noise," claims to eliminate the uncertainty of output but lacks supporting evidence, raising doubts about its validity. It is recommended to provide visual evidence and analyze why this can eliminate the uncertainty and achieve better Real-ISR performance in detail. 2) Contribution 2, “utilizing variational score distillation for KL-divergence regularization,” it is recommended to provide some visualization of the output of diffusion network which are sent to the two regularizer networks to prove that it can align the distribution of generated images with natural image prior. 

(3) The metrics used need more explanation. Since CLIPIQA is text-related, although it has been proved to be effective and generalized in image quality assessment, it is not that convincing to directly apply it to evaluate the quality of real-world super-resolution images. More explanation can be provided.

(4) Details of the network structure are lacking in the figure. In Figure 2, the detailed network structure of VAE encoder and the diffusion network with trainable LoRA layers are lacking. It is recommended to provide detailed network structure.

(5) In “Ablation Study”, the super-resolution performance of the method with text prompt extractor is not competitive enough compared to removing text prompt. Although the qualitative comparisons show that the method with text prompt extractor can generate richer image details, the method without text prompt is much better for several common metrics.
questions: questions: (1) Could you please provide some visualization of the output of diffusion network which are sent to the two regularizer networks to further prove the effectiveness of the regularization loss in the latent space?

(2) Could you please explain in detail why "directly feeding the LQ images into the pre-trained SD model" can eliminate the uncertainty of output and achieve better Real-ISR performance than the methods which uses inputs with random noise?

(3) Could you please provide detailed network structure of VAE encoder and the diffusion network with trainable LoRA layers?

(4) Could you please provide more qualitative comparisons of the method with/without text prompt extractor to further prove the effectiveness of text prompt or further elaborate on the importance of the text prompt to prove that it is essential?
limitations: limitations: (1) Briefly explain why "directly feeding the LQ images into the pre-trained SD model" can eliminate the uncertainty of output, and why it can achieve better Real-ISR performance than the methods which uses inputs with random noise.

(2) Please provide some visualization of the output of diffusion network which are sent to the two regularizer networks.
paper_summary: paper_summary: In this paper, authors propose a novel one-step effective diffusion network, termed as OSEDiff, for the Real-ISR. The proposed OSEDiff adopts LQ image as the input and directly output the final output with the help of the decoder of VAE, thus eliminate the uncertainty introduced by random noise sampling and achieve one-step generation of diffusion model. Besides, the variational score distillation is introduced in the latent space to conduct KL-divergence regularization to enhance the quality of final Real-ISR output. In the experimental part, the proposed method outperforms other baselines in both quantitative and qualitative comparisons.
strength_weakness: strength_weakness: There are several weakness here:
1. Experimental part could be further improved, which will be further demonstrated in the following section.
2. The efficient and params of the proposed method is expected to be detailed demonstrated.
questions: questions: There are several concerns and suggestions here:
1. I am wondering the comparisons between the proposed method and SinSR. Does the improvement of performance gain from the pre-trained SD? 
2. Experiments of the paper is expected to be improved. For instance, total parameters instead of the training parameters are expected to be compared in the Table 2, which is more closely related to the deployment.
3. I am wondering the reason why the setting of "w/o VSD loss" may achieve better PSNR than proposed method. I would be better to provide more visual results of it to demonstrate the effectiveness of the VSD loss.
limitations: limitations: Could be referred to above.

Comments: title: Official Comment by Reviewer px3F
comment: Thanks for your response. All my concerns have been addressed. I will keep my positive rating.
We are pleased that this reviewer found our responses helpful, and we thank this reviewer for the further question on evaluation metrics, which is actually an open problem for real-world image super-resolution (Real-ISR) evaluation. 

**Q1. Performance evaluation metrics**

The key contribution of our work is the development of a fast one-step diffusion model for Real-ISR. As can be seen in the main paper, our OSEDiff model significantly reduces the inference time of previous SD-based models (e.g., it is 30x faster than StableSR and 10x faster than SeeSR). Apart from complexity and speed, for the performance evaluation metrics shown in Table 1 of our paper, we actually adopted the ones used in existing works in order for fair comparison with them. We didn’t deliberately select metrics which are advantages to our model. 

Those metrics can be generally classified into two categories: fidelity-oriented ones (e.g., PSNR, SSIM) and perception-oriented ones (e.g., NIQE, MUSIQ, CLIPIQA). For the Real-ISR task, usually we emphasize a little bit more on the perceptual quality of the restoration output. Unfortunately, by far the accurate perceptual image quality assessment (IQA) remains an open problem, and no one single metric can align with human visual perception very well. On the other hand, there are certain conflicts between fidelity-oriented metrics and perception-oriented ones. This also explains why one algorithm performs well on some metrics but not so well on other metrics. If an algorithm is tailored to some metrics, it may become disadvantageous on other metrics.
  
Being highly advantageous on inference speed, our approach shows comparable with or even superior results to the competing methods on the nine metrics. The visual comparisons also demonstrate its competitiveness. We will release our codes and trained models so that the peers can test OSEDiff’s performance on different scenarios. 

**Q2. Default models of comparison methods**

We follow the convention in the community to use the official models of competing methods for comparison, with the parameter settings recommended by the authors. We believe this is the fairest way to compare the different Real-ISR methods. If we tailor one model or change its parameter setting to improve some metrics, other metrics may be deteriorated (see our responses to Q1), and this is unfair for other competing methods.  Nonetheless, we agree that how to better evaluate the performance of Real-ISR models needs the collective efforts from the community.
Thank you very much for pointing out this problem. We carefully re-examined OSEDiff's inference code and found that it is a Python decorator function that consumes much time. This function is just to calculate the inference memory usage without affecting anything on the restoration process. We therefore removed this function and reassessed the inference time of different modules for both OSEDiff and SinSR. The results are shown in the table below:

| OSEDiff | DAPE | Text Encoder | VAE Encoder | UNet | VAE Decoder | SUM |
|:------------:|:------:|:------------:|:-----------:|:------:|:-----------:|:------:|
| FLOPs (G) | 104 | 22 | 542 | 355 | 1242 | 2265 |
| Time cost (s)| 0.023 | 0.010 | 0.020 | 0.026 | 0.037 | 0.116 |

| SinSR | VAE Encoder | UNet | VAE Decoder | SUM |
|:------------:|:-----------:|:------:|:-----------:|:------:|
| FLOPs (G) | 898 | 202 | 1549 | 2649 |
| Time cost (s)| 0.034 | 0.045 | 0.051 | 0.130 |

We can see that the overall time consumption of OSEDiff is 0.116s, which is actually less than SinSR (0.130s). Though the DAPE module in OSEDiff costs additional time, the VAE Encoder, UNet, and VAE Decoder modules of OSEDiff consume less time than those of SinSR. Note that although SinSR's UNet has lower FLOPs (202G) than OSEDiff's UNet (355G), its inference time is nearly doubled because SinSR's UNet employs Swin Transformer blocks. The frequent window partitioning operations elevate memory access and data movement costs, resulting in increased latency. Similar observations can be seen for OSEDiff's DAPE module, which also uses a Swin Transformer backbone. Its FLOPs is less than 1/3 of that of UNet, yet its time consumption is comparable to UNet.

The time consumption of SinSR reported in our main paper is 0.16s, which includes the time for converting the model output tensor to an image. We will change it to 0.130s in the revision for fair comparison.

Again, we sincerely thank this reviewer for the careful reading of our paper and indicating this problem. We will correct this issue in the revision.
Thanks for the authors' rebuttal. Most of my concerns have been addressed. I still have one question about the SR results in Table 1. We can see from this table that 9 metrics are used for performance evaluation, and part of them are better than other methods. In my opinion, the metrics results of different models of the proposed method may vary significantly. Some models can be chosen to perform well on part of the metrics. In fact, the same applies to the comparison methods, i.e., they can also be tailored to specific metrics like this paper. This paper directly uses the default models of comparison methods, and haven’t chosen models for metrics like this paper. Could this approach potentially make some comparison methods perform better than the proposed method? This can be analyzed in the revised version.
Many thanks for authors' response. Most of the concerns have been satisfied. The only concerns relies on the inference times, I hope the authors could demonstrate more on why the proposed OSEDiff obtains few FLOPs but more inference times than SinSR. It would be helpful for the readers.
I acknowledge that balancing fidelity and perceptual quality is challenging and hope the authors will provide more experiments and discussions on this issue. Thank the authors for addressing my concerns. I would like to raise my score.
**Q1. How to balance perception and fidelity in OSEDiff?**

We sincerely thank this reviewer for the recognition on our contribution. This reviewer's comments on the role of VSD loss in OSEDiff is correct. VSD was originally proposed for the 3D generation task. We adopt it as a regularization term on image distribution, aiming to generate details and enhance the naturalness of the SR output. Adding heavier noise in VSD loss will improve generative ability, while adding lighter noise will result in better fidelity. In our implementation, we uniformly sampled $t$ in the VSD loss from 1 to 1000 in each iteration. Kindly note that in addition to VSD loss, we also employed the $L_2$ loss and LPIPS loss as fidelity constraints in OSEDiff. We balance perceptual quality and fidelity by adjusting the weights on different loss terms. We empirically set the weights of $L_2$ loss, LPIPS loss, and VSD Loss as 1, 2 and 1, respectively, and found that a good balance on perceptual quality and fidelity can be achieved, as evidenced by the experiments in our manuscript.

**Q2. Would using a larger super-resolution model for distillation be a more reasonable approach?**

Thanks for the insightful comments. Actually, at the very beginning of this work, we indeed have tried to use a large diffusion-based SR model, more specifically SeeSR, as the regularization model to distil OSEDiff. However, the trained model suffered from weak detail generation ability. This is because SeeSR, as an image restoration model but not a generation model, is already a trade-off between fidelity and perception. Its generative ability is reduced to ensure the fidelity of SR outputs. When using SeeSR as the regularization model, it is difficult to endow the OSEDiff model enough capacity to generate image details. Therefore, we directly use a pre-trained SD model as the regularization model.
Thank the authors for the careful reply. While the implementation draws some insights from ProlificDreamer and DMD, this paper is the first to address super-resolution tasks, and its contribution is valuable to the community. However, I have a further question regarding the VSD loss. Although the ablation experiments demonstrate its effectiveness, I have concerns. In super-resolution, the output must add details and remain consistent with the input. If the noise added in VSD loss is minimal, denoising might yield results similar to the original, making it hard to enhance details. On the other hand, too much noise could increase detail but compromise fidelity. Balancing these factors is challenging, which makes VSD more effective for generation tasks but less theoretically clear for image restoration. Could the authors offer a more illustration about this point? Would using a larger super-resolution model for distillation be a more reasonable approach?
We sincerely thank this reviewer for the positive feedback. For sure we will release the training codes and all the trained models soon. 

Authors of paper 4246
Thanks for the response. The authors have solved my problems and I am about to maintain my positive recommendation; hope to see the VSD training code soon.