Taming Diffusion Prior for Image Super-Resolution with Domain Shift SDEs

Authors: qinpeng  cui
yixuan  liu
xinyi  zhang
qiqi  bao
qingmin  liao
li  wang
tian  lu
zicheng  liu
zhongdao  wang
emad  barsoum
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 08:39:19

Abstract: Diffusion-based image super-resolution (SR) models have attracted substantial interest due to their powerful image restoration capabilities. However, prevailing diffusion models often struggle to strike an optimal balance between efficiency and performance. Typically, they either neglect to exploit the potential of existing extensive pretrained models, limiting their generative capacity, or they necessitate a dozens of forward passes starting from random noises, compromising inference efficiency. In this paper, we present DoSSR, a $\textbf{Do}$main $\textbf{S}$hift diffusion-based SR model that capitalizes on the generative powers of pretrained diffusion models while significantly enhancing efficiency by initiating the diffusion process with low-resolution (LR) images. At the core of our approach is a domain shift equation that integrates seamlessly with existing diffusion models. This integration not only improves the use of diffusion prior but also boosts inference efficiency. Moreover, we advance our method by transitioning the discrete shift process to a continuous formulation, termed as DoS-SDEs. This advancement leads to the fast and customized solvers that further enhance sampling efficiency. Empirical results demonstrate that our proposed method achieves state-of-the-art performance on synthetic and real-world datasets, while notably requiring $\textbf{\emph{only 5 sampling steps}}$. Compared to previous diffusion prior based methods, our approach achieves a remarkable speedup of 5-7 times, demonstrating its superior efficiency.

Main paper: 
 introduction: 
 Image super-resolution (SR) is a classical task in computer vision that involves enhancing a lowresolution (LR) image to create a perceptually convincing high-resolution (HR) image [28] . Traditionally, this field has operated under the assumption of simple image degradations, such as bicubic down-sampling, which has led to the development of numerous effective SR models [6, 25, 57, 12] . However, these models often fall short when confronted with real-world degradations, which are typically more complex than those assumed in academic settings. Recently, diffusion models has emerged as a pivotal research direction in real-world SR, using their robust generative capabilities to enhance perceptual quality. This shift highlights their superior performance in practical applications.Currently, diffusion-based SR strategies can be broadly categorized into two approaches. The first approach leverages large-scale pretrained diffusion models (e.g., Stable Diffusion [41] ) as generative prior, using LR images (or preprocessed LR images) as conditional inputs to generate HR images [45, 27, 51] . Despite achieving remarkable results, it exhibits low inference efficiency, as the inference starting point is a random Gaussian noise instead of the LR image. Although techniques such as sampler optimization [33, 38, 30, 9, 22] or model distillation [32, 37] have been proposed to mitigate this issue, they inevitably compromise SR performance. The second approach involves redefining the diffusion process and retraining a model from scratch for the SR task [53, 36] . Consequently, the generative prior from pretrained diffusion models is not leveraged. ResShift [53] , as a typical representative, revises the forward process of DDPM [16] to better accommodate the SR task. By starting from LR rather than Gaussian noise, it improves inference efficiency. However, its modification of the diffusion pattern, which deviates significantly from existing noise schedules in diffusion models, hinders its integration with large-scale pretrained diffusion models for leveraging their generative prior. The diffusion generative prior has been proven to be highly beneficial for SR tasks [45] , enabling models to transcend the limitations of knowledge learned solely from the training dataset, thereby equipping them to handle various complex real-world scenarios. Thus, crafting a diffusion process tailored for the SR task that also remains compatible with established diffusion prior presents a significant challenge.To tackle this challenge, we propose DoSSR, a Domain Shift diffusion-based SR model. We initially view the SR task as a gradual shift from the LR domain to the HR domain, describing this transition with a linear equation, which is called domain shift equation. Then, we combine this domain shift equation with existing diffusion equations, facilitating the fine-tuning of large-scale pretrained diffusion models to harness diffusion prior effectively. Moreover, by carefully designing a shifting sequence, inference can begin from LR images rather than Gaussian noises, thereby boosting inference efficiency. To further enhance efficiency, we employ sampler optimization techniques, extensively explored in image generation [38, 30, 9] , but not previously tailored for diffusion-based SR tasks. Specifically, we expand the customized diffusion equation from discrete to continuous, enabling its formulation as stochastic differential equations (SDEs). We subsequently present the corresponding backward-time SDE as Domain Shift SDE in the reverse process and provide an exact formulation of its solution. Based on our formulation, we customize fast solvers for sampling. Experimental results demonstrate that our method achieves superior or comparable performance compared to current state-of-the-art methods on both synthetic and real-world datasets, with only 5 sampling steps, striking an optimal balance between efficiency and effectiveness. Furthermore, our approach can match the performance of previous methods even with just a single step.In summary, the main contributions of our work are as follows:• We propose a novel diffusion equation, which models SR from the perspective of domain shift, enabling inference to start from LR images and leveraging diffusion prior to ensure both efficiency and performance. • We further propose the SDEs to describe the process of domain shift and provide an exact solution for the corresponding reverse-time SDEs. Based on the solution, we design customized fast samplers, resulting in even higher efficiency, thereby achieving the state-of-the-art efficiencyperformance trade-off. 
related work: 
 Neural Network-based Super-Resolution. Neural network-based methods have emerged as the dominant approach in image SR tasks. The introduction of convolutional neural networks (CNNs) and Transformer architecture, with the primary focus on network architecture design [12, 10, 25, 26, 58, 21, 56, 57] , have demonstrated superior performance over traditional methods. This improvement is facilitated by the introduction of residual blocks, dense blocks and attention mechanisms. These methods primarily aim for better image fidelity measures such as PSNR and SSIM [49] indices, therefore, they often yield over-smoothed outcomes. To enhance visual perception, Generative adversarial network (GAN)-based SR methods have been developed. By incorporating adversarial loss during training, many SR models [13, 23, 17] can generate perceptually realistic details, thereby enhancing visual quality. To further study SR problems in real-world scenarios, some studies [54, 46, 24] have proposed simulating the intricate real-world image degradation process through random combinations of fundamental degradation operations. Despite the remarkable advancements, GANbased SR methods can introduce undesirable visual artifacts.Diffusion-based Super-Resolution. Recently, diffusion-based SR methods [35, 36, 8, 7, 45, 27] have demonstrated excellent performance, especially in terms of perceptual quality. These methods can generate more authentic details while avoiding unpleasant visual artifacts like GAN-based methods. Current diffusion models for super-resolution can be broadly categorized into two main approaches. The first approach involves leveraging large-scale pretrained diffusion models, such as Stable Diffusion [41] , as prior, and then using LR images as conditional inputs to generate HR images. StableSR [45] and DiffBIR [27] represent representative works that leverage diffusion prior, leading to enhanced fidelity when conditioning on LR or preprocessed LR. SeeSR [51] and CoSeR [42] demonstrate that extracting semantic text information from LR images as additional control conditions for the T2I model helps improve performance. The second approach involves redefining the diffusion process and retraining a model from scratch for SR [18, 36] . To address the slow inference speed issue of diffusion-based SR methods, ResShift [53] constructs a Markov chain that transitions between HR and LR images by shifting residuals between them, enabling accelerated sampling. SinSR [48] proposed a method of distilling ResShift to achieve comparable performance in a single step. Despite the remarkable advancements achieved by ResShift and SinSR, they necessitate retraining from scratch for SR tasks (or further distillation) and are unable to leverage diffusion prior. Therefore, improving the inference efficiency while leveraging the potential of large-scale pretrained diffusion models to assist SR requires thorough investigation, which is the goal of this work.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: In Appendix 5, we expound upon the limitations of the work conducted and provide a brief discussion thereof. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 We aim to optimize the balance between efficiency and performance in diffusion-based superresolution (SR) models. Our approach is grounded in two key principles: First, initiating inference from LR images rather than noise; Second, effectively harnessing pretrained diffusion prior. In Section 3.1, we introduce a novel diffusion equation designed to fulfill both criteria simultaneously. Subsequently, in Section 3.2, we extend this diffusion process to continuous scenarios, formulating it through Stochastic Differential Equations (SDEs). Building on these SDEs, we develop an efficient solver detailed in Section 3.3, further enhancing inference efficiency.Our goal is to characterize the shift from the source domain to the target domain as a diffusion process. In the task of SR, the distribution of LR images p data ( x0 ) represents the source domain, while the distribution of HR images p data (x 0 ) represents the target domain. Firstly, we conceptualize domain shift as a gradual transition from the source domain to the target domain through a linear drift coefficient η t , the domain shift equation is formulated asEQUATIONwhere shifting sequence {η t } T t=1 monotonically non-decreases with timestep t. In order to enable linear combination, we can interpolate x0 to match the same dimensions as x 0 if necessary. Secondly, we combine this domain shift with the diffusion equation. To integrate with pretrained diffusion models, we adopt the most commonly used diffusion scheme from DDPM [16] and express the formula of marginal distribution at any timestep t as follows:EQUATIONwhere α t , σ t ≥ 0 and α 2 t + σ 2 t = 1, I is the identity matrix. Based on our proposed marginal distribution Eq. ( 2), we demonstrate the transition distribution as follows:EQUATIONwhere e 0 = x0 -x 0 is the residual between the source and target domain.Relation to DDPM [16] . The formulation of Eq. ( 2) is based on the DDPM [16] forward process, with a crucial difference lying in its mean α t D( x0 , x 0 ) instead of α t x 0 . This integration encapsulates the domain shift within the variation of its mean, while the diffusion process with added noise maintains consistency with it, thereby smoothing this transformation. Meanwhile, it enhances the diffusion model to learn the pathway from the source domain to the target domain. For an intuitive understanding, we plot and compare the score function ∇ x log q t (x t ) learned by a vanilla Stable Diffusion (SD) model and DoSSR in Fig. 2(c ). While SD learns reasonable score field in the whole space, DoSSR inherits its capability in the ambient space and further learns more accurate scores along the path between LR and HR domains. Therefore sampling efficiency is improved. Furthermore, the choice of α t and σ t , referred to as the noise schedule, follows the existing pretrained diffusion model, allowing us to fine-tune it rather than training it from scratch.Relation to ResShift [53] . The form of equation Eq. (3) suggests that this shift essentially constructs a Markov chain in a manner similar to that described in ResShift [53] . However, the equation constructed by ResShift adopts an entirely different noise schedule compared to the pretrained diffusion model. This makes it difficult to apply pretrained diffusion models for subsequent finetuning, necessitating training from scratch instead. Therefore, it is unable to utilize the diffusion prior, thereby limiting the model's performance. See Appendix A.7 for detailed theoretical differences from ResShift. In Appendix C.1, we present experimental results on ImageNet [11] showing that DoSSR uses two orders of magnitude less training data than ResShift while achieving superior performance.Shifting Sequence. The parameter η t plays a crucial role in guiding the diffusion process, serving as a bridge between the source and target domains. Specifically, η t = 1 represents standard diffusion forward perturbations in the source domain, whereas η t = 0 corresponds to the target domain. The transition between these domains occurs for 0 < η t < 1, indicating a domain shift. To effectively utilize the diffusion prior, we adopt the noise schedule from DDPM. This adoption dictates that as t approaches the final time step T , the scale parameter α T tends towards zero, and the distribution q(x T ) approximates a standard Gaussian, N (0, I). To retain prior information from the source domain while shortening the diffusion path, we set η t = 1 for t ∈ [t 1 , T ], as defined by:EQUATIONThe advantage of such a setting lies in the fact that during the reverse process, the values of x t for t ∈ [t 1 , T ] are known and can be obtained through the forward process Eq. ( 2). Consequently, the inference does not need to start from time step T , but can commence at t 1 , thereby preserving the prior information of the source domain while enhancing the efficiency of inference. An overview of the impact of η t is presented in Fig. 2 .To improve the efficiency of inference in diffusion models, many prior works [30, 31, 9] have designed efficient samplers by solving the diffusion SDEs. Therefore, in this section, we extend the aforementioned discrete shift process to an SDE for description, in preparation for designing efficient samplers in the following section. Specifically, inspired by the work of [40] , we generalize this finite shift process further to an infinite number of noise scales, such that the data distribution of domain shift evolves according to an SDE as noise intensifies. Then we provide the corresponding reversetime SDE and elucidate the training of diffusion models from the perspective of score matching [39] .Next, we will elaborate extensively on how to describe diffusion models using SDEs.Forward Process. Expanding the time variable t in Eq. ( 2) to a continuous range, t ∈ [0, T ], we have that α t , σ t , η t are differentiable functions of t with bounded derivatives. Furthermore, Song et al. [40] have demonstrated that the diffusion process can be modeled as the solution to an Itô SDE and we formulate the SDE as follows:EQUATIONwhere w t is the standard Wiener process, and q 0 (x 0 ) is the target domain data distribution. It has the same marginal distribution q(x t |x 0 , x0 ) as in Eq. (2) for any t ∈ [0, T ] with the coefficients satisfying (proof in Appendix A.2)EQUATIONReverse Process. The reverse of a diffusion process is also a diffusion process [2] which can similarly be described by a reverse-time SDE (proof in Appendix A.3):EQUATIONwhere w t is also a standard Wiener process when time flows backwards. In this paper, we refer to this SDE as Domain Shift SDE (DoS-SDE).Score Matching. The only unknown term in Eq. ( 7) is the score function ∇ x log q t (x t ) that can be estimated by training a score-based model on samples with score matching [39] . In practice, we use a neural network ϵ θ (x t , x0 , t) conditioned on x0 , parameterized by θ, to estimate the scaled score function (alternatively referred to as noise), following [16, 40] . The parameter θ is optimized by minimizing the following objectives:EQUATIONwhere w(t) is a weighting function, x t = α t (η t x0 + (1 -η t )x 0 ) + σ t ϵ, and ϵ ∼ N (0, I).Thus, we have completed the expression of the diffusion model using SDEs. Sampling from diffusion models can alternatively be seen as solving the corresponding diffusion DoS-SDEs.In this section, we present an exact formulation of the solution of diffusion DoS-SDEs and design efficient samplers for fast sampling. To facilitate the solution of equation Eq. ( 7), we utilize the data prediction model x θ (x t , x0 , t), which directly estimates the original target data x 0 from the noisy samples. The relationship between score function and data prediction model is as follows (proof in Appendix A.4):EQUATIONIn practice, we employ our trained noise prediction model ϵ θ (x t , x0 , t) for data prediction x θ (x t , x0 , t) as described in Appendix A. 4 . By substituting Eq. ( 6) and Eq. ( 9) into Eq. ( 7) and introducing the substitutions λ t = σt αt(1-ηt) and y t = xt αt(1-ηt) along with the notation dw λt := dλt dt dw t , x λ := x t(λ) , w λ := w λt , we rewrite Eq. ( 7) w.r.t λ asEQUATIONWe propose the exact solution for Eq. ( 10) using the variation-of-constants formula, following [31, 9] . Proposition 3.1 (Exact solution of diffusion DoS-SDEs). Given an initial value x s at time s > 0, the solution x t for the diffusion DoS-SDEs defined in Eq. (7) at time t ∈ [0, s] is as follows:EQUATIONwhere λ t = σt αt(1-ηt) and z s ∼ N (0, I).The detailed derivation of this proposition is provided in Appendix A.5. Notably, the nonlinear term in Eq. (11) involves the integration of a non-analytical neural network x θ (x λ , x0 , λ), which can be challenging to compute. For practical applicability, we employ Itô-Taylor expansion to approximate the integral of x θ from λ s to λ t to compute xt , thereby approximating x t . Additionally, we approximate the derivatives of x θ using the forward differential method. These approximations allow us to derive SDE solvers of any order for diffusion DoS-SDEs. For the sake of brevity, we employ a first-order solver for demonstration. In this case, Eq. ( 11) becomesEQUATIONThe detailed derivation, as well as high-order solvers, are provided in Appendix A.6, and detailed algorithms are proposed in Appendix B. Typically, higher-order solvers converge even faster because of more accurate estimation of the the nonlinear integral term. The solvers provided for sampling allow us to iteratively generate HR images using a trained diffusion model. It is worth noting that Eq. ( 12) comprises four terms, including the additional linear term x0 , as compared to the ancestral sampling algorithm [16] . We refer to this additional term as the domain shift guidance (DoSG) which leverages prior information from the source domain and enhances the efficiency of inference.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: The code will be released once the submission is accepted.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experimental setup: 
 For training, we train our DoSSR using a variety of datasets including DIV2K [1] , DIV8K [15] , Flickr2K [43] , and OST [47] . To synthesize LR and HR training pairs, we adopt the degradation pipeline from RealESRGAN [46] . For the network architecture, we employ the LAControlNet [27] with SD 2.1-base 3 as the pretrained T2I model. In cases where LR images are severely degraded, potentially leading to the diffusion model mistaking degradation for semantic content, we implement RealESRNet [46] as a preprocessing step. This ensures our source domain consists of preprocessed LR images, thereby refining the input quality for better model training and performance. The model is fine-tuned for 50k iterations using the Adam optimizer [20] , with a batch size of 32 and a learning rate set to 5 × 10 -5 , on 512 × 512 resolution images.For testing, we evaluate our method on both synthetic and real-world datasets, employing the same configuration as StableSR 4 . For synthetic data, we randomly crop 3K patches with a resolution of 512 × 512 from the DIV2K validation set [1] , and degrade them following the degradation pipeline of RealESRGAN [46] . For real-world datasets, we generate LR images with a resolution of 128 × 128 by center-cropping on RealSR [4] , DRealSR [50] and RealLR200 [51] .We compare DoSSR with the state-of-the-art real-world SR methods, including BSRGAN [54] , Real-ESRGAN [46] , LDL [23] , DASR [24] , StableSR [45] , ResShift [53] , DiffBIR [27] , and SeeSR [51] .We use the publicly available codes and pretrained models to facilitate fair comparisons.Quantitative Comparison. We show the quantitative comparison on the four synthetic and realworld datasets in Table 1 . To comprehensively evaluate the performance of various methods, we utilize the following metrics 5 for quantitative comparison: reference-based metrics PSNR, SSIM [49] , LPIPS [55] , and non-reference metrics CLIPIQA [44] , MUSIQ [19] , MANIQA [52] , TOPIQ [5] . Notably, DoSSR consistently achieves the highest scores in CLIPIQA, MANIQA, and TOPIQ, with the exception of being second in TOPIQ on DIV2K, and attains the second highest score in MUSIQ across all four datasets. At the same time, we also note that diffusion-based methods generally achieve poorer performance in reference metrics compared to GAN-based methods due to their ability to generate more realistic details at the expense of fidelity. Additionally, our DoSSR manages to achieve improved no-reference metric performance compared to the data presented in Table 1 as NFE increases slightly, a detail further elaborated on in Section 4. Efficiency Comparison. The comparative analysis of model parameters and latency for competing SR models is shown in Fig. 1 (a) and Table 1 . The latency is calculated on the ×4 SR task for 128×128 LR images with V100 GPU. StableSR, DiffBIR, SeeSR, and our DoSSR utilize the pretrained SD model, resulting in a similar parameter count, with SeeSR incorporating a prompt extractor to enhance SR results, making it the largest among these methods. ResShift, utilizing the network structure from LDM [35] , is trained from scratch and has significantly fewer parameters. It employs a 15step process to achieve faster inference speeds. Among the pretrained SD-based methods, DoSSR demonstrates superior performance efficiency, requiring only 5 function evaluations to achieve speeds 5-7 times faster than previous SD-based models such as SeeSR. Additionally, DoSSR not only demonstrates faster or comparable latency to ResShift but also achieves significantly better super-resolution performance. Effectiveness of DoSG. To verify the effectiveness of the DoSG introduced in the diffusion equation, we conduct an experiment using identical network architectures but with two different diffusion equations: the original diffusion equation as described by Ho et al. [16] and our newly formulated equation (Eq. ( 2)). To isolate the impact of DoSG from our shifting sequence design, we set t 1 = T , ensuring that the starting point of our inference in both scenarios approximates Gaussian noise. Quantitative comparisons can be found in the first two rows of Table 2 . It is evident that the introduction of DoSG leads to a significant improvement across all metrics in the table, highlighting the effectiveness of DoSG in enhancing the performance of diffusion-based SR models. Additionally, it is worth noting that the original diffusion equation can be considered a special case within our framework where η t = 0 and t 1 = T . Therefore, our sampler can accommodate the original diffusion equation, and for a fair comparison, we employ the same sampler for both models. More comprehensive comparison is provided in Appendix Table 6 , where it can be seen that our DoSSR demonstrates superior performance compared to the corresponding order solver with DDPM, benefiting from the inclusion of DoSG in our DoS SDE-Solver. The number of step. We assess the impact of different inference steps on DoSSR by analyzing changes in representative metrics for both reference-based and non-reference-based evaluations, as shown in Fig. 4 (a). As the number of inference steps increases, reference-based metrics tend to decline, suggesting a loss in fidelity, while non-reference metrics improve, indicating enhanced realism and detail in the generated images. We also conduct visual comparisons in Fig. 4 The order of our sampler. We provide a suite of solvers for sampling in our DoSSR model, including a first-order solver presented in Eq. ( 11), and more advanced second-and third-order solvers detailed in Appendix A.6. We investigate the impact of samplers with different orders on our experimental results through qualitative and quantitative comparisons, as illustrated in Table 3 and Fig. 10 . From Table 3 , it becomes evident that high-order samplers can achieve superior non-reference metrics under the same limited inference step conditions. This is because the acceleration of higher-order samplers allows diffusion models to generate more details, as demonstrated in the first example of Fig. 10 , where the tower generated by the high-order sampler exhibits richer textures. More comprehensive comparison is provided in Appendix Table 6 . In our implementation, we use third-order sampler by default. 
conclusion: 
 In this paper, we present DoSSR, a diffusion-based super-resolution framework that significantly enhances both efficiency and performance by integrating a domain shift strategy with pretrained diffusion models. This approach not only enhances generative capacity but also enhances further inference efficiency through our novel proposed DoS-SDEs formulation and customized solvers. Empirical validation on diverse SR benchmarks confirms that DoSSR achieves a 5-7 times speed improvement over existing methods, setting a new state-of-the-art. Our work paves the way for more efficient diffusion-based solutions in image super-resolution.Limitation. Despite the strong overall performance demonstrated by the proposed DoSSR, it occasionally generates visually unfriendly details when employing an unfavorable random seed, a challenge also encountered by other diffusion-based methods. Typically, we fix the random seed for all image super-resolution tasks to stabilize the results, but this particular seed may not be suitable for certain specific images. As depicted in Fig. 8 , different initializations of random seeds result in significant variations in the details of the lion's eyes. Some of the initialized random seeds produce eyes that are reasonable and acceptable, while others exhibit noticeable inconsistencies with LR.For bad cases, we can also obtain a satisfactory result by adjusting the random seed multiple times. However, this often requires numerous attempts, and the quality of the results heavily relies on luck. This inspires us to find a suitable initialization for each specific LR image, which can enhance the performance of the model. Hence, for diffusion-based methods, exploring how to obtain a reasonable random seed based on known LR images may be a future research direction.Societal impact. Our advancements in the diffusion-based image super-resolution model, DoSSR, present both positive and negative societal impacts. On the positive side, it enhances medical imaging, potentially leading to more accurate diagnoses and reducing the need for invasive procedures. In surveillance, it aids in better identification and tracking, improving public safety. Moreover, in remote sensing and environmental monitoring, it facilitates informed decision-making for disaster management and environmental conservation. However, there are concerns regarding privacy and surveillance. Enhanced resolution capabilities could infringe upon privacy rights and lead to increased surveillance in public spaces, raising questions about civil liberties. Additionally, in digital media, while high-resolution imagery enhances visual content, it may perpetuate unrealistic beauty standards and digital manipulation, impacting self-esteem. In summary, while DoSSR brings promising advancements, it's crucial to address concerns around privacy, security, and digital ethics to ensure responsible and ethical deployment of the technology. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [NA]Justification: The experiments conducted in our paper do not involve the use of error bars or statistical significance analysis, thus this aspect is not applicable to our study.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: For the latency testing experiments, we furnished detailed specifications of the GPU models used along with their corresponding tasks. Furthermore, we included specific information regarding the model training batch size and the number of training epochs.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We discuss both potential positive societal impacts and negative societal impacts of the work performed in Appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of assets, such as code, data, or models, used in the paper, are properly credited. Additionally, the license and terms of use associated with these assets are explicitly mentioned and respected in accordance with ethical and legal standards.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: Our paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Derivation of Eq.( 3)According to Eq. ( 2), we can express x t as a linear combination of x 0 , x0 and a noise variable ϵ:EQUATIONSubsequently, the relationship between x t and x t-1 is derived as follows:x t = α t (η t-1 x0 + (1 -η t-1 )x 0 + (η t -η t-1 )( x0 -x 0 )) + σ t ϵ = α t α t-1 α t-1 (η t-1 x0 + (1 -η t-1 )x 0 ) + σ t-1 ϵ 1 + α t (η t -η t-1 )e 0 + σ 2 t - α 2 t α 2 t-1 σ 2 t-1 ϵ 2 = α t α t-1 x t-1 + α t (η t -η t-1 )e 0 + σ 2 t - α 2 t α 2 t-1 σ 2 t-1 ϵ 2where e 0 = x0 -x 0 , and ϵ, ϵ 1 , ϵ 2 ∼ N (0, I). Taking into account α 2 t + σ 2 t = 1, the above equation can be further simplified as follows:EQUATIONHence, the transition distribution between x t and x t-1 is as follows:EQUATIONA.2 Derivation of Eq.( 6)In this section, we derive the coefficients of the forward SDE. Discretizing Eq. ( 5) yields:x t+∆t -x t = f (t)x t ∆t + h(t) x0 ∆t + g(t)√ ∆tz 1 , where z 1 ∼ N (0, I)x t+∆t = (f (t)∆t + 1)x t + h(t) x0 ∆t + g(t)√ ∆tz 1 .(16) Substituting Eq. ( 13) into Eq. ( 16), we haveEQUATIONwhere z 2 , z ∼ N (0, I). For Eq. ( 13) at time t + ∆t, we have:EQUATION)Equating the corresponding parts of Eq. ( 17) and Eq. ( 18) yields:EQUATIONThen, letting ∆t → 0, the aforementioned three equations can be solved separately to yield:EQUATION)A.3 Derivation of Eq.( 7)As outlined in Sec. 3.2, the forward process can be expressed as the SDE shown in Eq. ( 5). In accordance with the Fokker-Plank Equation [34] , we obtain:∂q t (x t ) ∂t = -∇ x {[f (t)x t + h(t) x0 ]q t (x t )} + ∂ ∂x i ∂x j [ 1 2 g 2 (t)q t (x t )] = -∇ x {[f (t)x t + h(t) x0 ]q t (x t )} + ∇ x [ 1 2 g 2 (t)∇ x q t (x t )] = -∇ x {[f (t)x t + h(t) x0 ]q t (x t )} + ∇ x [ 1 2 g 2 (t)q t (x t )∇ x log q t (x t )] = -∇ x {[f (t)x t + h(t) x0 - 1 2 g 2 (t)∇ x log q t (x t )]q t (x t )},where q(x t ) denotes the probability density function of state x t . Most process defined by a forwardtime or conventional diffusion equation model possess a corresponding reverse-time model [2] , which can be formulated as:EQUATIONAccording to the backward Fokker-Plank Equation [34] , we have:∂q t (x t ) ∂t = -∇ x [µ(t, x t )q t (x t )] - ∂ ∂x i ∂x j [ 1 2 σ 2 (t, x t )q t (x t )] = -∇ x [µ(t, x t )q t (x t )] -∇ x [ 1 2 σ 2 (t, x t )∇ x q t (x t )] = -∇ x [µ(t, x t )q t (x t )] -∇ x [ 1 2 σ 2 (t, x t )q t (x t )∇ x log q t (x t )] = -∇ x {[µ(t, x t ) + 1 2 σ 2 (t, x t )∇ x log q t (x t )]q t (x t )}.Our goal is for the reverse process to have the same distribution as the forward process, specifically:EQUATIONTypically, we set σ(t, x t ) = g(t) [40] , yielding:EQUATIONTherefore, the reverse-time SDE can be expressed as follows:EQUATIONA.4 Derivation of Eq.( 9)The data prediction model x θ (x t , x0 , t) directly estimates the original target data x 0 from the noisy samples, indicating that x θ (x t , x0 , t) ≈ x 0 . Based on Eq. ( 2), the expression for q t (x t ) can be formulated as follows:EQUATIONHence, score function is:EQUATIONSubstituting x θ (x t , x0 , t) ≈ x 0 , we can establish the relationship between the score function and the data prediction model:∇ x log q t (x t ) = - x t -(α t (1 -η t )x θ (x t , x0 , t) + α t η t x0 ) σ 2 t (27)Furthermore, Eq. (8) shows that the noise prediction model is to estimate ϵ θ (x t , x0 , t) ≈ -σ t ∇ x log q t (x t ). (28) Hence, the relationship between the noise prediction model and the data prediction model is:EQUATIONindicating that we easily use trained noise prediction model for data prediction through the equation.In this section, we derive the solution to the equation Eq. (7) . By substituting Eq. ( 9) and Eq. ( 20) into Eq. ( 7), we obtain:EQUATIONCombining like terms, we get:EQUATIONEQUATIONSubsequently, dividing both sides by α t (1 -η t ) simultaneously, we have:EQUATIONLet λ t = σt αt(1-ηt) . Then λ t is monotonically increasing, and we have:EQUATIONTherefore, we have:EQUATIONBy performing the variable substitution y t = xt αt(1-ηt) and then substituting Eq. ( 34) into Eq. ( 32), we can simplify to obtain:EQUATIONDenoting dw λt := dλt dt dw t , x λ := x t(λ) , w λ := w λt , we rewrite the equation above w.r.t λ asEQUATION)Utilizing the variation-of-constants formula to solve the equation above, we obtain y t = eSimplifying and substituting back x t = α t (1 -η t )y t , we obtainEQUATIONThus, we obtain the exact solution to the DoS-SDEs.A.6 Derivation of Solvers for Diffusion DoS-SDEsDenote x (n) θ (x λ , x0 , λ) := d n x θ (x λ , x0,λ) dλ nas the n-th order total derivative of x θ (x λ , λ) w.r.t λ. For k ≥ 1, the k -1-th order Itô-Taylor expansion of x θ (x λ , x0 , λ) w.r.t λ at s isEQUATIONwhere the residual R k comprises of deterministic iterated integrals of length greater than k and all iterated with at least one stochastic component.Substituting the above Itô-Taylor expansion into Eq. ( 38) yieldsEQUATIONwhere Rk can be easily obtained from R k and the integral λt λs2λ 2 t λ 3 (λ-λs) n n!dλ can be analytically computed by repeated applying n times of integration-by-parts. By dropping the R k error and approximating the first k -1-th total derivatives with forward differential method, we can derive k-th order SDE solvers for diffusion DoS-SDEs. In fact, it is inaccurate to call it "order" when k ≥ 2, because the proposed algorithm has a global error of at least O(λ -λ s ) [14] . Thus, only when k = 1, it is referred to as a first-order solver with a strong convergence guarantee, as stated in [14] . Nevertheless, for practical convenience, we still refer to this approximation as k-th order. Here we present the expressions for first-order as well as second and third-order solvers. We name such solvers as DoS-SDE Solver overall, and DoS-SDE Solver-k for a specific order k.EQUATIONSubstituting into Eq.( 38), we obtain first-order solver for DoS-SDEsEQUATIONDoS-SDE Solver-2 When k = 2, the integral in Eq.( 40) becomesEQUATIONSubstituting into Eq.( 40), we obtain 2-th order solver for DoS-SDEsEQUATIONwhere x(1) θ (x s , x0 , s) can be estimated by forward differential method. We haveEQUATION)where time t < s < r and x θ (x r , x0 , r) represents the output of the network at the previous time step.DoS-SDE Solver-3 Samely, when k = 3, the integral in Eq.( 40) becomesEQUATIONSubstituting into Eq.( 40), we obtain 3-th order solver for DoS-SDEsEQUATIONθ (x s , x0 , s)EQUATIONwhere x(1) θ (x s , x0 , s) and x(2) θ (x s , x0 , s) can be estimated by forward differential method. We haveEQUATIONwhere time t < s < r and x θ (x r , x0 , r) represents the output of the network at the previous time step. AndEQUATION)where time t < s < r < q and x (1) θ (x s , x0 , s) and x(1) θ (x r , x0 , r) respectively represent the approximations of the first-order derivatives at the current and previous steps.Detailed algorithms for our solvers are proposed in Sec. BPrevious work has introduced a method called ResShift [53] , which shortens the length of the Markov chain in the diffusion process through residual shifting to achieve efficient super-resolution in diffusion models. In this section, we theoretically elaborate on the similarities and differences between our method and ResShift.ResShift expresses the forward diffusion process in the form of residual shifting, as shown in the following equation:EQUATIONwhere e 0 = y 0 -x 0 respensents the residual between the LR image y 0 and the HR image x 0 . Hence, it can be rewrite as,EQUATIONTherefore, essentially, the ResShift concept also represents a linear combination of the source domain and the target domain. However, the crucial factor lies in our design of the diffusion equation, which determines whether we can effectively utilize the diffusion prior. This is our most significant differentiating point. Currently, the mainstream approach to leveraging diffusion prior involves fine-tuning large-scale pretrained diffusion models to achieve SR. As we all know, Stable Diffusion, a typical representative of large-scale diffusion models, employs the diffusion equation of DDPM [16] .Its forward process can be expressed as:EQUATIONwhere α t , σ t ≥ 0 and α 2 t + σ 2 t = 1, referred to as noise schedule. If we consider η t y 0 + (1 -η t )x 0 as a whole, we can intuitively observe that Eq. ( 51) lacks a decay coefficient α t compared to Eq. ( 52). Therefore, applying Eq. ( 51) to fine-tune a pretrained Stable Diffusion model poses significant challenges. The equation proposed by our DoSSR is restated as follows:EQUATIONOur diffusion equation incorporates the noise schedule of Stable Diffusion, enabling seamless compatibility with existing DDPM-type diffusion models. Therefore, in this sense, our method is specifically designed as a diffusion equation tailored to adapt to Stable Diffusion.In fact, Eq. ( 51) and Eq. ( 52) (or Eq. ( 53)) belong to two different types of diffusion models, which correspond to the discretizations of two types of SDEs(VE SDE and VP SDE) [40] , respectively.The diffusion equation that satisfies the discretizations of VE SDEs typically takes the following form:EQUATIONwhere σ t increases with t, and σ T is typically a very large number, ensuring that q(x T ) = N (x 0 ; σ 2 T I) ≈ N (0; σ 2 T I). Therefore, Eq. ( 51) can be considered as a variant of the above equation. By setting hyperparameters such that k 2 η t = σ 2 t , it appears possible to fine-tune a diffusion model pretrained with Eq. ( 54) as the diffusion equation, thereby leveraging the diffusion prior. However, this contradicts its original intention, as the starting point of inference almost approximates Gaussian noise, retaining little of the prior information of LR. Our design of shifting sequence {η t } T t=1 is aimed at addressing this issue, which the ResShift lacks.The second type of diffusion equation that satisfies the discretizations of VP SDEs is of the DDPM type. In Eq. ( 52), the noise schedule satisfies the noise schedule 0 ≤ σ t ≤ 1 and it is typically set to σ T ≈ 1, α T ≈ 0 to ensure that q(x T ) = N (α T x 0 ; σ 2 T I) ≈ N (0; I). To my knowledge, most large-scale pretrained diffusion models, represented by Stable Diffusion, adopt the VP-type (DDPM-type) diffusion equation. Therefore, our design is highly significant. Moreover, our theory is not limited to the analysis of discrete cases but extends to more general continuous cases, expressed as SDEs. Based on this, we have developed our fast samplers for our DoSSR. These are the distinctions between our work and ResShift.To further demonstrate the effect of the diffusion prior, we conducted an additional comparative experiment with ResShift, as detailed in Appendix C.Here, algorithms for first, second, and third-order solvers for DoS-SDEs are presented as follows.Algorithm 1 DoSSR Solver-1.Require: starting point t 1 , used time steps {t i } N i=1 , nosie schedule α t and σ t , preprocessed LR image x0 , data prediction model x θ . 1:x t1 ← α t1 x0 + σ t1 ϵ ▷ initial value 2: for i ← 2 to N do 3: DoSG ← α ti (1 -η ti )( ηt i 1-ηt i - ηt i-1 1-ηt i-1 λ 2 t i λ 2 t i-1▷ domain shift guidanceLinear Term ←αt i (1-ηt i ) αt i-1 (1-ηt i-1 ) λ 2 t i λ 2 t i-1 x ti-1 5: Noise Term ← α ti (1 -η ti ) λ 2 ti - λ 4 t i λ 2 t i-1 x0 6: PAT ← α ti (1 -η ti )(1 - λ 2 t i λ 2 t i-1 )x θ (x ti-1 , x0 , t i-1 ) ▷ Prediction Approximation Term 7:x ti ← Linear Term + DoSG + PAT + Noise Term 8: end for 9: Return:x t N Algorithm 2 DoSSR Solver-2.Require: starting point t 1 , used time steps {t i } N i=1 , noise schedule α t and σ t , preprocessed LR image x0 , data prediction model x θ . 1:x t1 ← α t1 x0 + σ t1 ϵ ▷ initial value 2: Q ← N one 3: for i ← 2 to N do 4: DoSG ← α ti (1 -η ti )( ηt i 1-ηt i - ηt i-1 1-ηt i-1 λ 2 t i λ 2 t i-1▷ domain shift guidance 5:Linear Term ←EQUATIONif Q = N one then 8:PAT ← α ti (1 -η ti )(1 - λ 2 t i λ 2 t i-1 )x θ (x ti-1 , x0 , t i-1 ) 9:else 10:D i ← x θ (xt i-1 , x0,ti-1)-xθ(xt i-2 , x0,ti-2) ti-1-ti-2▷ first order derivative 11:PAT ← α ti (1 -η ti )(1 - λ 2 t i λ 2 t i-1 )x θ (x ti-1 , x0 , t i-1 ) -α ti (1 -η ti ) (λt i -λt i-1 ) 2 λt i-1 D i 12:end if 13:x ti ← Linear Term + DoSG + PAT + Noise Term 14: end for 15: Return: x t N Algorithm 3 DoSSR Solver-3.Require: starting point t 1 , used time steps {t i } N i=1 , noise schedule α t and σ t , preprocessed LR image x0 , data prediction model x θ . 1:x t1 ← α t1 x0 + σ t1 ϵ ▷ initial value 2: Q ← N one, Q d ← N one▷ first and second order derivatives 3: for i ← 2 to N do 4:DoSG ← α ti (1 -η ti )( ηt i 1-ηt i - ηt i-1 1-ηt i-1 λ 2 t i λ 2 t i-1▷ domain shift guidance 5:Linear Term ←αt i (1-ηt i ) αt i-1 (1-ηt i-1 ) λ 2 t i λ 2 t i-1 x ti-1 6: Noise Term ← α ti (1 -η ti ) λ 2 ti - λ 4 t i λ 2 t i-1 x0 7: if Q = N one and Q d = N one then 8: PAT ← α ti (1 -η ti )(1 - λ 2 t i λ 2 t i-1 )x θ (x ti-1 , x0 , t i-1 ) 9:else if Q ̸ = N one and Q d = N one then 10:D i ← x θ (xt i-1 , x0,ti-1)-xθ(xt i-2 , x0,ti-2) ti-1-ti-2▷ first order derivative 11:PAT ← α ti (1 -η ti )(1 - λ 2 t i λ 2 t i-1 )x θ (x ti-1 , x0 , t i-1 ) -α ti (1 -η ti ) (λt i -λt i-1 ) 2 λt i-1 D i 12:else 13:D i ← x θ (xt i-1 , x0,ti-1)-xθ(xt i-2 , x0,ti-2) ti-1-ti-2▷ first order derivative 14:EQUATION▷ second order derivative 15:PAT ← α ti (1 -η ti )(1 - λ 2 t i λ 2 t i-1 )x θ (x ti-1 , x0 , t i-1 ) -α ti (1 -η ti ) (λt i -λt i-1 ) 2 λt i-1 D i + α ti (1 -η ti ) (λt i-1 -3λt i )(λt i-1 -λt i ) 2 -λ 2 ti ln( λt i λt i-1 ) U i 16: end if 17:x ti ← Linear Term + DoSG + PAT + Noise Term 18: end for 19: Return: x t NTo demonstrate the impact of the diffusion prior, we conduct a comparative experiment with the ResShift model [53] , which does not leverage a pretrained diffusion model. Considering that ResShift is trained on ImageNet [11] while our DoSSR model is trained on commonly used datasets for super-resolution tasks (e.g., DIV2K [1] ), to eliminate the influence of the training dataset, we retrain our DoSSR model using ImageNet as well. To highlight the effect of the diffusion prior, we only utilize a subset of ImageNet as our training data. This subset consists of randomly selected 10 images from each of the 1000 categories in ImageNet, totaling 10,000 images, as illustrated in Table 4 . It can be observed that our DoSSR achieves superior metrics compared to ResShift despite utilizing significantly fewer training data and epochs for iteration. This indicates that leveraging the diffusion prior is highly beneficial for super-resolution tasks, even without utilizing commonly used high-resolution datasets for training, we can still achieve satisfactory results. We also perform qualitative comparisons between our retrained DoSSR and ResShift, as illustrated in Fig. 5 . It is evident that the utilization of the diffusion prior significantly enhances the quality of the generated images, both in terms of fidelity and realism.Aside from ResShift, we also compare our method with other plausible alternative formulations of diffusion processes, which encompass ColdDiff [3] and Rectified Flow [29] . ColdDiff is similar to DDPM, but differs by employing alternative degradation methods, such as blurring and masking, rather than additive Gaussian noise as used in DDPM. In our case for image super-resolusion, we select the blur degradation. Note that ColdDiff is originally applied in the pixel space, while we have to implement it in the latent space for fair comparison with our method. Rectified Flow defines the forward process as x t = t x0 + (1 -t)x 0 where x0 is the data sample and x 0 is Gaussian noise. In our case for image super-resolusion, we choose FlowIE [59] as an implementation of rectified flow which replaces x 0 with the LR image as the starting point.For a fair comparison, we reimplemented the three methods-ResShift [53] , ColdDiff [3] , and FlowIE [59] -using the same network architecture, training dataset, and initialization as our method.The results are shown in the Table 5 . Results of ColdDiff is significantly worse than other methods.The main reason can be summerized as two aspects. First, applying bluring kernels to latent features is not equivalent to applying them to raw images, while ColdDiff is originally designed for the latter.Second, the blurring kernel can only be designed in a hand-craft manner and may not represent the real-world degradation, so when tested with real LR images there would be a domain gap. The limilation of hand-crafted degradation is also well-known in many previous image restoration works. In contrast, other methods, including our DoSSR, does not assume a fixed degradation, so the performance is much better. Because FlowIE emphasizes single step evaluation, we follow its setting and compare DoSSR with 1-setp evaluation with it. It can be see that FlowIE is also not satisfactory and largely underperforms our DoSSR (-11.87% MUSIQ). This is mainly attributed to FlowIE's bad adaptation from the DDPM pretrained T2I model, since the learning objective of flow matching differs from score matching. By contrast, our design makes full use of the DDPM pretrained weights so performs much better. ResShift also significantly underperforms our method, similar to FlowIE, because its formulation does not account for adaptation from the pretrained diffusion prior, as discussed in Appendix A.7 and C.1. To summarize, our formulation differs from other alternatives especially in terms of better leverage and adaptation from DDPM pretrained T2I models.The network structure of our model is illustrated in Fig. 6 . In general, we adopt the same model architecture as in DiffBIR [27] , using LR images as conditional inputs for the ControlNet module. The ControlNet is initialized from the Stable Diffusion 2.1 Unet encoder and trained to generate HR images given LR inputs. Zoomed LR order=1, step=5 order=1, step=10 order=2, step=5 order=3, step=5 The abstract provides a concise summary of the main findings and contributions of the paper, while the introduction elaborates on the problem statement and research objectives, thereby clarifying the contributions. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: We have carefully reviewed the NeurIPS Code of Ethics and adhere to its principles.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).https://huggingface.co/stabilityai/stable-diffusion-2-1-basehttps://huggingface.co/datasets/Iceclear/StableSR-TestSetsWe use the repository available at https://github.com/chaofengc/IQA-PyTorch 


Decision: The rebuttal has well addressed the raised issues. All the reviewers recommend to accept this paper. The area chairs agree with this recommendation.

Reviews: review: paper_summary: paper_summary: This paper presents DoSSR, a novel framework for diffusion-based image super-resolution that aims to balance efficiency and performance. A domain shift equation that integrates with existing diffusion models, allowing inference to start from low-resolution images while leveraging pretrained diffusion priors. A continuous formulation of the domain shift process using stochastic differential equations (SDEs), enabling fast and customized solvers.
strength_weakness: strength_weakness: I have doubts about the interpolation between LR and HR. Theoretically, from the moment of interpolating, the image contains both the degradation of LR and the details of HR. For details, please refer to the article [Interpreting Super-Resolution Networks with Local Attribution Maps], which also mentions similar gradient and difference operations. If each image contains information about LR and HR, is it reasonable to use this for training? Based on the proposed method and the actual effect of LR&HR, I guess that using only LR or adding various noise/blur methods to LR may also achieve similar results. Since the topic of the paper is related to this approach, this concept is still important. In addition, skipping the first half of the diffusion model is a more intuitive method. I don’t know if there is a paper discussing this, but in practice this method seems to have appeared for some time (this will not affect my score much).
questions: questions: About the issue mentioned in Weaknesses, can we have some experiments to show the reason of doing such interpolation?
limitations: limitations: yes
paper_summary: paper_summary: This paper proposes a Domain Shift diffusion-based SR model, named DoSSR, that capitalizes on the generative powers of pretrained diffusion models while significantly enhancing efficiency by initiating the diffusion process with low-resolution (LR) images. Introducing the SDEs to describe the process of domain shift and provide an exact solution for the corresponding reverse-time SDEs. Designing customized fast samplers, resulting in even higher efficiency, thereby achieving the state-of-the-art efficiency performance trade-off.
strength_weakness: strength_weakness: 1.	In Table 1, the results of the proposed method on the reference index seem unsatisfactory, even much lower than other diffusion model methods.

2.	In colddiff [r1], Bansal et al proposed a method of reasoning from an arbitrary starting point. What is the difference between the method of reasoning from LR mentioned in this article and it? Such clarification would be useful.

3.	In order to facilitate observation, it is recommended to perform a partial zoom on Fig. 4 (b) for closer examination.

[r1] Bansal, Arpit, et al. "Cold diffusion: Inverting arbitrary image transforms without noise." Advances in Neural Information Processing Systems 36 (2024).
questions: questions: 1. As mentioned in Weakness, why do the resulting graphs look very beautiful, but the reference indicators are so far behind?

2. Explain the differences with DPM-solver, since ito-taylor expansionin also exists in your paper.
limitations: limitations: N/A
paper_summary: paper_summary: The existing diffusion-based image restoration models do not fully exploit the generative prior of the pre-trained diffusion models and produce high-quality images starting from Gaussian noise, leading to inefficiency. To address these issues, the paper a Domain Shift diffusion-based SR model (DoSSR) which can leverage the generative power of the pre-trained diffusion models and reconstruct super-resolved images based on the input corrupted images. It shows that the proposed method can effectively improve efficiency with few evaluation steps. Specifically, experiment results show that the proposed model achieves a slightly better performance on some benchmark datasets, including synthesized data and real-world data, in terms of various perceptual quality measurements.
strength_weakness: strength_weakness: 1. In implementation, this research adopts LAControlNet and SD 2.1-base as the pre-trained T2I model. However, the paper does not illustrate the overall structure of their proposed method, so it is hard to understand how to adopt the corrupted input as conditional information.

2. The novelty of the paper is limited. 1). Leveraging pre-trained large-scale diffusion models for image super-resolution under the framework of ControlNet has been investigated in many previous studies [1-4]. These methods can also produce high-quality restored images from the corrupted input. 2). The goal of the proposed domain shift SDE  is to design a diffusion bridge between the corrupted input images and the corresponding high-quality images, which is very similar to flow matching-based methods, such as rectified flow [5]. However, the paper does not discuss the flow-based models.

3. By leveraging flow-based models, several methods [6-7] have been proposed for image restoration.

4. The performance improvement on several datasets is limited. It is hard to evaluate the visual quality of images generated by the proposed method through visual comparison because no GT image is illustrated.


[1] Wang J, Yue Z, Zhou S, Chan KC, Loy CC. Exploiting diffusion prior for real-world image super-resolution. IJCV, 2024.

[2] Chen X, Tan J, Wang T, Zhang K, Luo W, Cao X. Towards real-world blind face restoration with generative diffusion prior. IEEE Transactions on Circuits and Systems for Video Technology. 2024 Apr 1.

[3] Yu F, Gu J, Li Z, Hu J, Kong X, Wang X, He J, Qiao Y, Dong C. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild. In CVPR 2024.

[4] Yang T, Ren P, Xie X, Zhang L. Pixel-aware stable diffusion for realistic image super-resolution and personalized stylization. arXiv preprint arXiv:2308.14469. 2023.

[5] Liu X, Gong C, Liu Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. ICLR, 2023.

[6] Liu J, Wang Q, Fan H, Wang Y, Tang Y, Qu L. Residual denoising diffusion models. CVPR, 2024.

[7] Zhu Y, Zhao W, Li A, Tang Y, Zhou J, Lu J. FlowIE: Efficient Image Enhancement via Rectified Flow. CVPR, 2024.
questions: questions: 1. It is better to illustrate the overall structure of the adopted models. For example, how to adopt the corrupted input as conditional information for the T2I model. 
2. The novelty of the paper is a significant concern. It is better to demonstrate the advances of the proposed method, compared with the previous studies.
3. The paper proposes to specifically design the drift term in SDE for image super-resolution, which leads to the reduction of evaluation steps. It easily causes confusion, because the additional drift term is to interpolate the corrupted input and the corresponding high-quality images. The sampling method is the key point. These two points should be clarified.
limitations: limitations: The paper does not discuss the limitations and social impact of the proposed method. It is hard to evaluate the model fairness, generalization ability, robustness, and transparency of the proposed method.
paper_summary: paper_summary: This paper proposed a Domain Shift diffusion-based SR model, namely DoSSR, which capitalizes on the generative powers of pretrained diffusion models while significantly enhancing efficiency by initiating the diffusion process with low-resolution (LR) images. In addition, authors advance their method by transitioning the discrete shift process to a continuous formulation to further enhance sampling efficiency. The proposed method is evaluated on synthetic and real-world datasets and achieved competitive results.
strength_weakness: strength_weakness: 1. The Domain Shift scheme seems to be similar to ResShift, where noise schedule is replaced with the solution in DDPM. Can the authors further explain the novelty of proposed Domain Shift scheme?
2. The experimental setting of the paper is insufficient. See questions for details.
questions: questions: 1. The method proposed in the paper is based on the RealESRGAN preprocessing and LAControlNet (from DiffBIR). Is the model trained based on the pre-trained model of DiffBIR? What about the results on the baseline1 "RealESRGAN preprocessing + DiffBIR" or baseline2 "RealESRGAN preprocessing + w/o Domian Shift Guidance(DoSG)"? Are there some experiments to verify the performance improvement of the proposed method compared to the baseline1 or baseline2? 
2. What are the advantages of the DoSSR Solvers proposed in the paper compared to existing accelerated sampling schemes (such as DDIM [1], Dpm-solver++ [2], EDM [3])? Are there any ablation studies to verify it? 

The relevant references are listed as follows, 

[1] Song, Jiaming, Chenlin Meng, and Stefano Ermon. "Denoising diffusion implicit models." arXiv preprint arXiv:2010.02502 (2020). \
[2] Lu, Cheng, et al. "Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models." arXiv preprint arXiv:2211.01095 (2022). \
[3] Karras, Tero, et al. "Elucidating the design space of diffusion-based generative models." Advances in neural information processing systems 35 (2022).
limitations: limitations: The authors fully explain the limitations and potential societal impact in this paper.

Comments: title: Response to the rebuttal
Follow-up Feedback
comment: I have read the author's response, as well as the comments and discussions with other reviewers. The author has partially addressed my concerns. I keep my init positive score.
Thanks for the quick feedback! We here further address the concern regarding the practical application value of the proposed DoS SDE-Sovler. This can be explained from the following two perspectives.

**1. Adapting Existing Solvers to the DoS Diffusion Model:**

While it is feasible to apply solvers like DDIM, EDM, and DPM Solver to our DoS diffusion model, these adaptations require some effort in re-derivation due to differences in the forward process. No additional training is necessary for this adaptation. For instance, the DDIM solver applied to the DoS diffusion model is equivalent to the first-order version of our DoS SDE-Solver, while the DPM Solver++ corresponds to the third-order version. Although we haven't shown an exact equivalence for the EDM solver due to differing noise and scaling schedules, it is analogous to the second-order version of our solver. In conclusion, adapting DDIM, EDM, and DPM-Solver for the DoS diffusion model is unnecessary, as our DoS SDE-Solver already provides a comprehensive solution.

As a summary, we do not need adapting DDIM, EDM and DPM-Solver for the DoS diffusion model, because the proposed DoS SDE-Solver already provides a unified solution. 

**2. Applying DoS SDE-Solver to other models and tasks:** It is also feasible to apply our solver to other models and tasks.

-  Different models:  if we apply it to a DDPM pretrained model (such as the Stable Diffusion series), we do need additional finetuning. However, we do not see this as a defect. Diffusion-based image SR methods ALWAYS necessitate finetuning for good performance. For example, in our baseline model, DiffBIR[1], the finetuning cost is over 384 A100 GPU hours(2~3 days on 8x A100), and in StableSR[2], it costs about 960 v100 GPU hours(5 days to one week on 8x V100 GPUs). In contrast, our finetuning cost is relatively low, at just ~192 v100 GPU hours (1 day on 8x V100), delivering state-of-the-art super-resolution quality and a 5x-7x speedup during inference.

- Different tasks:  The DoS SDE-Solver can also be extended to image-to-image translation tasks, including image restoration (e.g., deblurring, deraining) and style transfer. These tasks fit within the domain shift framework, requiring only paired image data to replace the LR-HR pair used in our super-resolution setting.

We hope this response addresses your concerns. Please feel free to reach out with any further questions.

***
References:

[1] Lin X, He J, Chen Z, et al. Diffbir: Towards blind image restoration with generative diffusion prior[J]. arXiv preprint arXiv:2308.15070, 2023.

[2] Wang, Jianyi, et al. "Exploiting diffusion prior for real-world image super-resolution." International Journal of Computer Vision (2024): 1-21.
thank you for your responses and comprehensive explanation. Your responses have addressed my concern and the final score has been raised to borderline accept.
Thank you for your respond. However, I still have some concerns about the real application value of the proposed DoS SDE-Solver. As the author said, "Our proposed sampler, DoS SDE-Solver, is specifically designed for the model under our forward-noise scheme framework" and "Existing samplers, such as DDIM, DPM-Solver, and EDM ...  cannot be adapted to our model". Therefore, I doubt whether DoS SDE-Solver can be easily extended to other models or tasks, and whether it will bring a large training cost.
The authors' rebuttal solves my concerns, I raise my rating to WA.