Taming Generative Diffusion Prior for Universal Blind Image Restoration

Authors: siwei  tu
weidong  yang
ben  fei
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 11:43:18

Abstract: Diffusion models have been widely utilized for image restoration. However, previous blind image restoration methods still need to assume the type of degradation model while leaving the parameters to be optimized, limiting their real-world applications. Therefore, we aim to tame generative diffusion prior for universal blind image restoration dubbed BIR-D, which utilizes an optimizable convolutional kernel to simulate the degradation model and dynamically update the parameters of the kernel in the diffusion steps, enabling it to achieve blind image restoration results even in various complex situations. Besides, based on mathematical reasoning, we have provided an empirical formula for the chosen of adaptive guidance scale, eliminating the need for a grid search for the optimal parameter. Experimentally, Our BIR-D has demonstrated superior practicality and versatility than off-the-shelf unsupervised methods across various tasks both on real-world and synthetic datasets, qualitatively and quantitatively. BIR-D is able to fulfill multi-guidance blind image restoration. Moreover, BIR-D can also restore images that undergo multiple and complicated degradations, demonstrating the practical applications. The code is available at https://github.com/Tusiwei/BIR-D.

Main paper: 
 introduction: 
 Images inevitably suffer from a degradation in quality in the process of capturing, storing, and compressing. Thus, the image restoration task intends to establish a mapping between the degraded image and the original image, to recover a high-quality image from the degraded image. In an ideal scenario, the ultimate goal is to undo and restore the degradation process of the image. However, in reality, the complexity of the degradation mode often leads to the incapability to fully restore the original high-quality image, which also makes traditional supervised approaches unsuitable for all types of image restoration tasks. According to the degradation mode, image restoration tasks can be divided into two types: non-blind and blind problems. Blind problems, such as low light enhancement, motion blur reduction and HDR image restoration, refer to image restoration problems where the degradation functions and parameters are totally unknown.The blind image restoration problem has attracted increasing attention with the development of generative models. The unsupervised blind image restoration methods represented by Generative Adversarial Networks (GANs) [1; 2; 3; 4] have the capability to train networks on large datasets of clean images and learn real-world knowledge. However, GANs are still difficult to avoid falling into limitations such as poor diversity and difficulty in model training. In parallel, diffusion model [5; 6; 7; 8; 9] have shown strong performance in terms of quality and diversity compared to GANs. Pioneer works such as GDP [10] , DDRM [11] , and DDNM [12] attempt to solve such problems by incorporating the degraded image y as guidance in the sampling process of diffusion models. By modeling posterior distributions in an unsupervised sampling manner, these approaches showcase the potential for practical guidance in blind image restoration, offering promising implications for real-world applications. However, the degradation types in these models still need to be assumed, limiting the practicality of natural image restoration where the complicated degradation models always remain unknown.To this end, we propose an effective and versatile Blind Image Restoration Diffusion Model (BIR-D). It utilizes well-trained DDPM [13] as an effective prior and is guided by degraded images to form a universal method for various image restoration and enhancement tasks. To uniformly model the unknown degradation function of blind image restoration, an optimizable convolutional kernel is dynamically optimized and utilized to simulate the degradation function at each denoising step. Specifically, BIR-D updates the convolution kernel parameters based on the gradient of distance loss between the generated image undergoing our optimizable convolutional kernel and the given degraded image. At the same time, all existing image restoration methods [10; 11; 12] that use diffusion models manually set the guidance scale as a hyperparameter to control the magnitude of guided generation, which also remains unchanged throughout the sampling process. However, for images from different tasks, the guidance scale required for each diffusion step is not entirely the same. To deal with this issue, we have derived an empirical formula for the guidance scale, which can calculate the optimal guidance scale for the next denoising step in real-time during the sampling process. This improvement avoids the need to manually grid search the optimal value of the guidance scale when solving different tasks and also enhances the quality of generated images. With the help of a well-trained DDPM, the above designs enable BIR-D to tackle various blind image restoration tasks. BIR-D can also achieve multi-degradation or multi-guidance image restoration. Furthermore, it showcases satisfactory results in addressing restoration issues related to complex degradation types encountered in real-world scenarios. 
bir-d: universal blind image restoration diffusion model: 
 In this study, we aim to use a well-trained DDPM [13] to learn the prior distribution of images and ultimately solve non-blind and blind problems in various image restoration tasks.For a natural image x, its corresponding degraded image y can be obtained by the degradation function y = D(x). Most of the blind image restoration methods [10; 12] are used to solve the situation where the degradation function D is known while leaving the parameters of D are unknown. However, when dealing with real-world image restoration problems, the degradation function D is not only an unknown quantity but also difficult to accurately represent mathematically. Therefore, we propose an optimized convolutional kernel to simulate complex degradation functions. The Moreover, in the real-world scenario, considering that there are different noises in different subtle areas of the image, using only one optimized convolutional kernel may not fully cover this situation. Therefore, we propose to utilize a mask M to model and estimate these noises. Thus, the entire degradation process can be represented as: y = K(x) + M, where K refers to the optimized convolutional kernel used in the model and M is a mask with the same dimension as image x. K and M have their own optimizable parameters, forming the degradation function mathcalD. In this way, any degradation process can be simulated by this degradation function.In the reverse denoising process of DDPM, the generated images can be conditioned on degraded image y [39] . Specifically, the distribution p θ (x t-1 |x t ) of reverse denoising is converted into Algorithm 1: Unconditional diffusion model with the guidance of degraded image y, given a diffusion model noise prediction function ϵ θ (x t , t).Input: Degraded image y, degradation function D composed of optimized convolutional kernels K with parameters φ and mask M with parameters ϕ, learning rate l, distant measure L. [13] that the difference between it and the original formula lies in the addition distribution of p(y|x t ), which serves as a probability representation for denoising x t into a high-quality image consistent with y. Previous work [10] proposed a feasible calculation to approximate this indicator by using heuristic algorithms:Output: Output image x0 conditioned on y. Sample xT from N (0, I) for t from T to 1 do x0 = x t √ ᾱt - √ 1-ᾱt ϵ θ (x t ,t) √ ᾱt L φ,ϕ,x 0 = L(y, D φ,ϕ (x0)) s = -(x t -µ) T g+C+log N L(D φ,ϕ ( x0 ),y) x0 ← x0 -s(1-ᾱt ) √ ᾱt-1 β t ∇x 0 L φ,ϕ,x 0 μt = √ ᾱt-1 β t 1-ᾱt x0 + √ ᾱt (1-ᾱt-1 ) 1-ᾱt xt βt = 1-ᾱt-1 1-ᾱt βt Sample xt-1 from N (μt, βtI) φ ← φ -l∇φL φ,ϕ,x 0 ϕ ← ϕ -l∇ ϕ L φ,ϕ,x 0 return x0EQUATIONwhere N is the normalization factor, which is the distribution p θ (y|x t+1 ), and s is the scalar factor used to control the importance of guidance, named guidance scale. L is the distance metric. The value of the guidance scale plays a crucial role in the quality of the image generation result. A larger value can lead to overall blurring of the image, while a smaller value can result in missing details in the restoration. However, the guidance scale in existing works [10; 9; 12] can only be manually set as a hyperparameter. But in specific experiments, the optimal value of the guidance scale varies in different masks, degraded images, and diffusion steps. The original configuration necessitates thorough testing for the initial setup. Additionally, employing the same guidance scale for every denoising step is not an optimal choice.Therefore, we propose an empirical formula for the guidance scale, which can dynamically calculate and update the optimal values of guidance factors in real-time at each diffusion step of degraded images in specific repair tasks. Specifically, we noticed that the distribution log p θ (y|x t ) can be applied to perform Taylor expansion around x = µ and take the first two terms. The detailed process of proving can be found in Appendix D.EQUATIONwhereg = ∇ xt log p θ (y | x t ) | xt=µ , C = log p(y | x t ) | xt=µ .By combining the heuristic approximation formula and Taylor expansion formula mentioned above, we can simplify the empirical formula for the guidance scale:EQUATION)The guidance scale is related to the generated images x, degraded image y, and the degradation function D. This value of this Adaptive Guidance Scale can be dynamically updated in each diffusion step so that each step in the diffusion model can use the most appropriate guidance scale.Through empirical formulas, we can obtain the conditional transition formula in the reverse process of the diffusion model.EQUATIONwhere z conforms to the distribution N (z; µ θ (x t , t) + Σg, Σ). The intermediate quantity g = ∇ xt log p(y|x t ). The value of g can be obtained by calculating the gradient in heuristic algorithms in eq. ( 1), which includes the parameter of guidance scale:EQUATIONThe other terms N 1 , N 2 , and the variance of the reverse process Σ = Σ θ (x t ) in eq. ( 4) and eq. ( 5) are constants, and the unconditional distribution p θ (x t-1 |x t ) is given by traditional diffusion models.Therefore, the conditional transition distribution p(x t-1 |x t , y) can be approximately estimated by adding -(sΣ∇ xt L(D(x t ), y)) to the mean of the traditional unconditional transition distribution.Previous studies [10] have shown that the addition of Σ has a negative impact on the quality of generated images. Therefore, in this experiment, we omitted the term Σ, and the complete sampling process is shown in algorithm 1.Detailly, in the diffusion step t of the sampling process, the noise of x t is first predicted from the given pre-trained DDPM and eliminated to obtain an estimated value of x 0 . Subsequently, apply the degradation function of step t to x 0 and calculate its reconstruction loss with the degraded image y. We utilize our adaptive guidance scale for sampling the next step latent x t-1 . In this process, it is necessary to calculate the gradient about x 0 and the parameters of each convolution kernel in the distance metric loss, which is used to update the convolution kernel parameters in real time for the next sampling process.Pre-process. The empirical formula for the guidance scale we construct is related to the degradation function. Herein, when the model simulates the degradation function more reasonably, BIR-D can obtain more appropriate guidance scale values accordingly. To this end, we introduce a first-stage pre-training model from [17] to further enhance the model's capability to correct initial deviations. This enables the model to have a strong correction ability for significant deviations in the degradation function during the initial diffusion step, ultimately generating ideal image restoration results. Table 3 : Quantitative comparison among various zero-shot learning methods of low-light enhancement task on LOL [22] and VE-LOL-L [23] Bold font represents the best metric result. Multi-degradation Image Restoration. In the real world, the degradation process often involves a combination of multiple different complex types. To improve the image restoration capability of the model in complex situations and enhance its practicality, we propose to extend BIR-D into multi-task scenarios. To our surprise, BIR-D can fulfill multi-degradation image restoration without any modification (Figure 9 ) thanks to the mixture of degradation types can also be simulated as an unknown degradation by an optimizable convolutional kernel.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The discussion on limitations in the paper has been placed in the appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
experiments: 
 In this section, we systematically compare BIR-D with other blind image restoration methods in real-world and synthetic datasets. We have attached some more specific details, such as the dataset, implementation, evaluation, and other results in the Appendix.Blind Image Restoration on Real-world Datasets. Firstly, we evaluate the blind image restoration capability of BIR-D on two real-world datasets, namely LFW dataset [14] and WIDER dataset [15] . As shown in Figure 3 , BIR-D successfully simulated and removed blur, and achieved more ideal facial detail restoration. The quantitative results in Table 1 shows that BIR-D outperforms PGDiff [16] and DiffBIR [17] in NIQE metric on both datasets and FID metric on WIDER, demonstrating better blind image restoration performance.Comparison on Common Linear Inverse Problems. We conducted experiments on linear inverse problems on ImageNet 1k to compare BIR-D with off-the-shelf methods. For each experiment, we calculated the average Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), Consistency, and FID results, where PSNR, SSIM, and Consistency are used to quantify the faithfulness between the generated image and the original image, while FID is used to measure the quality of the generated image. To make fair comparisons, other methods are given known degradation functions as reported in the original paper while BIR-D utilizes universal degradation functions for different tasks. Table 2 shows that BIR-D outperforms other methods in terms of Consistency and FID in almost all tasks. As shown in Figure 5 , the images generated by BIR-D demonstrate a high level of image quality and details. Moreover, Figure 4 also demonstrates that BIR-D can generate various results in image restoration tasks.Low Light Enhancement. We further evaluated the effectiveness of BIR-D in low-light image enhancement. Following the previous works [10] , we utilized three datasets, LOL [22] , VE-LOL-L [23] , and LoLi-Phone [24] , to test the restoration ability of BIR-D. As shown in Table 3 , our BIR-D outperforms all the zero-shot methods in both FID and Lightness Order Error (LOE) [40] , and demonstrates significant improvement in Perceptual Index (PI) [41] . A lower PI value reflects better perceptual quality, while a lower LOE reflects a better natural preservation ability of the generated image, making images to have a more natural sensory experience. As shown in Figure 6 and the Appendix, BIR-D exhibits reasonable and well-exposed results.Figure 8 : Comparison of image quality for motion blur reduction results on GoPro [30] and HIDE dataset [31] . HDR Image Recovery. In the HDR image restoration task, we compared BIR-D with other leading methods, including DeepHDR [33] , AHDRNet [35] , HDR-GAN [37] , and GDP [10] , on the NTIRE2021 Multi-Frame HDR Challenge [29] dataset. The quantitative and qualitative results are presented in Table 4 and Figure 7 , with BIR-D showing the best PSNR and SSIM levels, and successfully generating results with rich and accurate detailed information.Motion Blur Reduction. To evaluate the performance of BIR-D in the motion blur reduction tasks, we compare BIR-D with the state-of-the-art motion blur reduction methods on GoPro dataset [30] and HIDE dataset [31] . We used the same input image, which also means that the motion blur of the input image is the same, ensuring fairness in comparison. The comparison results of the metrics are presented in Table 4 , where BIR-D outperforms existing methods in both PSNR and SSIM. As shown in Figure 8 , BIR-D can effectively achieve the elimination of motion blur. The generated images not only achieve a better quality but also receive restoration with more clear details.Multi-Degradation Image Restoration. Encouraged by the excellent restoration performance of BIR-D on single restoration task, we further tested the image restoration performance of BIR-D in solving multi-task image restoration. As shown in Figure 9 , we take a degraded image on the ImageNet dataset where two types of degradation are mixed as an example. The optimizable convolution kernel of BIR-D can also simulate these complicated degradation functions. The generated images obtained have excellent results in both image quality and details.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
ablation study: 
 The Effectiveness of Optimizable Convolutional Kernel and Adaptive Guidance Scale. The ablation studies on the real-time optimizable convolutional kernel parameters and guidance scale were performed to reveal the effectiveness of these settings. We further tested the LOL [22] and the most challenging LoLi-phone [24] datasets. Model A fixed the convolutional kernel parameters and guidance scale. Models B and C represent fixed parameters for the convolution kernel and the fixed guidance scale, respectively. As illustrated in Figure 10 , the fixed guidance scale with a bias set at s = 80000 resulted in the emergence of mineral textures in the images. By contrast, as shown in Table 5 , BIR-D outperformed other models in all indicators, demonstrating the effectiveness of an optimizable convolutional kernel and adaptive guidance scale.The Effectiveness of the First Stage Pre-training Model. We conducted further experiments on the deblur task to demonstrate the impact of the first-state pre-training model. As shown in Table 6 , for a randomly initialized convolution kernel parameter, all metrics of BIR-D were better than BIR-D without the pre-training model. These results indicate that the first-stage pre-trained model is able to provide better initial state of images for our BIR-D. 
parameter analysis: 
 The Parameter Variations of the Optimizable Convolution Kernel and Mask in the Reverse Steps. In order to visualize the variation trends of the parameters of convolution kernel mask in the reverse process, we conducted experiments on the test set of the LOL dataset from the low-light enhancement task. As shown in Figure 11 (a), the mean values of the convolution kernel parameters and degradation mask are given by random initialization and gradually increase with the progress of the time steps. This increase in magnitude is influenced by the gradient of the distance metric with respect to the corresponding parameters. When the sampling step t<500, the difference between x0 and y changes slightly, resulting in correspondingly smaller gradient values. 12 , during the sampling process, the degradation mask learns the detailed information of the image, including local regions with significant brightness differences. This process is obtained by updating the gradient of the distance metric with respect to the degradation mask parameters.The Theoretical Analysis of the Changing Trend of Guidance Scale in the Reverse Steps. We take the variation in the guidance scale of BIR-D on the LOL dataset as an example to analyze the trend of its changes during the reverse steps. As shown in Figure 11 (c), the guidance scale gradually decreases with the sampling step, which aligns with the actual situation. When the sampling step t<500, as t decreases, the difference between x t and x t-1 decreases with decreasing t, indicating a reduction in the simulated noise at each step. Therefore, the level of guidance required for each sampling step should also be reduced accordingly, leading to a decrease in the required guidance scale values. According to Equation (3), when step t is small, the gradient term g also decreases due to the small change in x t at each step. The speed of the gradient term decreases is greater than the speed of distance metric decreases, resulting in a decrease in the value of the guidance scale.Figure 12 : The changing of degradation mask during the sampling process in HDR recovery. 
conclusion: 
 In this paper, we propose Blind Image Restoration Diffusion, which is a unified model that can be used to solve various blind image restoration problems. We utilize optimized convolutional kernels to simulate and update the degradation function in the diffusion step in real time, and derive the empirical formula of the guidance scale in detail, so that it can better utilize the unconditional diffusion model to generate high-quality images. The ability to solve various blind image restoration tasks, including low-light enhancement and motion blur reduction, has also been verified through various indicators of datasets.In this appendix, we provide a more detailed derivation process for adaptive guidance scale and more image restoration results. Appendix A provides more blind image restoration results for images undergoing various multi-degradation modes. Appendix B is a preliminary for the diffusion model. Appendix C introduces the related work, including an introduction to diffusion-based image restoration models and existing leading methods in the field of blind image restoration. Appendix D is the complete derivation process of the empirical setting of the adaptive guidance scale. Appendix E proposes the algorithmic process of BIR-D for accomplishing multi-guidance blind image restoration. Appendix F conducts further ablation experiments on the size and other parameters of the convolution kernel and provides the optimal convolution kernel parameter settings for different image restoration tasks. Appendix G presents and analyzes the variations of the mask parameters in the reverse steps of LOL dataset. Appendix H presents the results of blind image restoration tasks, including blind face restoration, low-light enhancement, motion blur reduction, and HDR image restoration. Appendix I provides more results on the ImageNet dataset for four basic tasks, including super-resolution, colorization, deblurring and inpainting, as well as corresponding experimental details and parameter settings. Appendix J discusses limitations and future works for BIR-D. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [No] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The original papers used in the article have been cited and annotated with their sources.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The assets introduced in the paper are all submitted after being documented.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [N/A] Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [N/A] Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this section, we attempted more complex degradation scenarios to test the image restoration ability of BIR-D. The input images are derived from adding different degradation types, which include 4 × super-resolution, colorization, deblurring, and inpainting. As shown in Figure 13 , BIR-D achieved satisfactory restoration results in four different multi-task image restoration tasks, which indicates the image restoration capability of BIR-D under complex degraded scenarios.Diffusion models consists of the forward and reverse processes. The forward process continuously adds noise to a natural image x 0 through T diffusion steps to obtain the noise distribution x T ∼ N (0, I), where N represents the Gaussian distribution. The reverse process aims to simulate the noise in each diffusion step and eliminate it, ultimately obtaining the restored generated image x 0 .The forward process is a Markov chain defined by the following equation:EQUATIONIt corrupts the initial data x 0 into distribution x T that is close to Gaussian noise after T steps of diffusion, with each sample process defined by q(x t |x t-1 ) = N (x t ; √ 1 -β t x t-1 , β t I), where β t is the variance of a forward process. The variance can be set as a constant or learned by reparameterization. Simultaneously defining α t = 1 -β t ,ᾱ t = t i=1 α i . It has been proven by [42] that through mathematical reasoning, x t at any diffusion step can be directly calculated from the starting x 0 :EQUATIONwhere ϵ ∼ N (0, I). When T is large enough, √ ᾱt approaches 0, and at this point q(x T |x 0 ) is closer to the latent distribution of x T .The reverse process is also a Markov chain, which gradually denoises a standard multivariate Gaussian distribution into a denoised image x 0 . Firstly, sample x t ∼ N (0, I). The conditional distribution of the reverse process is p θ (x t-1 |x t ) = N (x t-1 ; µ θ (x t , t), Σ θ I). According to the Bayesian formula, it can be transformed as follows:EQUATIONExpand and simplify the three terms at the right end of the equation. The variance Σ θ of the reverse process can be obtained as a fixed value. Note that [43] indicates that it can also be learned parameters.And the mean of the reverse process µ θ related to x t and x0 : According to the formula of the forward process, x0 can be predicted by x t , where ϵ is a noise function approximator obtained by a neural network θ.EQUATIONEQUATIONSubstitute it into Equation (10) to obtain the mean value µ θ :EQUATIONC Related Work On top of GANs and other relevant methods, DDPMs are more studied for this task due to the enhanced diversity. For instance, both DiffBIR [17] and GDP [10] leverage generative diffusion priors for blind image restoration. BlindDPS [53] introduces parallel diffusion models for solving blind inverse problems when the functional forms are known. PromptIR [54] uses prompts to encode degradation-specific information and dynamically guide the recovery of the network. Nevertheless, these methods are still limited to specific tasks. BIR-D can be regarded as a unified solver for multiple restoration tasks by simultaneously estimating the recovered images and specific degradation models.In the reverse process of the diffusion model, we added guidance from y to transform the original reverse denoising distribution p θ (x t | x t+1 ) into a conditional distribution p θ (x t | x t+1 , y). This distribution can be further simplified:EQUATIONEQUATIONEQUATIONEQUATIONEQUATIONIn this formula, distribution p θ (y | x t+1 ) is independent of x t , so we use the constant N instead:EQUATIONTherefore, compared to the original diffusion model, the conditional reverse process requires approximation of p θ (y | x t ). We used a heuristic approximation method:EQUATIONWhere L is image distance metric, which represents the MSE loss in this experiment. K is the constant in the above formula, which serves as a normalization factor here. And s is a guidance scale, which is used to control the magnitude of guidance. Take the logarithm of both sides of the equation:EQUATIONWhen the diffusion step approaches infinity, ∥Σ∥ → 0, so we can assume that distribution log p θ (y | x t ) has low curvature compared to Σ -1 . We can perform Taylor expansion on distribution log p θ (y | x t ) around x = µ and take the first two terms:EQUATIONWhereg = ∇ xt log p θ (y | x t ) | xt=µ , C = log p(y | x t ) | xt=µ .By combining the heuristic approximation formula and Taylor expansion formula mentioned above, we can simplify the empirical formula for the guidance scale:EQUATIONThe empirical formula is shown below. For each image at every moment t, the applicable value of the guidance scale can be calculated.EQUATIONBecause here y is a loss image without noise, while x t itself has noise. The use of MSE errors between x t and y can lead to the introduction of noise into the guidance process. Therefore, we are using the MSE error between the estimated value of x0 and y here, and the above formula needs to be corrected as:EQUATIONBIR-D is capable of accepting multiple input images to incorporate multi-guidance during the reverse steps. Taking HDR image restoration task as an example, BIR-D receives three images as inputs separately. As shown in Figure 14 and Algorithm 2, BIR-D uses three degradation functions for three input images. In each sampling step, after obtaining x0 , x0 is respectively substituted into three degradation function at diffusion step t. The parameters of convolution kernels and masks are updated by measuring the gradient of its parameters with the distance metric. The average of three distance metrics are used as the overall loss to update the mean and variance used during sampling. The empirical formula of adaptive guidance scale is also based on this loss.F The Optimal Size of Optimizable Convolutional Kernel.In the main paper, in order to assure the versatility of BIR-D, we used convolution kernels of size 7 × 7 for all tasks. Nevertheless, for different types of tasks, the size of the convolution kernel might Input: Degraded image set y i |i = 1, 2, . . . , n . For each image y i in the set, there is a corresponding degradation function D i composed of optimized convolutional kernels K i with parameters φ i and mask M i with parameters ϕ i , learning rate l, distant measure L i . Output: Output image x0 conditioned on set y i |i = 1, 2, . . . , n . Sample xT from N (0, I) be different. To explore the impact of kernel size on the quality of generated images, we conducted experiments using convolution kernels of different sizes in various types of image restoration tasks.for t from T to 1 do x0 = x t √ ᾱt - √ 1-ᾱt ϵ θ (x t ,t) √ ᾱt for i from 1 to n do L φ i ,ϕ i ,x 0 = L(y i , D φ i ,ϕ i (x0)) φ i ← φ i -l∇ φ i L φ i ,ϕ i ,x 0 ϕ i ← ϕ i -l∇ ϕ i L φ i ,ϕ i ,x 0 L φ,ϕ,x 0 = n i=1 L φ i ,ϕ i ,x 0 s = -(x t -µ) T g+C+log N L φ,ϕ,x 0 x0 ← x0 -s(1-ᾱt ) √ ᾱt-1 β t ∇x 0 L φ,ϕ,x 0 μt = √ ᾱt-1 β t 1-ᾱt x0 + √ ᾱt (1-ᾱt-1 ) 1-ᾱt xt βt = 1-ᾱt-1 1-ᾱt βt Sample xt-1 from N (μt,As shown in Table 7 , for blind image restoration tasks, the experiment showed that the results of a 5×5 convolution kernel perform best. For linear inverse tasks (Table 8 ), the optimal convolution kernel size was 9×9.In order to visualize the variation of the parameters of convolution kernel mask in the reverse process, we conducted experiments on the test set of the LOL dataset from the low-light enhancement task. As shown in Figure 15 , the mask has successfully captured certain intricate details within the low-light images, thereby facilitating the restoration of brightness during the reverse step.In this section, we present more generated results for several types of image restoration tasks in blind issues, including low-light enhancement, motion blur reduction, and HDR image restoration.For the blind face restoration task, we randomly selected 1000 images from the LFW [14] and WIDER [15] test sets as input samples. More image results are shown in Figure 16 . BIR-D effectively eliminates the blurring in degraded images, resulting in clearer and more detailed facial images.For the low-light enhancement task, we use datasets including LOL [22] and VE-LOL-L [23] . The image results are shown in Figure 17 . BIR-D achieved excellent results on different datasets, with the brightness of the generated images very close to the ground truth without losing any details during the restoration process.For the motion blur reduction task, the datasets used include GoPro dataset [30] and HIDE dataset [31] . Figure 18 shows the results of BIR-D in eliminating motion blur. This also demonstrates the capabilities of BIR-D for unknown degradation functions. In this section, we will provide more image restoration results on commonly used tasks, including super-resolution, colorization, deblurring, and inpainting tasks. Noted that although BIR-D has the capability to perform image restoration when the diffusion function is unknown, in order to ensure a fair comparison with other methods, an initially set and known degradation function will be used, and all image generation results and comparison metrics will be obtained under this condition. The images used in the test are all from the ImageNet dataset, and the degradation method and parameters of the input images are the same as the state-of-the-art methods, including DGP [18] , SNIPS [20] , DDRM [11] , GDP [10] , which ensures the effectiveness and fairness of the comparison.As shown in Figure 20 , BIR-D effectively removes the blur in the input image and effectively restores the details in the image. For the inpainting task, we set 25% of pixels in images to have missing pixel values, and use this as a degraded image to test the image restoration ability of BIR-D. Figure 21 shows that BIR-D has the ability to restore these 25% of missing pixels, resulting in an overall output image that is closer to the ground truth. Meanwhile, we set the ratio to 4 in the super-resolution task. As shown in Figure 22 , low-resolution images can be restored from BIR-D to high-resolution images without losing clarity, while preserving various subtle details of natural images. Besides, for a given grayscale image, Figure 23 shows the level of color restoration by BIR-D for the image, and the output results also indicate the ability of BIR-D to solve the colorization task. Since we utilize a single pre-trained unconditional diffusion model provided by [13] , the generation speed will increase when dealing with image size increasing due to the patch-based solution. In the future, it is a very promising direction to use stable diffusion to achieve faster speed in large-size image restoration. Justification: The abstract provides an overview of the main content of the paper and the introduction summarizes three contributions of the paper, which have been experimentally proven to have ideal effects.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.37th Conference on Neural Information Processing Systems (NeurIPS 2023). 


Decision: This paper received an Accept, a Borderline Accept and a Reject. The reviewers appreciate the technical contributions of the paper and its novelty. The main concerns raised by the reviewers are: 1) Missing references, 2) Missing analysis and presentation of the method, 3) Technical issues with the method and experiments. The authors submitted a rebuttal and engaged in a discussion with the reviewers. The AC has read the rebuttal, the reviews, and the discussions. Two reviewers were satisfied with the rebuttal and discussion and raised their rating. The AC finds that the authors adequately clarified all the concerns raised by the reviewers and recommends the paper for acceptance.

Reviews: review: paper_summary: paper_summary: The paper introduces BIR-D, a novel approach utilizing generative diffusion models for blind image restoration without requiring predefined degradation types. Traditional methods assume degradation models and optimize their parameters, limiting their applicability. BIR-D overcomes this by employing an optimizable convolutional kernel that simulates degradation dynamically during diffusion steps, allowing it to handle various complex degradations.
strength_weakness: strength_weakness: The reviewer appreciates the innovative use of an optimizable convolutional kernel to dynamically adapt the degradation model during the diffusion steps. This is considered the most significant contribution of the work. However, this section lacks sufficient analysis and visualization. While the paper asserts that GDP [1] assumes specific degradation types and is not suitable for complex degradation models, the differences and improvements of the proposed degradation model compared to the one in GDP are not clearly explained.

Additionally, the paper is missing some relevant references for blind IR [2] and earlier generative prior-based IR methods [3,4]. Overall, the reviewer appreciates the work and would be happy to adjust the rating if the aforementioned concerns are addressed.

[1] Generative diffusion prior for unified image restoration and enhancement. CVPR'23

[2] AND: Adversarial neural degradation for learning blind image super-resolution. NeurIPS'23

[3] Image restoration with deep generative models. ICASSP'18

[4] Maximum a posteriori on a submanifold: a general image restoration method with gan. IJCNN'20
questions: questions: See weaknesses.
limitations: limitations: The authors have addressed the limitations.
paper_summary: paper_summary: This research introduces BIR-D, a novel approach to the universal challenge of blind image restoration. It leverages an adaptable convolutional kernel designed to emulate the degradation model, with the capability to refine its parameters progressively during the diffusion process. Furthermore, the work presents an empirical formula to guide the selection of the adaptive scale, a critical component in enhancing restoration accuracy. Extensive experiments substantiate the method's exceptional performance across a spectrum of restoration tasks, showcasing its robustness and efficacy.
strength_weakness: strength_weakness: 1. The contribution in question, which utilizes an optimizable convolutional kernel to simulate the degradation model and dynamically update the parameters of the kernel during the diffusion steps, may be perceived as lacking in novelty. You should provide a detailed comparison with the referenced [10], "Generative Diffusion Prior for Unified Image Restoration and Enhancement," about the different strategy for updating the degradation model.
2. The paper's exploration of the 'optimizable convolutional kernel' and 'adaptive guidance scale' could be enhanced by including an analysis of convergence trends or parameter behavior over time. Such analyses would clarify how these elements contribute to the method's performance.
3. The notations in this paper may lead to misunderstandings. Specifically, in Formula (6), the representation of $g$ lacks the subscript $xt=μ$, which is critical for clarity. Furthermore, the $N$ in Formula (19) should be distinguished from the $N$ used in Formula (20) to avoid ambiguity. Additionally, on line 418, the symbol $K$ should be replaced with $N$ for consistency.
4. The paper's explanation is not sufficiently clear, such as how BIR-D can accomplish multi-guidance blind image restoration.
questions: questions: 1. Upon my review, the formula for calculating the guidance scale $s$ in equation (3), once combined and simplified with equation (1), yields an identity. This suggests that the information obtained from the current $xt$ sampling is independent of the update to $s$. Could you clarify this issue?
limitations: limitations: None
paper_summary: paper_summary: This paper proposes a blind image restoration method by using a pre-trained diffusion model without additional prior knowledge. The proposed adaptive guidance scale is fancy which uses the loss function to judge its value while the degradation function design is confusing. The results on real-world benchmarks show great performance success, but the experiment to validate their proposed modules is fragile.
strength_weakness: strength_weakness: 1. The introduction of the method section is confusing, first, what is the “$\sum$” in Line 112 and Line 421? Second, the design of the degradation function D does not make sense, why does adding the M term estimate the noises and what does the noise mean here (Line 72)? 
2. The proposed method is unreliable, the author said that they do not have additional prior information, but in my view, the usage of the pre-trained model (Diff-BIR) is the prior knowledge, this paper is just an extra refiner to refine a coarse-clean image to a better one. To validate the proposed method, do the extra ablation study on all the benchmarks without pre-trained model and show the quantitative and qualitative results w, w/o it.
3. The ablation study is fragile. i) Visualize the result of the degradation function D for degrading the clean image. ii) Visualize what M learned for different tasks. iii) Show the value changing of guidance scales during the denoising stage and propose the theoretical analysis of the changing trend.
questions: questions: See the weakness.
limitations: limitations: Yes.

Comments: title: Looking forward to discussion
Looking forward to discussion
Official Comment by Authors
comment: Dear reviewer rhDr,

We sincerely thank you for your valuable time and feedback. We hope our existing rebuttal and official comments could address your previous concerns well. As the discussion phase is nearing its end, we remain open to addressing any remaining questions or concerns. If you have any further questions during the next discussion period, please let us know, and we will be happy to answer them. We look forward to receiving your feedback. Thank you once again!

Sincerely,

Best regards,

The Authors
Dear Reviewer CUUT:

We sincerely appreciate your helpful and constructive review and are pleased to see your decision to raise your score. Based on your valuable suggestions, we will provide detailed explanations of the differences between BIR-D and GDP in the future version to highlight the strengths and improvements of BIR-D. Meanwhile, the parameter trends of the kernel and mask will be incorporated into our future version. We will also integrate the pipeline of multi-guidance BIR-D in the future version to make our multi-guidance method clearer. To further support our paper, we will carefully release our code. Thank you once again for your recognition of our work and the valuable time you have invested in this review.

Best regards,

The Authors
Most of the concerns have been addressed in the authors' response. I will raise the score.
Dear Reviewer rhDr:

We sincerely appreciate your time and effort in reviewing our paper. We would like to clarify the following issues you mentioned.

1. The setting of mask $\mathcal{M}$ is mainly used to solve the image restoration task of local regions with significant differences in brightness, as using an optimizable convolution kernel alone may not be able to effectively solve the brightness correction task of such regions. As shown in Global PDF Figures 4 and 5, a mask with the same dimension as the degraded image can learn the brightness and detail information of each local region of the image, which also assists the optimizable convolution kernel in simulating the degradation function.

2. The first stage pre-training model is only used to improve model performance in the two tasks of deblurring and motion blur reduction. Without the first stage pre-training model, our BIR-D can still achieve image restoration of deblurring and motion blur reduction tasks (see Table 6 in the main text and Figure 15 of Appendix G). It is worth noting that this first stage pre-training model is not used in the other blind image restoration tasks since other tasks can be effectively modeled by our devised optimizable convolution kernel. The universal capacity of our BIR-D comes from the design of an optimizable convolution kernel, which can effectively simulate any degradation models of most blind image restoration tasks. The experiment we conducted also proved this contribution. Here is a summary of the experiment we conducted.


| Category       | Task                        | Dataset               | Figure    | Table |
|----------------|-----------------------------|-----------------------|-----------|-------|
|                | Deblurring                  |                       | 1,5(b),15 | 2     |
|                | Colorization                |                       | 1,4,18    | 2     |
| Linear Inverse | Super-resolution            | ImageNet 1k           | 1,5(a),17 | 2     |
|                | Inpainting                  |                       | 1,5(c),16 | 2     |
|                | Multi-task                  |                       | 1,9,10    | -     |
|----------------|-----------------------------|-----------------------|-----------|-------|
|                | BIR in Real-world Dataset   | LFW,Wider             | 1,3,11    | 1     |
|                | Low-light Enhancement       | LOL,VE-LOL,LoLi-Phone | 1,6,12    | 3     |
| Non-linear     | Motion Blur Reduction       | Gopro,HIDE            | 1,8,13    | 4     |
|                | HDR Image Recovery          | NTIRE 2021            | 1,7,14    | 4     |
|                | Realistic Image Restoration | Website               | 1         | -     |


___

We hope these clarifications will enhance your comprehension of our paper. If you have any further comments, please do not hesitate to mention it. We look forward to further communicating with you.

Best wishes,

The Authors
The author only addresses partial of my concern. 
For the noise mask, the visualization is hard to understand and the explanation is not convincing. 
For pretrained weight, as you are doing the setting of universal, how can you add it to some of the tasks? This makes me question the veracity of the author's experiment in UNIVERSAL。
All in all, the original paper lacks too many experiments and I do not think it can be refined directly. 
I will keep my rating.
Dear Reviewer rhDr:

We sincerely thank you for taking the time to this review and providing valuable comments.

___

Based on the reviewers' comments, we have made revisions to our manuscript to include the following changes.

* We provide the trends of parameters in the adaptive guidance scale and optimizable convolution kernel in the sampling process to clarify how these designs contribute to the performance of BIR-D.
* We clarify that we only use the first stage pre-training model for better initialization, rather than the pre-training diffusion model of DiffBIR.  And we analyze the improvement of BIR-D using this first-stage pre-training model.
* We have supplemented more details on the visualization, description and explanation of some symbols the reviewer mentioned in the review.

___

We hope our explanations have addressed your concerns. As we are in the discussion phase, we welcome any additional comments or questions regarding our response or the main paper. If further clarification is needed, please do not hesitate to mention it, and we will promptly address your inquiries. We look forward to receiving your feedback.

Best wishes,

The Authors
Dear Reviewer CUUT:

We sincerely thank you for devoting time to this review and providing valuable comments.

___

Based on the reviewers' comments, we have made revisions to our manuscript in the following areas.

* We have supplemented the trends of parameters in the adaptive guidance scale and optimizable convolution kernel in the sampling process to better clarify how these designs contribute to the BIR-D's performance.
* We have listed and analyzed the advantages and improvements of BIR-D compared to GDP from various perspectives. Importantly, we have re-clarified that the challenges previously associated with GDP are effectively addressed by BIR-D.
* We have provided more details on multi-guidance blind image restoration, including diagram and pseudocode in Global PDF.
* We have clarified the necessity of proposing empirical formulas with guidance scales and optimized the derivation approach and process to make it clearer to the readers. 

___


We hope our explanations have addressed your concerns. As we are in the discussion phase, we welcome any additional comments or questions regarding our response or the main paper. If further clarification is needed, please do not hesitate to mention it, and we will promptly address your inquiries. We look forward to receiving your feedback.

Best regards,

The Authors
We sincerely appreciate your thought-provoking reviews and are pleased to see your upgrading decision. Following your valuable suggestions, we will carefully incorporate these revisions into our future version. To substantiate our results, we will release the code. Thank you once again for your positive rating and the time devoted to this review.

Best regards,

The Authors
Thank you for providing the additional details and clarifications in your response. I believes that the visualizations provided in the rebuttal file could help potential readers better understand the paper's contributions. The comparison to the GDP method also addresses my concerns. Therefore, I have increased my rating by one point.