Diffusion Priors for Variational Likelihood Estimation and Image Denoising

Authors: jun  cheng
shan  tan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 16:04:16

Abstract: Real-world noise removal is crucial in low-level computer vision. Due to the remarkable generation capabilities of diffusion models, recent attention has shifted towards leveraging diffusion priors for image restoration tasks. However, existing diffusion priors-based methods either consider simple noise types or rely on approximate posterior estimation, limiting their effectiveness in addressing structured and signal-dependent noise commonly found in real-world images. In this paper, we build upon diffusion priors and propose adaptive likelihood estimation and MAP inference during the reverse diffusion process to tackle real-world noise. We introduce an independent, non-identically distributed likelihood combined with the noise precision (inverse variance) prior and dynamically infer the precision posterior using variational Bayes during the generation process. Meanwhile, we rectify the estimated noise variance through local Gaussian convolution. The final denoised image is obtained by propagating intermediate MAP solutions that balance the updated likelihood and diffusion prior. Additionally, we explore the local diffusion prior inherent in low-resolution diffusion models, enabling direct handling of high-resolution noisy images. Extensive experiments and analyses on diverse real-world datasets demonstrate the effectiveness of our method. Code is available at https://github.com/HUST-Tan/DiffusionVI.

Main paper: 
 introduction: 
 Real-world imaging modalities, such as photography and biomedical imaging, frequently encounter complex image noise that is both signal-dependent and spatially correlated [23, 35] . Removing such noise is critical for subsequent image analysis and understanding. Existing deep learning-based image denoising methods rely on either large amounts of paired images for supervised training [50, 48, 49, 25] or noisy images for self-supervised training [31, 23, 46, 33] . However, collecting massive amounts of data is expensive and time-consuming in real-world scenarios. Therefore, designing effective and data-efficient real-world denoising methods is of significant practical importance.Incorporating image priors and the likelihood, and conducting the corresponding posterior inference (e.g., maximum-a-posteriori (MAP) estimation and mean estimation), is a classical and data-efficient approach to image restoration [26, 18] . Nowadays, deep generative models such as VAE, GAN, and Normalizing-flow have shown the capacity to capture and model complex image statistics, surpassing traditional analytical image priors [32] . Recent diffusion models have demonstrated state-of-the-art image generation capabilities [14, 37] and have been incorporated into various image restoration tasks as powerful image priors [19, 12, 5] .The essence of applying diffusion priors to address image denoising lies in accurately integrating the degraded image into the generation process of pre-trained diffusion models, underscoring the importance of the likelihood function. Unlike Gaussian white noise, real-world image noise is intricate and challenging to model precisely and analytically [17, 21] . Many existing methods based on diffusion priors solely account for i.i.d. Gaussian noise [19, 5, 43, 10] , making them ineffective for real-world noise removal. On the other hand, some approaches targeting complex and non-linear degradations employ hard data-consistency strategies and approximate posterior inference during the generation process [12, 45, 8] , yielding unsatisfactory real-world denoising outcomes due to their coarse likelihood modeling. Although a structured and heteroscedastic Gaussian likelihood function can well approximate real-world noise, such a model is computationally expensive due to the large covariance matrix and also impractical due to the unknown noise variance.To tackle these challenges, this paper integrates variational Bayes and presents adaptive likelihood estimation and MAP inference during the generation process of diffusion priors to handle real-world noise. We introduce an independent, non-identically distributed (i.ni.d.) likelihood combined with a precision prior to model real-world noise. Such a choice allows modeling the spatially variant feature of noise and meanwhile avoids modeling covariance, trading off the accuracy for practical feasibility. Based on variational Bayes, the i.ni.d. precision posterior at each step in the reverse process is subsequently inferred, which adaptively refines the likelihood function and aligns with the real-world noise model. Additionally, we introduce local Gaussian convolution to rectify the estimated noise variance, compensating for the lack of spatial correlation in the i.ni.d. likelihood function. By adaptively updating the likelihood at each reverse diffusion step, the final denoised result is achieved by progressively propagating the intermediate MAP solutions that strike the best balance between the noisy image and diffusion prior. Furthermore, real-world images exhibit diverse resolutions, often differing from those of pre-trained diffusion models. Existing methods utilize patch-based [12] or resize-based [19] operations, which are laborious and may impact low-level details. We observe that diffusion models pre-trained with lowresolution (LR) images tend to yield local diffusion priors effective for image restoration, enabling the direct treatment of high-resolution (HR) noisy images. The main contributions of this paper are summarized as follows:• We propose adaptive likelihood estimation and MAP inference based on diffusion priors and variational Bayes to address real-world complex noise.• We explore the local prior exhibited by diffusion models pre-trained with LR images.• Our method outperforms other unsurpervised denoising methods as well as diffusion priorsbased methods on diverse real-world image denoising datasets. 
related works: 
 Deep Learning-based Image Denoising. By harnessing modern deep architectures and large-scale paired training datasets, supervised learning-based denoising methods such as VDN [48] , Restormer [49] , and GRL [25] have significantly enhanced in-distribution denoising performance. However, their reliance on extensive paired data poses challenges for real-world applications, prompting the exploration of self-supervised denoising approaches. These include Blind spot (BS)-based methods (e.g., SSDN [22] , Noise2Self [3] , Nei2Nei [16] , and B2U [44] ), resampling-based methods (e.g., Nr2N [29] and R2R [34] ), and regularization-based methods (e.g., Noise2Score [20] and Stein [38] ). However, these methods assume spatially independent or analytical noise, which deviates from the structured and complex real-world noise. Recent advancements, such as AP-BSN [23] and LG-BPN [46] , have integrated Pixel-shuffle and masked convolution into BS networks to address real-world noise. Other approaches have leveraged disentangled representation learning [11, 31, 6] . Nonetheless, these methods often require large quantities of noisy images and lack data efficiency. Several single image-based deep learning methods (e.g., DIP [42] , Self2Self [36] , ZS-N2N [27] , and ScoreDVI [7] ) have been proposed, but their performance in real-world denoising scenarios remains suboptimal, underscoring the need for more effective approaches.Diffusion Priors for Image Restoration. Diffusion models have exhibited remarkable image generation capabilities [14, 40, 37] and have been integrated into inverse problems as diffusion priors to address various image restoration tasks in an unsupervised manner [24] . Existing methods based on diffusion priors can be broadly categorized into two aspects: linear inverse problem-solving based on exact degradation models (e.g., DDRM [19] , DDNM [43] , MCGDiff [5] , and FPS [10] ), and nonlinear inverse problem-solving based on approximate posterior inference (e.g., DPS [8] , GDP [12] , and DR2 [45] ). In the former, these methods typically assume analytical noise, such as additive white Gaussian noise, overlooking real-world noise that is signal-dependent and spatially correlated. Given the difficulty in precisely modeling and estimating real-world noise, these methods often prove ineffective in handling such noise.In the latter, hard data-consistency terms are introduced to replace accurate likelihood modeling, and approximate posterior inference is conducted during the inverse diffusion process to address complex degradations. However, due to the absence of explicit constraints from likelihood functions, these methods heavily rely on proper hyperparameters (e.g., guidance scales in GDP [12] , step size in DPS [8] , or downsampling factors in DR2 [45] ), leading to significant reconstruction errors. Diverging from these methods, we refrain from specifying the accurate noise model but introduce the noise precision prior and dynamically estimate its posterior using variational Bayes in the reverse diffusion process, enabling adaptive estimation of likelihood functions and better posterior inference.3 MethodsQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discussed the limitation of our method in the main paper Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminary: 
 The Diffusion model is a class of generative models used to model the distribution q(x 0 ). Its forward process is a Markov chain with fixed Gaussian transition and length T , which gradually corrupts the data x 0 by adding Gaussian noise according to a pre-defined variance scheduleEQUATIONA nice property of the forward process is that sample x t at any step t can be obtained from x 0 in a closed-form manner:EQUATIONwhere a t = 1 -η t , āt = t s=1 a s , ϵ ∼ N (0, I). In the reverse process, the diffusion model progressively recovers data from noise distribution p(x T ), which is again a Markov chain with learned Gaussian transition:EQUATIONwhere σ 2 t is a constant relating to η t and can be pre-computed. And µ θ (x t , t) is usually parameterized by a DNN ϵ θ (x t , t):EQUATIONConsider the image formation process y 0 = f (x 0 , n) in the real-world scenario, where n is the raw noise, f is the transformation function, y 0 ∈ R N and x 0 ∈ R N (N is the total pixel number) are the observed noisy image and original clean image, respectively. Noise in y 0 generally exhibits signal-dependent and spatially-correlated characteristics, e.g., sRGB noise [23] , due to the Poisson nature of photons and compound transformation in f . Suppose there are no non-linear parts in f , the image formation process can be simplified as y 0 = x 0 + n 0 (x 0 ) with corr(n i 0 , n j 0 ) > 0, where n 0 is the image noise related to the signal x 0 ; corr(i, j) represents the correlation coefficient between two neighboring elements i and j.Real-world image denoising task is to recover clean and high-quality x 0 from noisy y 0 , which turns into solving the posterior p(x 0 |y 0 ) in Bayesian statistics. As the pre-trained diffusion model possesses superior image priors, it is natural to inject the observed information y 0 into its reverse diffusion (or generation) process defined in Eq. (3) to achieve the conditional inference of x 0:T given y 0 , i.e., p(x 0:T |y 0 ). Due to that y 0 and intermediate x t exhibit large distribution gaps (as they have different noise types and strength) and it is also difficult to directly model p(y 0 |x t ) [8] , we hence follow [39, 45] and re-corrupt y 0 using Eq. ( 2) in each step t to obtain y t , which serves as intermediate conditions. Such choice results in the target conditional distribution:EQUATIONwhere y t = √ āt y 0 + √ 1 -āt ϵ, and p(x T |y T ) ≈ p(x T ) as x T and y T are approximated independent normal distributions.Structured and heteroscedastic Gaussian likelihood model. The right-hand side of Eq. ( 5) explicitly involves the prior p(x t-1 |x t ) and likelihood p(y t-1 |x t-1 ) at each step t. The prior p(x t-1 |x t ) has been available from the pre-trained diffusion model in Eq. ( 3), necessitating the modeling of p(y t-1 |x t-1 ). As discussed above, in principle we can assume that y 0 follows the structured and heteroscedastic Gaussian distribution N (x 0 , Σ(x 0 )), where Σ is the non-diagonal covariance matrix with variances related to signal x 0 , which allows modeling the signal-dependent and spatially-correlated properties of real-world noise. As a result, we derive thaty t-1 = α t-1 (x 0 + Aϵ 2 ) + 1 -α t-1 ϵ = x t-1 + α t-1 Aϵ 2 , AA T = Σ(x 0 ), ϵ 2 ∼ N (0, I) (6) which indicates that p(y t-1 |x t-1 ) = N (y t-1 ; x t-1 , Σ(x t-1 )) with Σ(x t-1 ) = α t-1 Σ(x 0 ). See detailed derivation in the Appendix A.1.As the prior p(x t-1 |x t ) in Eq. ( 3) and likelihood p(y t-1 |x t-1 ) in Eq. ( 6) both have Gaussian forms, the posterior distribution in Eq. ( 5) is theoretically computable. Nevertheless, we note that the above formulation presents several practical challenges. First, specifying an accurate Σ(x 0 ) for y 0 is difficult, which involves the estimation of noise variance and the noise correlation between neighboring pixels. These estimations are hard to achieve based on a single y 0 and are open research problems [17, 21] . In addition, the posterior inference with non-diagonal covariance matrix Σ is both memory-demanding and computationally expensive. These challenges prevent the direct application of the structured heteroscedastic Gaussian likelihood.To deal with these difficulties, we consider p(y 0 |x 0 ) = N (x 0 , diag(ϕ 0 ) -1 ), which has diagonal precision matrix diag(ϕ 0 ) (i.e., the inverse of the covariance matrix, ϕ 0 ∈ R N ). Such diagonal Gaussian likelihood allows modeling the spatially variant feature of real-world noise but ignores the noise correlation at this stage. Based on Eq. ( 6), p(y t-1 |x t-1 ) then becomes:EQUATIONHyperprior for precision ϕ t . Instead of specifying an accurate ϕ 0 , which is again difficult, we introduce the independent Gamma hyperprior p(ϕ 0 ) = N i=1 Gamma(ϕ i 0 ; α, β) for ϕ 0 (α and β are scalars), which serves as the rough precision prior for noise in y 0 . Meanwhile, based on Eq. ( 7), it is straightforward that ϕ t-1 followsEQUATIONBecause p(ϕ t-1 ) merely provides initial gauss about the noise precision (also variance) at each step t, we then expect to find the corresponding precision posterior p(ϕ t-1 |x t-1 , y t-1 ), which is more accurate and allows a better likelihood function p(y t-1 |x t-1 ). As posterior ϕ t-1 depends on x t-1 , we have to simuteniously infer them together, i.e., the following joint distribution:EQUATIONwhere γ ≤ 1 is the temperature parameter, which is typically utilized in variational inference to scale the likelihood function [2] .Variational inference of precision posterior. As p(x t-1 , ϕ t-1 |x t , y t-1 ) in Eq. ( 9) is a non-trivial distribution, we hence choose a trivial and factorized variational distribution g(x t-1 , ϕ t-1 ) = g(x t-1 )g(ϕ t-1 ) to approximate the true posterior p(x t-1 , ϕ t-1 |x t , y t-1 ), under the KL-divergence distance. Following the mean-field variational Bayes presented in [4] , the optimal g(x t-1 )g(ϕ t-1 ) can be solved by cycling through x t-1 and ϕ t-1 and replacing each in turn with a revised estimate of g(x t-1 ) and g(ϕ t-1 ). Specifically, we derive the following alternate update scheme for finding the optimal g(ϕ t-1 ):1. Update g(x t-1 ). Given g(ϕ t-1 ), the optimal g * (x t-1 ) is provided byEQUATIONwhich corresponds toEQUATIONwith mean μt-1 and variance σ2 t-1 asEQUATIONwhere ⊙ denotes element-wise multiplication; E(ϕ t-1 ) at step T is initialized as a constant (which is robust to different initializations) and then is updated as E(ϕ t-1 ) = αt-1 / βt-1 (see Update 2).2. Update g(ϕ t-1 ). Similar to g(x t-1 ), the optimal g * (ϕ t-1 ) given g(x t-1 ) isEQUATIONwith shape αt-1 and rate βt-1 asEQUATIONThe detailed derivation of Eqs. ( 11) and ( 13) is given in the Appendix A.2.MAP estimation with updated likelihood. During the updates, αt-1 and βt-1 in g(ϕ t-1 ) are adptively updated and become signal-dependent as indicated by Eq. ( 14). Once the cycling converges, we can obtain the approximated posterior distribution g(ϕ t-1 ) and the updated likelihood p(y t-1 |x t-1 ) = E ϕt-1∼g(ϕt-1) p(y t-1 |x t-1 , ϕ t-1 ) at step t. As a result, by maximizing the conditional distribution in Eq. ( 5) at step t, the optimal x * t-1 that balances the image prior and the observed y t-1 can be obtained as follows:EQUATIONwhere E(ϕ t-1 ) = αt-1 / βt-1 is the expectation of noise precision posterior (and hence 1/E(ϕ t-1 ) = βt-1 /α t-1 is the estimated noise variance at step t), and πt ∈ [0, 1] suggests that the optimal x * t-1 is the convex combination of y t-1 and µ θ (x t , t). Note that, in the second row of Eq. ( 15), we utilize the following Jensen's InequalityEQUATION) and employ the lower bound of log p(y t-1 |x t-1 ). Optimizing this lower bound generally produces satisfactory solutions, like in variational inference, VAE, and diffusion models.Rectification of 1/E(ϕ t-1 ). By considering the diagonal Gaussian likelihood, the noise correlation between neighboring pixels in y 0 (and y t ) is ignored, which affects the estimation of the precisionInput: Pre-trained diffusion model, noisy observation y 0 , hyperparameters α, β, temperature γ1: x T ∼ N (0, I), E(ϕ T ) = ⃗ 1 2: for t = T, • • • , 1 do 3:Compute µ θ (x t , t) based on Eq. ( 4); Compute y t-1 based on Eq. (7) 4:Set μold t-1 = ⃗ 0, μt-1 = µ θ (x t , t) 5:while∥μ old t-1 -μt-1 ∥ 2 2 ≥ 1e -6 do 6:Update g(x t-1 ) = N (μ t-1 , σ2 t-1 ) using Eq. ( 12)7:Update g(ϕ t-1 ) = N i=1 Gamma(α i t-1 , βi t-1 ) using Eq. ( 14)8:end while 9:Solve optimal x t-1 using Eq. ( 15) or Eq. ( 17 ). This is apparent as transformations in f (e.g., demosaicking in the ISP pipeline) will cause the local correlation of noise variance (or precision) while elements in E(ϕ t-1 ) are updated independently as shown in Eq. ( 14). Motivated by the analysis of spatial correlation of real-world noise in [23] , we introduce a local 2D convolution with a normalized Gaussian kernel G(l, s) (l is kernel size and s is the scale) to manually rectify 1/E(ϕ t-1 ). That is,EQUATIONBy progressively calculating x * t-1 at each step t based on Eqs. ( 15) and ( 17), the final denoised result x 0 can be obtained. The whole denoising algorithm is presented in Alg. 1.The common practice of sampling images from pre-trained diffusion models is to maintain the sampling resolution identical to the training resolution, which produces the best generation performance. In image restoration tasks, however, the resolution of the observed noisy image generally mismatches that of the pre-trained diffusion model. Existing approaches typically use patches [12] or resize operations [19] to address this issue, which is cumbersome and may affect local details.We observe that when a diffusion model trained with the U-Net architecture on low-resolution (LR) images is employed to sample high-resolution (HR) images, it exhibits local properties. Fig. 1 shows sampled 512 × 512 and 256 × 256 images from pre-trained 256 × 2562 and 128 × 1283 diffusion models, respectively, and it is clear that the generated textures mainly focus on local areas. As HR images contain more redundant information, denoising HR images is simpler than denoising their LR counterparts under the same noise level [41] . When the LR diffusion model is used to generate HR samples, there is only a short time window for it to decide the structures of the sampled images [15] , thus the generation tends to be local. Similar to traditional TV priors and Markov random fields that focus on designing local image statistics, we note that the local property of LR diffusion models is also effective for image denoising. This allows us to directly adopt the pre-trained LR diffusion prior to denoise HR noisy images without additional operations. Compared methods. We consider several representative single image-based unsupervised denoising methods, including DIP [42] , Self2Self [36] , PD-denoising [52] , ZS-N2N [27] , and ScoreDVI [7] . We also compare against AP-BSN [23], a self-supervised denoising method that can be trained on noisy images of test sets directly. In addition, several diffusion priors-based image restoration methods, including DDRM [19] , GDP [12] , and DR2 [45] , are chosen to denoise real-world images. For these compared methods, we utilize their official source code and report the corresponding performance.Particularly, these diffusion-based methods employ the same diffusion prior 2 as ours. As DDRM can only handle i.i.d. Gaussian noise and requires the noise std σ ddrm , we hence set σ ddrm = β/α, i.e., the prior noise std in our method. GDP introduced the hard data-fidelity term ŝ∥x 0 -y 0 ∥ 2 2 , where x0 is the estimated denoised result at each reverse diffusion step, and ŝ is the guidance scale. We tune different ŝ for different datasets. DR2 first obtained the intermediate x t-1 by adopting a low-pass filter Φ D and setting x t-1 = Φ D y t-1 + (1 -Φ D )x t-1 and then conducted inference from step τ + 0.25T to τ . We set D = 8 (Downsampling factor) and τ = 100 for DR2. We evaluate the quantitative denoising quality using PSNR and SSIM metrics.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
main results: 
 We present quantitative comparisons of different methods in Table 2 and visual comparisons in Figs. 2, 3 (and Figs. 5, 6 , 7 in the Appendix A.4). Overall, our method achieves the best quantitative (on average) and qualitative performance across all compared methods. First, while Self2Self excels on PolyU and CC datasets, it shows poor denoising capacity (both PSNR/SSIM values and visual results) on SIDD and FMDD that contain severe image noise, as shown in Fig. 5 . PD-denoising is generally effective but often introduces small artifacts that hinder visual effects, as seen in Fig. 5 . ZS-N2N struggles with real-world noisy images and typically leaves noticeable noise after denoising due to its reliance on the independent noise assumption. Regarding diffusion priors-based methods, GDP and GR2 perform poorly on real-world denoising, introducing artifacts (see Figs 5a and 7 ) and over-smoothing (see Fig. 3 ), possibly because they adopt hard data-consistency methods, which significantly deviate from the true likelihood function. As DDRM assumes Gaussian white noise, the denoised images often retain some noise, which cannot be entirely removed, as shown in Figs 3 and 5b . ScoreDVI is the most competitive method against ours, but it sometimes blurs images and loses local textures, as indicated by Fig. 2 . Unlike these methods, our approach effectively removes severe noise while preserving image details and textures. Although APBSN achieves the best PSNR values in the SIDD dataset, it frequently introduces noticeable color artifacts (see Figs. 2, 3 ) and oversmooths images (see Fig. 7 ). In addition, when applied to FMDD, PolyU, and CC datasets that contain fewer noisy images, its denoising performance significantly degrades and underperforms our method. This highlights the advantage of our dataefficient approach.Adaptive likelihood estimation (ALE). We analyze 1/E(ϕ 0 ) = β0 /α 0 , the estimated noise variance, and present quantitative and qualitative results in Fig. 4 . Fig. 4a demonstrates that β0 /α 0 effectively reflects the noise variance of y 0 . That is, β0 /α 0 exhibits larger values in noisier areas of y 0 and smaller values (black) in less noisy areas. Fig. 4b implies that the average of β0 /α 0 are inversely correlated with PSNR values of denoised images, which is reasonable since noisier images are more challenging to denoise and hence have lower PSNR. Conversely, the prior noise variance β/α = 3e -3 , indicated by the black plot in Fig. 4b are constant for all noisy images. These analyses suggest that the ALE captures the signal-dependent features of real-world noise effectively.In addition, we skip the variational inference process and directly use the prior precision p(ϕ t-1 ) to derivex * t-1 , resulting in x * t-1 = π t-1 y t-1 + (1 -π t-1 )µ θ (x t , t), π t-1 = σ 2 t α/(σ 2 t α + β). The corresponding denoising result (i.e., without ALE) is reported in the second row of Table 3 and is largely behind the denoising result of using ALE, further verifying the effectiveness of ALE.Rectification in Eq. (17) . By introducing the local Gaussian convolution operation, we explicitly refine the estimated noise variance 1/E(ϕ t-1 ). As shown in the fourth row of Table 3 , using 1/E(ϕ t-1 ) consistently improves the quantitative performance. where the 256 × 256 diffusion prior for HR images is more effective than its 512 × 512 counterpart. This is likely because training HR diffusion models (e.g., ≥ 512) is more challenging, resulting in inferior generation performance (e.g., FID) compared to LR diffusion models [9] . Consequently, the local prior inherent in medium-resolution diffusion models is superior for restoring HR images.Although our method is designed to handle real-world noise, it can also address other non-Gaussian noises, including Poisson noise and multiplicative Bernoulli noise. As these synthetic noises are spatially independent, we do not utilize Eq. ( 17) in our method. We report the denoising performance in Table 7 , with ZS-N2N [16] selected for comparison. Experimental details and visual results are given in the Appendix A.3.2 and A.5. Our method achieves better quantitative metrics than ZS-N2N on Poisson denoising and also preserves more local details and textures (see Fig. 8 ). While ZS-N2N shows better SSIM than ours on Bernoulli denoising, it causes intensity shifts (see Fig. 9 ) and thus has poorer PSNR.In addition to denoising, our method is readily available for image restoration with pixel-wise degradation, e.g., image demosaicing. To adapt our method to this task, we define the forward process y 0 = M ⊙ x 0 , where M is the degradation operator, and denotes element-wise multiplication. For demosaicing, M is the binary mask with 0 values indicating missing pixels of y 0 . We can incorporate M into p(y t-1 |x t-1 , ϕ t-1 ) in Eq. ( 15), which results in πt-1 =M σ 2 t M σ 2 t +1/E(ϕt-1). In Table 8 , we compare our method against DDRM on image demosaicing (CFA pattern: RGGB), and our method shows better results. Limitation. Our method builds on the DDPM sampling with a total of 1000 diffusion steps. Denoising a single noisy image with a resolution of 256 × 256 on an Nvidia 2080Ti GPU takes approximately 230 seconds, which is inefficient. In comparison, ZS-N2N takes about 16 seconds, despite its inferior denoising performance. Naively reducing diffusion steps in our method leads to apparent performance decreases, as indicated in Table 9 . Our next move is to incorporate advanced accelerated sampling strategies into our method to reduce inference time while maintaining performance. 
conclusion: 
 In this paper, we built upon diffusion priors and variational Bayes and proposed adaptive likelihood estimation and MAP inference during the reverse diffusion process, to handle real-world image noise that is structured and signal-dependent. The employed i.ni.d. likelihood function, combined with the precision prior and variational Bayes, allowed for the dynamical update of i.ni.d. noise precision posterior in each step of the generation process. This strategy adaptively refined the likelihood function and enabled the better MAP inference. Our method achieved excellent denoising performance on diverse real-world image denoising datasets and was also effective for removing other non-Gaussian synthetic noises. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Image denoising is a basic image processing task.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cited the paper that provided pre-trained diffusion models and we also included the corresponding URL.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: A.1 Derivation of Eq. ( 6)As analyzed in Section 3.2, in principle we can model the likelihood function of real-world noisy images as p(y 0 |x 0 ) = N (x 0 , Σ(x 0 )), where Σ is a non-diagonal covariance matrix and its variance is related to its mean x 0 (or signal). In order to incorporate y 0 into the inverse diffusion process and shorten its gap to x t-1 , we can construct y t-1 based Eq. ( 2) to obtainy t-1 = √ ᾱt-1 y 0 + √ 1 -ᾱt-1 ϵ 2 .For y 0 , it can be sampled from the multi-variate Gaussian p(y 0 |x 0 ) = N (x 0 , Σ(x 0 )), i.e., y 0 = x 0 + Aϵ, where AA T = Σ(x 0 ), and A is obtained by Cholesky decomposition. Finally, we obtainEQUATIONA.2 Derivations of Eqs. (11) and (13) Regarding Eq. ( 10), we haveEQUATIONwhere µ t = µ θ (x t , t). We observe that Eq. ( 19) has the summation and quadratic form of x i t-1 , and hence g(x t-1 ) is identified as a diagonal Gaussian distribution. By completing the square, we can obtainEQUATIONSimilarly, given g(x t-1 ), the optimal g * (ϕ t-1 ) is provided byEQUATIONwhich corresponds toEQUATIONFrom Eq. ( 22), we identify each g(ϕ i t-1 ) is a independent gamma distribution and hence g(ϕ t-1 ) isEQUATIONWe note that [9] only provides the 256 × 256 unconditional diffusion model ϵ θ (x t , t) trained on ImageNet, and the remaining diffusion models are class-conditional, i.e., ϵ θ (x t , c, t) with the class label c. There are two ways to utilize these class-conditional diffusion priors for image denoising.Regarding each noisy image, we first sample a class label c from randint(0, 1000) and input it to the ϵ θ (x t , c, t) combined with x t and t. One way is then ignoring the guidance from the pre-trained classifier during the generation process and directly computing µ θ (x t , t) based on ϵ θ (x t , c, t) and Eq. ( 4). The other way is further updating µ θ (x t , t) to µ θ (x t , t) + g s σ 2 t ∇ xt log p ψ (c|x t ), where p ψ (c|x t ) is the classifier and g s is the guidance scale. Basically, we found these two ways resulted in similar denoising performance, and Table 6 used the first way.Synthesis of noisy images. Regarding Bernoulli noise, we obtain the noisy image by y 0 = x 0 ⊙ M, M = torch.bernoulli(torch.ones_like(x 0 ) * p), p = 0.2; Regarding Poisson noise, we obtain the noisy image by y 0 = torch.poisson(λ * x 0 )/λ, λ = 30.Hyperparameters. For our method, we set β = 1e -2 and β = 8e -3 for Bernoulli noise and Poisson noise removal, respectively. The remaining hyperparameters are identical to those of our main experiments. Justification: see the abstract and introduction in the main text Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.https://openaipublic.blob.core.windows.net/diffusion/jul-2021/256x256_diffusion_uncond.pthttps://openaipublic.blob.core.windows.net/diffusion/jul-2021/128x128_diffusion.pthttps://openaipublic.blob.core.windows.net/diffusion/jul-2021/512x512_diffusion.pt 


Decision: This paper presents a novel method for real-world image denoising using diffusion priors combined with adaptive likelihood estimation and Maximum A Posteriori (MAP) inference. The method effectively integrates variational Bayes to dynamically infer the precision posterior, leading to a more accurate modeling of real-world noise (Reviewers eDuR, sawX, and Wzuv). The paper is well-written, with clear derivations and thorough experimental evaluation across multiple real-world datasets, showcasing superior performance compared to other unsupervised denoising methods (Reviewers UCEq, sawX, Wzuv).

After a careful consideration of the reviewers' comments, AC recommends accepting this paper. Despite some concerns regarding the justification of certain design choices and the computational cost, the paper presents a good contribution to the field of image denoising. The combination of diffusion priors with adaptive likelihood estimation is both novel and effective, addressing a critical problem in real-world imaging.

Reviews: review: paper_summary: paper_summary: The paper proposes a method to tackle real world noise using an adaptive likelihood estimation . For this the authors develop a technique to dynamically infer the precision posterior using variational bayes. The authors perform comprehensive evaluation on real world denoising datasets to show the effectiveness of the method.
strength_weakness: strength_weakness: 1. The authors have utilized a gamma based hyperprior without reasoning the design choice or comparing with other possible prior distribution. In order to show the need for a gamma prior, further experimental analysis are required
2. The methods seems to be applicable only for real world denoising limiting its practical utility.
3. Analysis with key diffusion based restoration methods like [1,2] are missing
    [1] Diffusion Posterior Sampling for General Noisy Inverse Problems
    [2] Manifold Preserving Guided Diffusion
4. [2] achieves good results with just 50 steps of sampling.Could the authors provide an analysis of the variation of performance with different number of sampling steps.
questions: questions: 1. Can the authors provide an intuition of utilizing a gamma hyperprior over a gaussian prior like in [1,2]. 
[1] Diffusion Posterior Sampling for General Noisy Inverse Problems
[2] Manifold Preserving Guided Diffusion

2. Is the method generalizable to other linear inverse problems than denoising [1,2] and DDRM, are all generic methods providing a general formulation to all linear inverse problems.

3. Could the authors provide an analysis of the variation of performance with different number of sampling steps.
limitations: limitations: Yes, The authors have presented the limitations in a separate section.
paper_summary: paper_summary: Overall, this paper presents a novel method in real-world image denoising by proposing a sophisticated method that combines adaptive likelihood estimation, MAP inference, and variational Bayes within the diffusion model framework.
strength_weakness: strength_weakness: 1.	The method relies on hyperparameters (e.g., prior precision, temperature parameter, kernel scale etc.), which might require careful tuning for optimal performance for different datasets. Authors criticize previous methods for being dependent on hyperparameters but do not overcome such limitation in this paper.
2.	The contribution of local Gaussian convolution might be overclaimed. As shown in  Table 3, the performance improvements brought by rectification are pretty limited, which look like experimental variations.
questions: questions: 1.	What is the NFE for DDRM used in this paper? What is the inference time for DDRM and your method respectively?
2.	In Algorithm 1, the threshold is set to 1e-6. Is performance sensitive to different values?
limitations: limitations: Authors discussed the limitation in the paper.
paper_summary: paper_summary: This work considers the problem of using adapt diffusion models for solving real-world image denoising problem, that is, the noise is not assumed to be i.i.d Gaussian. The authors statistically model the real-world noise as independent, non-identically distribution noise, and then incorporate the adaptive MAP inference into the reverse process of the diffusion model.
strength_weakness: strength_weakness: 1. The paper does not well motivate their choice of i.ni.d noise model. It is not clear why such an i.ni.d noise model can properly characterize the statistics of the real-world noise considered in the paper.
2. The paper is not well-written, lacking proper motivation and explanation of the design of the proposed method.
3. The approximation adopted in Eq.15 may lead to incorrect $x^\ast$. For example, when the function is non-convex, maximizing its Jenson low bounder produce an estimate that is far away from $x^\ast$.
questions: questions: 1. Why is an i.ni.d noise model used to characterize the real-world noise?
2. Why do we want to adaptively update the parameters in reverse diffusion? Please provide some high-level explanation.
3. Please justify the accuracy of the approximation used in Eq. 15.
4. In Fig.3, self2self shows the best numerical performance. Please discuss the result.
limitations: limitations: The authors have discussed the limitations.

--------------------- After Rebuttal --------------------
The responses have addressed my questions. I have raised my score to 5.
paper_summary: paper_summary: The authors propose a way to use diffusion priors for real-world image denoising where the noise statistics are complex and signal-dependent. They use variational inference to estimate the joint posterior of the noise precision and image throughout diffusion time. The result is a MAP estimate of the denoised image that takes into account signal dependence and spatial correlation of the observed noise. The authors also propose to use a low-resolution diffusion model to denoise high-resolution images. In experiments, the authors compare to single image-based denoising methods, a self-supervised denoising method, and diffusion-based methods. They show improved quantitative and qualitative performance against the baselines.
strength_weakness: strength_weakness: * The proposed method only provides MAP estimates (not posterior samples).
* Nit: Eq. 6 should be explained more clearly in the text. For example, A should be explicitly defined. And why is the same $\epsilon$ used twice? Shouldn’t it be $y_{t-1}=\sqrt{\bar{\alpha}_{t-1}}(x_0+A\epsilon_1)+\sqrt{1-\bar{\alpha}_{t-1}}\epsilon_2$ for $\epsilon_1,\epsilon_2$ independently drawn (since the measured noise is different from the synthetic noise added in the diffusion process)?
questions: questions: * How difficult would it be to adapt this method to provide posterior samples?
limitations: limitations: I appreciate the authors providing a limitations section. There are probably more limitations of the method that could be discussed, such as not addressing posterior sampling.
paper_summary: paper_summary: This paper proposed a diffusion-based image denoising method. The method leverages the MAP framework with proposed adaptive likelihood estimation method and a pre-trained diffusion prior. Experiments on four real-world datasets validate the advantages of the proposed method.
strength_weakness: strength_weakness: 1. The title of the paper does not accurately reflect its contributions. The use of diffusion priors to address inverse problems is widely studied, as the author has noted, and this paper builds upon the framework presented in [1]. The significant contribution of this paper appears to be the proposed alternative updating scheme for estimating the likelihood term. However, the title is too generic and does not capture this contribution.
2. The denoising computational cost is high, and the running time for the reverse diffusion process is lengthy.
3. In Algorithm 1: Difusion -> Diffusion.
questions: questions: The proposed algorithm involves numerous hyperparameters. How are these selected, and is the algorithm robust to variations in hyperparameter settings?
limitations: limitations: The authors have acknowledged the limitations of their work.

Comments: title: Thanks for the response.
Thanks for the response. Score raised.
comment: Thank you to the authors for their response. It would be good to add the explanation of Eq. 6 to the revised paper, but I would prefer not to add the posterior sampling variant. I would want to see further explanation/experiments to verify the proposed approximation of the posterior. I appreciate the authors' efforts at extending their method to posterior sampling nevertheless. I have no further questions/concerns and maintain my rating.
Thank you so much for the response. I have no further questions and will maintain my initial rating.
Thanks so much for your reply. We acknowledge that the proposed method requires large sampling steps, and improving the sampling efficiency will be our next move. Regarding the application to other image restoration tasks, we show that it is straightforward to apply our method to pixel-wise degradation tasks, e.g., image inpainting and demosaicing, as replied to Q2. In the future, we would like to adapt our method to other non-pixel-wise degradations, e.g., image deblurring and super-resolution.
Dear authors,

I thank you for the rebuttal. After going through the rebuttal, I have decided to maintain my rating. The key factors impacting this score is the large number of diffusion steps needed for  good performance and the limited applicability to just denoising problems.
Thanks for your reply and suggestion. Regarding using fewer sampling steps in the response to Q2, we used the DDPM sampler with fewer steps (e.g., $T=500$ means taking one sample per two steps compared with original $T=1000$). As you suggested, we will try to incorporate DDIM or DPM-Solver into our method to improve the sampling efficiency while maintaining the denoising performance.
Thank you for your response. Regarding using fewer sampling steps, are you referring to the DDIM sampler? Additionally, utilizing advanced diffusion samplers, such as DPM-Solver, may help accelerate the sampling speed. I have no further concerns and will maintain my score.
Thank you for taking the time to read the rebuttal. We are glad that our response solved your questions and thank you for increasing your score.
I thank the authors for the thorough response to my comments. I found them very clear and helpful for me to better understand the motivation of the noise model and derivation of the proposed method. After a careful re-evaluation, I have raised my score from 3 to 5.
Dear Reviewers,

We sincerely thank you for your precious time and efforts in reviewing our paper. 

We would like to inquire whether our response has addressed your questions and concerns. We are more than happy to discuss with you further and provide additional materials.

Best regards,

The Authors