SNIPS: Solving Noisy Inverse Problems Stochastically

Abstract: In this work we introduce a novel stochastic algorithm dubbed SNIPS, which draws samples from the posterior distribution of any linear inverse problem, where the observation is assumed to be contaminated by additive white Gaussian noise. Our solution incorporates ideas from Langevin dynamics and Newton's method, and exploits a pre-trained minimum mean squared error (MMSE) Gaussian denoiser. The proposed approach relies on an intricate derivation of the posterior score function that includes a singular value decomposition (SVD) of the degradation operator, in order to obtain a tractable iterative algorithm for the desired sampling. Due to its stochasticity, the algorithm can produce multiple high perceptual quality samples for the same noisy observation. We demonstrate the abilities of the proposed paradigm for image deblurring, super-resolution, and compressive sensing. We show that the samples produced are sharp, detailed and consistent with the given measurements, and their diversity exposes the inherent uncertainty in the inverse problem being solved.


Reviews and author responses:

This paper proposes a technique based on annealed Langevin dynamics to solve noisy linear inverse problems. The technique is built on top of [18, 19] and proposes to perform the Langevin dynamics in the spectral space of the linear degradation operator $H$. To improve the performance of the proposed method, the authors use coordinate wise step sizes obtained by approximating the hessian with its diagonal. Qualitative results from the proposed method are presented for compress sensing, inpainting, super-résolution and deblurring.  
Review Summary
--------------

This paper proposes a technique based on annealed Langevin dynamics to solve noisy linear inverse problems. IMO, the novelty of this paper is limited as it seems to boil down to applying [19] in the spectral space of the operator $H$ (**M1**). Moreover, the numerical experiments are quite limited, with mainly qualitative results and no comparison with other methods (**M2**). For these reasons, I recommend rejecting this paper. As I might be wrong for the spectral space part, I will carefully read the author response and revise my score if necessary.


Major comments and questions
----------------------------

* **M1:** One main concern for this paper is its limited novelty. The posterior sampling using Langevin dynamics has been described in [18] and [19] proposed a first adaptation to the inverse problem for specific operator $H$. The core contributions of the papers are thus computing the conditional score function in the spectral space of the operator and proposing to use per-coordinate step sizes:
    - **A:** The authors states that the extension of Langevin dynamics to spectral space is far from trivial but I am not really convinced about this. Looking at the equation (10), we can see that the dynamic is independent for each coordinate in the spectral space and the propose algorithm boils down to the one in [19] but applied to all variables in spectral space, where the noise levels are different for each coordinate. Equation bellow l.182 simply states that the added noise $\eta_{T,k}$ should sum to $z_T$ (*i.e.* the standard assumption but in the spectral space). The most complex part was to show that $\nabla_{\tilde x_{T, >}}\log p(\tilde x_{T, 0}|\tilde x_{T, \not 0}) = 0$ but this result is taken as an assumption (l.60 in the supplementary) while it should probably holds as $V^T$ is unitary and thus the added noise are independent for each coordinate in spectral space. Overall, it seems that the proposed derivation could be simplified a lot while retaining their efficiency.
    - **B:** For the step-sizes adaptation, it makes sense that it should be adapted on each coordinates as the noise levels are not the same. However, the benefit of using such a scheme is not demonstrated in the paper.

* **M2:** Another major concern for this paper is with the experimental results. Indeed, the paper does not propose any comparison with other methods. For inverse problem solution sampling, the authors could compare their results with the one from [30]. The mean results could also be compared with Plug-and-Play or Regularization by denoising methods, in particular with the PSNR from Table.1. Without such validation, it is hard to assess wether this method provides good results or not. Also, adding an ablation study to show the benefit of the proposed step-size compared to using the classical one would help convincing that it indeed accelerate the dynamic.


Minor comments, nitpicks and typos
----------------------------------


* **m1:** (Numerical experiments) The part of the LSUN and CelebA datasets used to train the NCSNv2 models used to approximate the prior distribution are not properly described in the text. From the code, it seems that the training set is used and that the inverse problems are constructed from the test/validation sets but this should be properly described in the paper.

* **m2:** (Numerical experiments) The parameters that are used in the experiments are not described in the paper. In particular, the sequence of sigma that is chosen (it seems to be geometrical from the code) and the step sizes

* **m3:** (Eq.(10) & Algorithme.1 - l.9) the notation $|\sigma_0^2 I - ... |^\dagger$ is confusing. I think it would be clearer to use $(\sigma_0^2 I - ... )^\dagger$ as done in Eq.(8). * **L1:** One major limitation of such method is that it is limited to simple noise models as one needs to derive an expression for the score function. Moreover, the noise level must be known in advance for the method to work. These limitation should be mention in the section.6. or discuss in the experimental results, for instance showing the results when the noise is more complex or only an approximation of the noise level is given.

* **L2:** Another limitation that should be mentioned is that this technique has only been tested for inverse problem in imaging. For more complex inverse problems where the ground truth is unknown and the noise more complex, such as in astrophysics or neuroscience, there is little chances to be able to apply such techniques to recover the signal of interest.

 I thank the authors for the answers and clarifications they provided. Originally, my main concerns with the paper were its novelty compared to [19] seems a bit inflated and the limited numerical evaluation/lack of application of the paper.

Regarding the numerical evaluation (**M2**), the authors propose in their answer to add a more quantitative evaluation and comparison with other algorithms. I find their answer satisfactory as it will allow to better evaluate the method.

Regarding the novelty (**M1**), the main proposition is to apply the method from [19] in the spectral space with adapted step sizes, as stated in the rebuttal by the authors. Re-reading l.70-75, it seems that this is already hinted in the manuscript however, the derivations in the paper does not make this very clear, making it hard to see what is a contribution or not. This gives me a mixed feeling about the paper.

Moreover, I am not totally clear about what would be realistic the applications of such methods (**L2**). As stated by reviewer KMnW, for most inverse problem, we aim for the best possible reconstruction for a given criterion. While I agree that Langevin dynamic can help quantify uncertainty and provide multiple reasonable images, I feel that the statement `When perceptual quality becomes our prime objective one should concentrate on producing a sample from the posterior distribution` is too strong.

Overall, I still think this paper should be reworked a bt before publication, with clearer exposition and at least a more concrete application. That being said, I will understand if this paper is accepted as is based on the other reviews. I will therefor raise my rating to 5.

 I thank the authors for the careful replies to my questions. I keep my recommendation that the paper should be comfortably accepted. 


 Thank you for the detailed review and thoughtful feedback. Below we address specific questions and comments.

**Limited Novelty? (M1):** Please note that the technique introduced in [18] does not handle noise in the measurements at all, thus solving a much easier problem, as compared to our work. We thus concentrate on [19] in the following answer.  
The formulation in [19] is indeed a special case of our derivation, but it is limited to denoising and inpainting, both of which rely on a relatively simple posterior sampling formulation in the pixel domain. Our generalization is far from trivial, both in (i) the derivation proposed and (ii) the practical impact of the obtained contribution.   
In terms of derivation, we show that a direct deployment of the annealed Langevin dynamics hits a dead-end, and thus we introduce a transition to the spectral space of $\mathbf{H}$, thus opening the horizon to a wide scope of problems that have not been treated as of yet with posterior sampling. Moreover, this alone is far from being sufficient, as using this spectral decoupling by itself leads to a very sensitive and slow sampler. While it is intuitively clear that different step-sizes should be deployed for the decoupled measurements equation, a direct scaling of them via their corresponding singular values (or the resulting effective noise levels) is actually incorrect. Our approach, relying on the main diagonal of the Hessian, introduces more appropriate such steps that are critical for the success of the proposed algorithm. All these contributions go far beyond previous work [18,19]. Please note that no other Langevin based algorithm that we know of has suggested such an adaptive step size vector (a notable exception is [40], but they used a Quasi-Newton method, not an analytically calculated step size vector).   
For completeness, we will add an ablation study that exposes the shortcomings of using a uniform step-size, and the unavoidable divergence when deploying simplified adaptive ones that do not rely on the Hessian.   
As for practical impact, our work is the first that enables a fast deployment of Langevin dynamics for general linear and noisy inverse problems, and it does so in an elegant unified framework. We believe that this, by itself, is a major achievement and a substantial novelty.

**Comparisons (M2):** When working on this project, we kept thinking about possible comparisons that could enrich our presentation, and found this to be very challenging. However, when assessing methods that target PSNR performance, as suggested in this comment, this is in fact doable. Thus, we warmly accept this request, and will include the following: Taking a group of samples from our algorithm and averaging them produces our MMSE approximation. This will be compared with PnP and RED, both using the same denoiser as ours, and the comparison will focus on PSNR results. Please note that while appealing, this experiment deviates from the main message of our work, of producing best perceptual quality results at the cost of reduced PSNR. Thus, we will also include LPIPS [1r] evaluation of the obtained images, aiming to show the advantage of our sampler over these more classic methods.   
Image denoising using Langevin dynamics was already done in [19]. Our work generalizes this approach to all linear inverse problems. Since the work in [30] handles only the denoising task, it is more comparable to [19] than this work.

**Numerical Experiments (m1):** We will clarify these in the revised version. We definitely separated the training set from the validation/test sets. In addition, we did not train the NCSNv2 models ourselves, but rather used the pretrained models from the original work [41]. 

**Numerical Experiments (m2):** Please refer to Appendix D of the supplementary material, as well as Table 1 in the same place. Table 1 shows that the sequence of sigmas is geometric, but perhaps we should address this in the text of Appendix D as well, in the camera-ready version. 

**Equation 10 (m3):** In Equation 8, the said matrix had positive terms, whereas the one in Equation 10 contains both positive and negative, hence the notation of the absolute value.

**Noise Models and other applications (L1 & L2):** We explain at the very beginning of section 3 that $\mathbf{H}$ and $\sigma_0$ are assumed to be known, as well as the assumption that the noise is Gaussian, all parts of the problem setting. Generalization to other cases is extremely exciting and worthy of future work, but is beyond the scope of this paper. 

References:   
[1r] Zhang, Richard, et al. "The unreasonable effectiveness of deep features as a perceptual metric." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.* 2018.

 Thank you for the detailed review and thoughtful feedback. Below we address specific questions and comments.


**Fundamental limitation:** Thank you for this valuable comment. The variety of solutions we obtain and the visible changes in them are not a characteristic of our approach but rather a manifestation of the inherent uncertainty in the inverse problem being served. Imagine that we solve a compressed sensing or super-resolution problem for a face image and our results are too spread, some showing teeth and others not. Replacing the solver by one that aims for the best possible reconstruction (e.g. MMSE) will not change the fundamental uncertainty in the obtained result. E.g., if the outcome contains teeth, can we trust this result to attest that teeth also appeared in the original image? The answer is negative. As correctly noted, and regardless of the solver used, in medical imaging the above will be remedied by adding measurements, so as to lower the uncertainty to an acceptable rate.

**Previous work and MAP/regularized estimation:** The reviewer is correct – most earlier work approached inverse problems by forming an energy function to be minimized, thus deploying either a MAP estimation or a regularized inversion. However, please observe that many (if not all) of these earlier papers assess their final outcome by evaluating and competing on PSNR performance. By the way, the hidden reason for many of these works not to consider MMSE estimate was the fact that it is far more complicated, being built on intractable integration, thus using the above alternatives as approximations for it. We will rephrase the sentence mentioned to reflect the above explanation.

**Posterior Expected Mean:** Thank you for this comment. We will add a note regarding the ability of the MMSE result to reside within the image manifold, and discuss properties of the prior in this regard.

**PnP and RED:** We agree that these methods do not necessarily use MMSE denoisers, and we will fix the text accordingly.

**Regarding lines 149 to 166:** While the explanations may seem trivial or unnecessary to the informed reader, we believe they help us better convey our message. Specifically, while Equation 4 does not aid us in the derivation itself, it helps motivate applying Langevin dynamics on the transformed versions of $\mathbf{\tilde{x}}$ and $\mathbf{y}$, showing that it is equivalent to the non-transformed distribution.

**FFT speedup:** We agree! Imposing a block circulant assumption on $\mathbf{H}$ enables a deployment of FFT and thus a speedup of the overall algorithm. This could be an interesting future direction to explore. 

**Other metrics:** As correctly stated, PSNR (and similar metrics such as SSIM) do not capture the perceptual quality of images. In order to comply with this important comment, we shall include LPIPS [1r] evaluations of our sampled results versus an alternative estimate.

**Compressed-Sensing visual results:** Indeed, these are not degraded images, but rather a visualization of the measurements (using the pseudo inverse of H). We shall clarify this in the text in the revised version.

**References:**  
[1r] Zhang, Richard, et al. "The unreasonable effectiveness of deep features as a perceptual metric." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.* 2018.

 Thank you for the detailed review and thoughtful feedback. Below we address specific questions and comments.

**Bigdeli et al. 2017:** Thank you for pointing us to this reference. Indeed, our work leverages the same connection between the blurred prior and the denoiser residual (which they refer to as mean-shift). However, whereas we aim for sampling from the posterior due to the desire for best perceptual quality results, this work is similar to RED [36] in spirit, using a more classic Bayesian energy minimization for the restoration. We will certainly add a brief discussion about this work’s connection to ours in the revised paper. 

**Objective Assessment of results:** As correctly stated, PSNR and similar metrics do not capture the perceptual quality of images. In order to comply with this important comment, we shall include LPIPS [1r] evaluations of our sampled results versus an alternative estimate. 

**Bibliography:** Thank you for this note about the bibliography. We will carefully review it for the camera-ready version and properly format our citations.

**References:**  
[1r] Zhang, Richard, et al. "The unreasonable effectiveness of deep features as a perceptual metric." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.* 2018.


 Thank you for the detailed review and thoughtful feedback. Below we address specific questions and comments.

**Significance of this work:** While solving inverse problems is a rich topic with many earlier contributions, our work is novel in targeting best perceptual quality results. This is achieved by sampling from the posterior and leveraging an MMSE denoiser for this goal. Previous solutions of inverse problems mainly compete on achieving best PSNR results, thus sacrificing visual quality. Some more recent generation-based work does consider perceptual quality for these problems. However, these methods either do not sample from the posterior, thus producing a single solution which is a compromise between PSNR and perceptual quality, or assume noiseless inverse problems. Our work is the first to remove both these limitations, addressing general noisy linear inverse problems.

**Compressed Sensing (CS):** In our CS experiments, the degradation matrix H was chosen to be a random rectangular (wide and short) matrix with fixed singular values of 1, as stated in line 263. This is equivalent to producing a subset of random projections of the unknown image. While some compressed sensing work uses the Fourier domain, we opted for the random transformation in the pixel domain, similar to the compressed sensing tests in [18], with the addition of noise.

**Question regarding Identity (3):** As we have recently learned, this identity is attributed to several well-known statisticians: Stein, Tweedie, and Miyasawa. Due to this important comment, we will cite all these sources in our paper.

The paper describes an apparently new approach to solving noisy inverse problems.

It shows how the approach can be applied to three different noisy inverse problems:
which it calls image deblurring, super-resolution, and compressive sensing. 

It validates the approach on synthetic data which is synthesized
(eg blurred and noise added) starting from publicly available imagery.

It is carefully written and I find it readable.  The paper describes an apparently new approach to solving noisy inverse problems.

It shows how the approach can be applied to three different noisy inverse problems:
which it calls image deblurring, super-resolution, and compressive sensing. 

It validates the approach on synthetic data which is synthesized
(eg blurred and noise added) starting from publicly available imagery.

It is carefully written and I find it readable.

Originality: I have seen work on langevin dynamics in inference problems before,
but I think that work was theoretically focused and the computational efforts
were just preliminary efforts rather than mature tool construction.
I believe the originality here is in the effort to deliver something
that's been carefully derived, using sophisticated identities
rather than handwaving, and carefully tested on (stylized approximations to) 
'real-world' problems.
 
Quality: The paper is very well written at the narrative level. 

Clarity: I find it very clear.

Significance: I am unable to judge whether the proposed method is an improvement over
existing methods. Does this method do better than the state of the art on the problems
in its purview (denoising, super-resolution, compressed sensing) where so many previous
efforts have been made and carefully documented? I don't know.

 I think what they call compressed sensing is a not at all compressed sensing.
They are not doing random undersampling either in Fourier space nor do they
seem to be doing random undersampling in some other randomly orthogonally transformed domain.
I think they are simply dealing with undersampled data in the original pixel domain.
Possibly this is a real problem in certain settings, but it shouldn't be
called compressed sensing.

I think the identity (3) that they call Brilliant is well known to Mathematical Statisticians for
decades as "Stein's Integration by Parts trick" after Charles Stein, a leading mathematical
statistician of the 20th century. 
See for example Stein, Charles M. (November 1981). "Estimation of the Mean of a Multivariate Normal Distribution". The Annals of Statistics. 9 (6): 1135–1151. doi:10.1214/aos/1176345632 (many other statistics papers use this idea, 
physicists also know it well by other names)

This paper proposes to combine Langevin dynamics and Newton's method to arrive at a posterior sampling algorithm, where the prior distribution is given by an MMSE Gaussian denoiser. The proposed method is evaluated on multiple imaging inverse problems like deblurring, super-resolution, etc.  Strengths: the method proposed in this paper is original and technically sound. 

The authors utilize denoiser-based deep prior and blurred posterior function, which is very similar to the approach used in (Bigdeli et al. 2017). It could be helpful if similarities and differences with that work are discussed.

Weakness:
- The introduction section extensively discusses the properties of the proposed method before actually introducing the method. IMHO the introduction should be precise, and more space can be dedicated to the experimental results section.
- The experimental results section heavily relies on subjective results. Although understandably, objective metrics like PSNR cannot reflect the perceptual quality, it is still desirable to include some objective results comparing to some baseline methods in the main text.
- In the bibliography, venues should be properly capitalized (e.g. line 335)

References:
Bigdeli, Siavash A., et al. "Deep mean-shift priors for image restoration." Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017. Have the authors adequately addressed the limitations and potential negative social impact of their work: Yes.

This paper proposes a method to sample from the posterior density in the Bayesian formulation of linear (imaging) inverse problems using Langevin dynamics. The main contribution of the paper is a way to generalize the idea (which is not new) beyond diagonal observation operators (denoising and inpainting), by resorting to an SVD of the operator.

   This paper proposes a method to sample from the posterior density in the Bayesian formulation of linear (imaging) inverse problems using Langevin dynamics. The main contribution of the paper is a way to generalize the idea (which is not new) beyond diagonal observation operators (denoising and inpainting), by resorting to an SVD of the operator and some "intricate" derivations. 

This is a very nice paper and the proposed formulation is solid and constitutes an important contribution to the topic of imaging inverse problems. However, there are a few minor issues that should be addressed. 

A fundamental limitation of the proposed method, which should be clearly acknowledged, is that sampling from the posterior is not applicable/acceptable in many domains. Look, for example, at the samples in the compressive sensing or super-resolution experiments; in some samples, the teeth are visible in others not. If this was a medical image reconstruction/restoration scenario, this could correspond to some samples showing a tumor, others not. In such a context, what is required is THE best possible reconstruction (under some optimality criterion), not samples from the posterior, which can vary significantly. 
 
Unlike what the authors claim, it is not true that "most image restoration techniques aim to form an MMSE estimator". In fact, the vast majority of image restoration approaches are formulated as optimization problems, which can be interpreted as MAP estimators or, from a regularization theory point of view, as regularized inversion of the (ill-posed) observation operator. There are a few attempts at leveraging MAP/regularization approaches to yield MMSE estimates, such as https://epubs.siam.org/doi/abs/10.1137/120902276, but it is certainly not the majority. 

The argument that "the MMSE solution averages all these candidate solutions, being the conditional mean of the posterior of x given y, leading to an image with loss of fine details" is not new, but it certainly requires some sort of justification, as it is far from evident that this is true. Unless the posterior is strongly multimodal, the posterior expected value could very well itself be a sample. As such, without any comment about the properties of the posterior, this is nothing but an empirical observation. 

In lines 104-107, the authors write “that using MMSE Gaussian denoisers iteratively for handling general linear inverse problems has been already proposed in the context
of the Plug-and-Play-Prior (PnP) method and RED, and their many follow-up papers. However, none of the cited methods use explicitly MMSE denoisers, but general-purpose denoisers. In fact, at the heart of the PnP is the idea that you can replace the proximity operator of some regularizer with a (Gaussian) denoiser, but this equivalence sees it as a MAP denoiser, not an MMSE one. The only work where PnP is used with an explicit MMSE denoiser is https://doi.org/10.1109/TIP.2018.2869727

The derivations and explanations in lines 149 to 166 are unnecessarily “intricate” (to use a word that the authors have used a lot). If you simply start with the equation between lines 161 and 162 and use the definitions in line 160, then everything (namely equation (5)) falls directly from there. In particular, you don’t need equation (4), which anyway is not referred to in the rest of the paper and is trivial from the formulation between lines 161 and 162. 

It would be interesting to see if the formulation (and the resulting computation) is significantly simplified in the case where H is a periodic convolution (thus with M=N), in which case H is diagonalized by the discrete Fourier transform and, consequently, multiplications by U and V can be done with O(N log N) cost using FFTs.

In the experiments, the main claim is that, although having a lower PSNR (which is natural, since they are not the MMSE estimate, which maximizes the PSNR), the samples are more "visually appealing". This claim could be, at least in part, supported on other quality metrics, such as SSIM (structural similarity index measure), which correlates better with human quality perception.

Finally, there is no point in showing the ”degraded” images in the compressive sensing experiments, since these are not degraded images, but observations that are meaningless to a human observer. 





 Some limitations have not been addressed, as mentioned in the main review above. Namely, that this type of approach may not make sense in some contexts, such as for medical image reconstruction. My recommendation is that this limitation should be briefly discussed in the paper.
 

Meta-review: The paper considers noisy linear inverse problems y = Hx +noise, and specifically, the problem of generating samples from the posterior p(x|y) using a Langevin dynamics approach. This has been considered before quite recently, but the novelty of the current paper is working with noise as well as extending ref [19] to work with more general linear operators H.

This paper generated a lot of discussion among the reviewers and between the reviewers and author(s). We all found the paper quite interesting!  There were many suggestions for improvement, even from reviewers with very high scores.  Some common themes are that some of the derivations are more complicated than necessary, and missing some prior work (e.g., from Stein), and that the innovation over [19] isn't strong enough.

Despite these weaknesses, the majority opinion is that the results are strong enough to warrant publication.  One reviewer was impressed that this work is able to make Langevin dynamic sampling so practical -- the actual experiment results shown in the figures speak for themselves.

I strongly recommend to the authors to take into account the reviewer comments in their final version of the paper. The objective is not to publish a paper in NeurIPS, but rather to publish the best possible paper that will have a large impact. We had a strong group of reviewers with good comments, so I suggest following their advice when possible.

Decision: accept