Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors proposed a stochastic coarse-to-fine gradient ascent method for drawing high-probability samples from the implicit prior to denoise the noisy image and generalizes it to more broad linear inverse problems.
main_review: The idea is quite interesting, i.e., building a plug-and-play denoiser to inverse problems.
However, the proposed approach is not very clearly explained. There are many incorrect or unclear claims, which are detailed in the below box.
The paper is also not very well organized, e.g., methodology and experiments are mixed in one section. The authors' contribution in methodology is not clearly separated from existing works.
limitations_and_societal_impact: Paper organization: 
The proposed method and experimental results are mixed in Section 3, which does not help readers understand the proposed solution better. It is more readable to separate the methodology and experiments into two sections and clearly present the methodology contribution. 

There is only a subsection in Section 2. Also, in Section 2.1, the title Image synthesis is not logically connected with the denoiser, which should be explicitly explained. Also, this section is about experiments and should be merged with Section 3.2.

Technical issues 
Line 173-174: This part is not clear since \bar{M} is not clearly defined. How y is decomposed as (y^c, y^u) should be explicitly explained using the property of orthogonal complement. Also, why y^c can be omitted conditional on x^c in p(y^u|y^c, x^c)? 
Last but not least, the dimension of y and x is very likely not the same in super-resolution, inpainting, compressive sensing, etc. Let’s say, x in R^n and y in R^m, thus m<n. If M^T can be left multiplied to x, how can it be also left multiplied to y as well?

Line 74: The authors’ claim that (2) is least square estimation is not correct. The conditional mean of the posterior is the MMSE estimator.

In Fig 6, the notation of denoiser D is not consistent with the text and Algo 1, which can easily confuse readers.

Note that the proposed method requires Monte Carlo sampling, which might be quite time-consuming. The computation complexity needs to be analyzed and in experiments, the running time for different methods should be compared.

Minor issues:

Line 78: Solves—> solves it.

In Fig 3, the reference for DeepRED is not correct.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Classical result by Miyasawa [25]  shows that the gradient of the log of the noisy signal density  can be expressed in terms of least-squares optimal denoisers for additive Gaussian noise.  Using this result, the authors develop an algorithm which uses  stochastic gradient ascent  to sample from the implicit prior of the denoiser, where the gradients  are computed using the residual of a trained bias-free universal CNN denoiser.  The algorithm includes noise injection which allows stochastic sampling of the manifold. They subsequently modify this algorithm to account for constraints from noise-less linear measurement model to solve linear inverse problems. The authors perform different linear image reconstruction tasks including spatial super-resolution, deblurring, compressive sensing, inpainting and reconstruction from random missing pixels. Experimentally, they demonstrate faster and visually better reconstructions as compared to the considered baselines.
main_review: The paper is nicely motivated, well-written and clear. The overall topic is highly relevant and the numerical results are encouraging. Yet, despite the detailed explanation given in lines 255 to 272, the difference to [20] did not become perfectly clear to me from an algorithmic perspective and thus raises questions about the originality of the research in my oppinion. The authors state that their method is not based on Langevin dynamics. The update equation for y in eq.(4) is, however, identical to the update equation of x in [20] except for the way the scaling factors / step sizes are computed. Both approaches update the variable with the help of (differently weighted) combinations of the denoiser residual and random noise. Thus, I believe it is crucial to discuss the difference of the proposed approach with respect to Langevin dynamics in more detail. In particular, the Langevin dynamics of [20] could be used / compared to in the numerical results. As the paper is validated empirically (with no convergence proof of the proposed scheme), I believe it is crucial to describe why the two approaches are different despite their very similar appearance of the algorithm. 

To better describe the relation of the proposed work in the overall context of reserach on using denoising priors, I am attaching some related works below, which could be of interest. (Some of them are preprints only and of course to not have to be considered, yet could be of general interest to the authors). 

Typos:
 - line 118: maniofld >> manifold
 - line 126- 127: residual of the denoising function proportional to the gradient of p(y) >>  residual of the denoising function proportional to the gradient of log p(y) 



[a] Guo, Bichuan, Yuxing Han, and Jiangtao Wen. "Agem: Solving linear inverse problems via deep priors and sampling." Advances in Neural Information Processing Systems 32 (2019).

[b] Ramzi, Zaccharie, Benjamin Rémy, Francois Lanusse, Jean-Luc Starck, and Philippe Ciuciu. "Denoising Score-Matching for Uncertainty Quantification in Inverse Problems." In NeurIPS 2020-34th Conference on Neural Information Processing Systems/Workshop on Deep Learning and Inverse Problems. 2020

[c] Regev Cohen, Michael Elad, and Peyman Milanfar. "Regularization by denoising via fixed-point projection (red-pro)." (2020)

[d] Kawar, Bahjat, Gregory Vaksman, and Michael Elad. "SNIPS: Solving Noisy Inverse Problems Stochastically." arXiv preprint arXiv:2105.14951 (2021)

[e] Saharia, Chitwan, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. "Image super-resolution via iterative refinement." arXiv preprint arXiv:2104.07636 (2021).

[f] Laumont, Rémi, Valentin De Bortoli, Andrés Almansa, Julie Delon, Alain Durmus, and Marcelo Pereyra. "Bayesian imaging using Plug & Play priors: when Langevin meets Tweedie." arXiv preprint arXiv:2103.04715 (2021)

-----------------------------
I would like to thank the authors for their explanation which helped me understand the difference to Langevin dynamics better. Due to the large algorithmic similarity to Langevin dynamics I would have still appreciated a comparison to (or at least a small test on) using an algorithm like [20] in the same fashion the authors are solving inverse problems based on the idea of algorithm 1. This would contribute to my understanding whether it is the use of stochasticity or the specific way it is introduced (i.e. controlling the amount of noise via the magnitude of the residual predicted by the denoiser) that leads to the superior results. Although it at least seems like [20] could easily be adapted to solve inverse problems the same way as proposed in this paper, I understand that [20] addressed generative tasks only (counting inpainting as a generative task). I will therefore raise my score to 7. 
limitations_and_societal_impact: The limitations are adequately discussed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a score-based method that solves deterministic linear inverse problems by drawing samples from the denoising priors. By iteratively maximizing the score $\textsf{log}p(y)$ using Langevin process, high-probability samples can be drawn from the distribution $p(x)$ embedded in the denoiser. However, in order to perform this, $\nabla_y\textsf{log}p(y)$ needs to be effectively computed. By leveraging the classic Miyasawa’s equation for MSE denoisers, the authors are able to use a residual CNN denoiser trained for minimizing MSE to approximate the term $\textsf{CNN}(y) = \hat{x}(y)-y = \nabla_y\textsf{log}p(y)$. 
The authors also extend their method to linear deterministic linear inverse problems by expressing $\textsf{log}p(y)$ into two orthogonal components accounting for noise density and constraint, respectively.

The method is related to the recent Plug-and-play (RED) and Regularization-by-denoising (RED) frameworks which take an optimization-centric view of using denoisers. However, it differs from the former in the sense that it explicitly relates the denoiser with the prior distribution. In the experiments, one self-supervised neural network and one RED algorithm are used as baselines, achieving inferior performance compared with the proposed method in single image super-resolution, denoising, deblurring, and compressive sensing. 
main_review: **Originality:** This work is novel in relating the MSE denoisers to priors in an explicit way by using the classic result. The difference between PnP/RED and the proposed method is clearly stated. However, I found a concurrent paper [1] that seems to propose a similar method. I believe a discussion between your and their methods can avoid future confusion. Moreover, investigations on the relationship between denoisers and explicit regularizers have also been conducted for MMSE denoisers within the PnP framework [2].

[1] Laumont et al. 'Bayesian imaging using Plug & Play priors: when Langevin meets Tweedie\
[2] Xu. et al. Provable Convergence of Plug-and-Play Priors With MMSE Denoisers

**Quality:** This paper conducts adequate experiments to support their claims. The results are technically sound and convincible. For questions please see my comments below. Also, the limitations of the proposed method are well-discussed.

**Clarity:** Since I am not very familiar with the score-match field, the paper is a little bit difficult for me to follow. A supplement with some introductory details (e.g. derivation of Langevin process) will be welcome for readers like me.

**Significance:** This paper provides interesting insights from a Bayesian perspective, which can be useful for the theoretical studies of PnP/RED. 


limitations_and_societal_impact: 1. While reading the manuscript, I encountered some confusions that need authors' explanation: 1) Throughout the paper, it seems to me that the final output is $\hat{x}(y)$. However, the algorithm summarized in Alg. 1 seeks to update $y$, which I assume to be related to the noisy observation. Is there a hidden step $\hat{x}(y)=f(y)+y$ for dumping the final output? or am I misinterpreting the variables? 2) Sampling $y$ seems quite unusual if my interpretation of $y_t$ is correct. In [1], I saw their formulations are for $x$. 3) I did not quite see why from the equation above eq 7 we can have eq 7. Could the authors provide some intuitive explanations? Derivations are also welcome.

2. A discussion between your method and [1] is needed.

3. Is it possible to construct a simple toy problem such that the oracle MMSE denoiser is reachable. In this way, we can compare CNN with this oracle denoiser, which may bridge the gap between theory and experiment. Can the experimental setup (Compressive Sensing with Bernoulli-Gaussian distribution) in [2] directly reusable to yours? 

4. In experiments, deepRED is treated as the PnP baseline. Although RED and PnP are closely related, the exact equivalence between the two is still an open question. I suggest including one exact PnP baseline (PnP-ISTA or PnP-ADMM) in the experiments.

5. Minor: In line 245 Section 4, the sentence implies that PnP is coupled with ADMM. However, this is a long-standing misunderstanding of the PnP framework. Many works have shown that PnP is compatible with gradient-based [3], primal-dual [4], and message passing algorithms [5]. 2) 


[3] Kamilov et al. A Plug-and-Play Priors Approach for Solving Nonlinear Imaging Inverse Problems\
[4] Ono et al. Primal-Dual Plug-and-Play Image Restoration\
[5] Metzler et al. Learned D-AMP: Principled Neural Network-Based Compressive Image Recovery
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper derives an algorithm for solving linear inverse problems using the prior that is implicit in an end-to-end denoiser in the form of a CNN.

Consider a denoising algorithm in form of a CNN that takes a noisy observation $y = x+z$ as input and yields an estimate of the signal $x$. CNNs trained end-to-end yield state-of-the-art denoising performance. After training, the image/signal prior is implicit in the CNN. The paper notes that the least-squares-estimate of the signal given a prior $p(x)$ is 

$$
\hat x(y) = y + \sigma^2 \nabla_y \log p(y),
$$

where $p(y)$ is the noisy observation density. Based on this relation, the paper proposes a method to obtain a high-probability image (sample) from the prior implicit in a denoiser. The idea is as follows: We can denoise a given image by evaluating $\hat x(y)$, and by computing the residual $f(y) = \hat x(y) - y$ we obtain an image that is proportional to $\nabla_y \log p(y)$. Based on this insight, the paper proposes to sample from the prior implicit in a denoiser by running stochastic gradient ascent through iterating $y_{t+1} = y_t + h_t f(y_t) + \gamma_t z_t$, where $z_t$ is Gaussian noise, and $h_t$ and $\gamma_t$ are tuning parameters. The paper demonstrates (Fig. 2) that this method can yield realistic samples of low-dimensional images. 

Next, the paper then modifies this approach to solve linear inverse problems by obtaining a high-probability sample based on partial linear measurements of a signal/image. The paper evaluates the method on standard inverse problems, including super-resolution, deblurring, and compressive sensing. The paper shows that the method works well, both in numbers and by providing example images. 
main_review: The paper introduces an original way to solve linear inverse problems and demonstrates with simulations that it works well, both computationally and in terms of imaging quality. The idea is well explained, the paper is well written, and the experiments support the claims. I would be happy to see the paper at the conference.

Here are a few comments/experiments that would make the paper even stronger:
- The paper uses as an denoiser the bias-free CNN from [36], and writes it obtains similar results with other-bias free denoisers. I would be curious to also see results for other standard CNN-denoisers, in particular for U-net, because U-net or similar encoder-decoder architectures work very well for denoising and other inverse problem. If the proposed approach does not work well with U-net, that wouldn't make the paper any less interesting of course. I would also be interesting to show results for standard CNN denoisers that are not bias-free, and potentially add a brief discussion whether bias-freeness ist important or not for this method to work. 
- For the numbers in Table 1 ``ours - avg'', it would be great to also give confidence intervals or a standard deviation, to get intuition on the variation across samples.
- Another interesting baseline that is also easy to train for table 3 is a U-net trained end-to-end for compressive sensing. Runtimes for CS would also be great to add.
- Can the method also give realistic samples of high-resolution images? I didn't understand from the paper what the resolution of the images in figure 4 or 5 is.

limitations_and_societal_impact: Overall the paper adequately discusses limitations. One interesting comparison to add is U-net-based reconstruction in Table 3 (so U-net applied to M^Ty, where y is the measurement), along with corresponding computational costs. I expect that this will show that U-net-based reconstruction is at least on par in terms of image quality, while being faster.
review_title: null

Meta-review: This paper provides an original way to solve linear inverse problems when given a denoiser.  The method uses forward calls to the denoiser in a way that leverages the signal prior implicit to it.  The paper establishes strong quantitative and qualitative recovery performance.  In the camera ready version of the paper, the authors should make the modifications discussed with the reviewers.  In particular, the authors should clarify the relationship between the proposed method and Langevin Dynamics.  The authors say the method draws "high probability samples"; the authors should clarify what these terms mean and whether or not it is the same as directly sampling from the relevant distribution.  
Decision: Accept (Poster)