Stochastic Solutions for Linear Inverse Problems using the Prior Implicit in a Denoiser
Authors: Zahra Kadkhodaie
Eero P Simoncelli
Venue: NeurIPS 2021 Poster (2021)
Keywords: inverse problems
image priors
unsupervised learning
semi-supervised learning
deblurring
superresolution
compressive sensing

Abstract: Deep neural networks have provided state-of-the-art solutions for problems such as image denoising, which implicitly rely on a prior probability model of natural images. Two recent lines of work – Denoising Score Matching and Plug-and-Play – propose methodologies for drawing samples from this implicit prior and using it to solve inverse problems, respectively. Here, we develop a parsimonious and robust generalization of these ideas. We rely on a classic statistical result that shows the least-squares solution for removing additive Gaussian noise can be written directly in terms of the gradient of the log of the noisy signal density. We use this to derive a stochastic coarse-to-fine gradient ascent procedure for drawing high-probability samples from the implicit prior embedded within a CNN trained to perform blind denoising. A generalization of this algorithm to constrained sampling provides a method for using the implicit prior to solve any deterministic linear inverse problem, with no additional training, thus extending the power of supervised learning for denoising to a much broader set of problems. The algorithm relies on minimal assumptions and exhibits robust convergence over a wide range of parameter choices. To demonstrate the generality of our method, we use it to obtain state-of-the-art levels of unsupervised performance for deblurring, super-resolution, and compressive sensing.


Stochastic Solutions for Linear Inverse Problems using the Prior Implicit in a Denoiser
Zahra Kadkhodaie Center for Data Science New York University zk388@nyu. edu
Eero Simoncelli
Center for Neural Science, and Courant Inst. Mathematical Sciences,
New York University
Flatiron Institute, Simons Foundation eero. wwm.ocelllmmnnn........ee..mllll @@yu..duu edu
Abstract
Deep neural networks have provided state-of-the- solutions for problems such as image denoising, which implicitly rely on prior probability model of natural images. Two recent lines of w Denoising Score Matching and Plug-and-Play propose methodologies for drawing samples from this implicit prior and using it to solve inverse problems, respectively. Here, we develop parsimonious and robust generalization of these ideas. We rely on a classic statistical result that shows the least-squares solution for removing additive Gaussian noise can be written directly in terms of the gradient of the log of the noisy signal density. We use this to derive stochastic coarse-to-fine gradient ascent procedure for drawing high-probability samples from the implicit prior embedded within CNN trained perform blind denoising. A generalization of this algorithm to constrained sampling provides a method for using the implicit prior to solve any deterministic linear inverse problem, with no additional training, thus extending the power of supervised learning for denoising to much broader set of problems. The algorithm relies on minimal assumptions and exhibits robust convergence over wide range of parameter choices. To demonstrate the generality of our method, we use it to obtain state-of-the-art levels of unsupervised performance for deblurring, super-resolution, and compressive sensing.
1 Introduction
Many problems in image processing and computer vision rely, explicitly or implicitly, prior probability models. Describing the full density of natural images is daunting problem, given the high dimensionality of the signal space. Traditionally, models have been developed by combining assumed symmetry properties (c.g., translation-invariancc,, dilation dilation-invaaiancee) with simple parametric forms (c.g., Gaussian, exponential, Gaussian mixtures), often within pre-specified transformed coordinate systems (e.g., Fourier transform, multi-scale wavelets). While these models have led to steady advances in problems such as denoising (e.g., (1. they are too simplistic to generate complex features that occur in our visual world, or to solve more demanding statistical inference problems.
In recent years, nearly all problems in image processing and computer vision have been revolution- alized by the use of deep Convolutional Neural Networks (CNNs). These networks are generally optimized in supervised fashion to obtain a direct input-output mapping for specific task. This approach does not explicitly rely on a known prior, and offers performance far superior to prior-based methods. The downside, however, is that the learned mappings intertwined with the task for which they are optimized, and most cases require training a separate network for each new applica- tion. In contrast, a prior probability model can provide a universal substrate for solving inference
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

problems, The superior performance of CNNs suggests that they embed, implicitly, sophisticated prior knowledge of images. These implicit priors arise from a combination of the distribution of the training data, the architecture of the network g regularization terms included in the optimization objective, and the optimization algorithm.
Here, our goal is to extract the implicit prior from a network trained for denoising and use it to solve other inverse problems without further training. We choose denoising not because it is of particular importance or interest, but because we can make the relationship between mapping of a denoiser and a prior explicit. We combine the advantages of prior-based and mapping- -ased approaches, deriving a general algorithm for solving linear inverse problems using the prior implicit in a trained denoiser. We start with result from classical statistics [ that states that a denoiser that aims to minimize squared error of images corrupted by additive Gaussian noise may be interpreted as computing the gradient of the log of the density of noisy images. This result is related to Score Matching [[O), but provides a more direct relationship between least-squares optimal denoising and the embedded prior [ 12. We develop a stochastic ascent algorithm that uses this denoiser-estimated gradient to draw high-probability samples from the embedded prior. Importantly, we use a blind denoiser that can handle noise contamination of unknown amplitude, which provides a means of adaptively controlling the gradient step sizes and the amplitude of injected noise, enabling robust and efficient convergence. We then modify the algorithm to incorporate constraints arising from any deterministic linear measurement of an image. The resulting procedure generates bigh-probabilityy samples from the prior conditioned on the mcasurements, thus providing a general stochastic solution for any deterministic linear inverse problem. We demonstrate that our method produces visually high-quality results in recovering missing pixels, and state-of-the-ard levels of unsupervised performance on superresolution, deblurring and compressive sensing Earlier versions of this work were presented in
This work is closely related to two lines of research. Nearly decade ago, strategy known as Plug-and- Mog-aan-PPaay (P&P) was proposed for using a denoiser as a regularizer in solving other inverse problems 14. and number of recent extensions have used this concept to develop MAP solutions
for linear inverse problems [15 22 [2] Generally, the objective is decoupled into data fidelity and regularization terms, introducing slack variable for use in a proximal optimization algorithm (e.g.. ADMM). The proximal operator of the regularization term is interpreted as the MAP solution of denoising problem, and is replaced by a denoiser. Of particular relevance to work, recent publications have proven convergence of such algorithms when used in conjunction with MMSE denoisers 23 24 A parallel line of research has focused on the use of generative models based on Score Matching (55 The connection between Score Matching [TC] and denoising autoencoders
by proving that the training criterion of denoising autoencoder is equivalent to matching the score of the model and a Parzan density estimate of the data. Most recently, this idea has been used as the basis for an MCMC algorithm for sampling from the prior implicit in a CNN denoiser
B1 was first shown in h
29 In Section we elaborate on how these methods are related to our results.
1.1 Image priors, manifolds, and noisy observations
Digital photographic images lie in high-dimensional space (IE-N, where N is the number of pixels), and simple thought experiments suggest that they are concentrated on or near low-dimensional manifolds whose local coordinates represent continuous deformations and intensity variations. In contrast, images generated with random pixels are almost always feature and content free, and thus not considered to be part of this manifold. We can associate with this manifold a prior probability model, p(x), by assuming that images within the manifold have constant or slowly-varying probability, while unnatural or distorted images (which lie off the manifold) have low zero probability. Suppose we make a noisy observation of an image, y = x + z, where & E RN is the original image drawn from p(x), and ~ (0, o² Iw) is a sample of Gaussian white noise. The observation density p(y) is related to the prior p(x) via marginalization:
ply) p(yl=)((odda = 1 g6v ())()dd())d
where g(2) the Gaussian noise distribution. Equation Tb is in the form of convolution, and thus p(y) is a Gaussian- version of the signal prior, p(z). Moreover, the family of observation
A software implementation of the sampling and linear inverse algorithms is available at /tttps://githuub com/ htap///bfrrroopppaaraooo//aaaals http:/rrssssrereeeeereerrroooaaa

---

densities over different noise variances, (1), forms Gaussian scale-space representation of the analogous the temporal evolution of diffusion process.
prior 33
1.2 Least squares denoising and CNNs
Given a noisy observation, y. the minimum mean squared error (MMSE) estimate of the true signal is well known to be the conditional mean of the posterior density:
p(yl=)p(z) dx
x(y) xpp(aly)dr=
p(y)
The structure of the equation mirrors the traditional approach to the problem: one chooses prior probability model, p(x), combines it with likelihood function describing the noisy measurement process, p(y]ar), and solves. Modern denoising solutions, the other hand, are often based on supervised learning of a direct mapping from noisy to denoised images. One expresses the estimation function (as opposed to the prior) in parametric form, and the parameters by minimizing the denoising MSE over large training set of example signals and their noise-corrupted counterparts (35]- [ Current sfate-of-the-art denoising results using CNNs obtained with this supervised approach Recent analysis of these networks demonstrates that when they are trained handle a broad range of noise levels, they perform an approximate projection onto a ww--imensional subspace fp
far superior to results of previous methods 33 A1).
In our context, we interpret this subspace as
tangent hyperplane of the image manifold.
1.3 Exposing the implicit prior through Empirical Bayes estimation
Trained CNN denoisers contain detailed prior knowledge of image structure, but Eq. (2) suggests that it is embedded within -ish-dimensional integral. How can make use of this implicit prior? Recent results have derived relationships between Score Matching density estimates and denoising.
and have used these relationships to make use of implicit prior information [3 4
29
35
Here,
we exploit a more direct but less-known result from the literature on Empirical Bayesian estimation.
The idea introduced in [dd, extended the case of Gaussian additive noise in 9
(see also [2],,
and generalized to many other measurement models [.] In the case of additive Gaussian noise, one can rewrite the estimator of Eq. as:
a(y) y + o² Vy, log)
(3)
The proof is relatively straightffrward. The gradient of the observation density of Eq. is:
1 0² / v)p(y, æ)dz.
V p(y) 0² y)g(y x =)p(=)dx
Multiplying both sides by a² /p(y) and separating the right side into terms gives:
22 V p(y) p(y)
= zp(xly)dr= - yp(xly)de ((y) y.
Rearranging terms and using the chain rule to compute the gradient of the log gives Eq. 3 This remarkable result re-expresses the integral over the prior and likelihood of Eq. 2 in terms of gradient. Note that 1) the relevant density not the prior, p(x), but the noisy observation density, p(y): 2) the gradient is computed on the log density (the associated "energy function")) and 3) the gradient adjustment not iterative the estimate is achieved in a single step, and holds for any noise level, a.
2 Drawing high-probabiliy samples from the implicit prior
Suppose we wish to draw sample from the prior implicit in denoiser. Equation B allows us to generate image proportional to the gradient of log p(y) by computing the denoiser residual, f(v) = a(v)= - y. Song and Ermon [29] developed a Markov chain Monte Carlo (MCMC) scheme, combining gradient steps derived from Score Matching and injected noise in Langevin sampling algorithm to draw samples from a sequence of densities por (y), while reducing a in a sequence of discrete steps, each associated with appropriately trained denoiser. In contrast, starting from random initialization, yo. we aim to find sigh-probbbility image (i.e., an image from the manifold)
using more direct and efficient stochastic gradient ascent procedure.
3

---

2.1 Unconstrained sampling algorithm
We compute gradients using the residual of a universal blind CNN denoiser, which automatically estimates and adapts each noise level. On each iteration, the algorithm takes small step in the direction specified by the denoiser, moving toward the image manifold and thereby reducing the amplitude of the effective noise. Under the interpretation that the denoiser performs projection onto the current approximation of the image manifold, this noise reduction occurs in the subspace orthogonal to that manifold, and noise components parallel to the manifold are retained. As the effective noise decreases, the observable dimensionality of the image manifold increases [22 enabling the synthesis of detailed image content. Since the family of observation densities, Pa (1) forms scale-space representation of p(a)), the algorithm may be viewed as form of coarse-to-fine optimization 47 150 Assuming the step sizes are adequately controlled, the procedure will converge to local optimum of the implicit prior i.e., point on the manifold. Figure provides a visualization of this process in two dimensions.
Fach iteration operates by taking deterministic step in the direction of the gradient (as obtained from the denoising function) and injecting some additional noise:
3/+ = Vt-1 + h (2h- + 7/
(4)
where (()) = =ty) - y is the residual of the denoising function, which is proportional to the gradient
of log p(y), from Eq. (3). The parameter h E [0, 1] controls the fraction of the denoising correction that is taken, and controls the amplitude of sample of white Gaussian noise, ~ N (O,, T). The purpose of injecting noise is two-fold. First, from an optimization perspective, it allows the method to avoid getting stuck in local maxima. Second, it allows stochastic exploration of the manifold, yielding a more diverse (higher entropy) family of solutions. The effective noise variance image Yt is:
of = (1 - 111-1,)20, + +
(5)
where the first term is the variance of the noise remaining after the denoiser correction (assuming the denoiser is perfect), and the second term is the variance arising from the injected noise. The assumption of perfect denoising is an idealization, but we show empirically (Fig. I that the algorithm converges reliably, with error levels falling as predicted by Eq. or faster, across different settings of 3 and hg.
To ensure convergence, require the effective noise variance each time step to be reduced, despite the injection of additional noise. For this purpose, we introduce parameter 3 E [0, to control the proportion of injected noise (B = indicates noise), and enforce the convergence by requiring that:
of (1 - -22) 2
(6)
Combining this with Eq.
S yields an expression for % in terms of be:
2/ = [(1 = [1- Bh,)² 2 (1 ho)a] [f( /N,
0z
where the second equation assumes that the magnitude of the denoising residual provides a good estimate of the effective noise standard deviation, as was found in 42). This allows the denoiser to adaptively control the gradient ascent step sizes, reducing them as the y approaches the manifold (see Fig. (7). This automatic adjustment results in efficient and reliable convergence, as demonstrated empirically in Fig. Our initial implementation with small constant fractional step size he = ho produced high quality results, but required many iterations. Intuitively, step sizes that are fixed proportion of the distance to the manifold lead exponential decay a form of Zeno's o's paradox. To accelcrate convergence, introduced a schedule for increasing the step size proportion, starting

---

hg 0.1
ho
8 8=1
50
8
40
8 = 0.01
==1
30
B ==
0.4
0.011
0.2
0.0
100
200 300 400 500 600
16
24
38
59
92
143
222
346
No 0.01
0.01
8-1
8 - 0.1
B = 0.01
40
8 0.01
30
0.4
0.2
0.0
100 200 300 400 500 600
16 24 38
59
92
143
222 346
number iterations before convergence (log-spaced)
Figure Convergence of sampling algorithm, quantified in terms of the effective noise standard deviation |d,ll for three different values of [3 and two valus of ho- Left. Convergence of vN
single examples (solid curves) is well-behaved and efficient in all cases. Dashed curves indicate the convergence predicted from the formulation the algorithm: a, = BBh,) ) For / (no injected noise). the empirical convergence closely approximates the prediction. For larger amounts of injected noise (smaller B), convergence is slower, but faster than predicted. Right. Distribution of number of iterations before convergence to criterion level of 0.01 for 50 images. Red symbols indicate average values for each and hg-
from ho E (0, 1). The sampling process is summarized in Algorithm D and is provided block diagram in Fig. 6
Algorithm 1: Coarse-to-fine stochastic ascent method for sampling from the implicit prior a denoiser, using denoiser residual f1) i(y) y.
parameters: do, dT., ho,
initialization: = 1, draw yo ~ NN..5. 1
while a. oT, do
h; kgt I=
f(ye-1);
af 13.
((1 Bh:)) 2 (1 h2)3) of:
Draw 2; ~ NNO..) I):
+ 11-1 + hide + 7:21
{ + t-1
end
2.2 Image synthesis examples
For the denoiser, we used BF-CNN 42 bias-free variant of DnCNN 39 We obtained similar results (not shown) using other CNN architectures described in 42 including Recurrent- CNN. Dense-Net, and truncated U-Net. We trained this network on three different datasets: 40 x 40 patches cropped from Berkeley segmentation training set (5I, color and grayscale, and MNIST dataset (see Appendix A for further details). We chose parameters on = 1, 0.01, and hg 0.01. Figure provides visualization of two example trajectories, and diversity samples. Additional visual examples, obtained with different levels of 3, are shown in Appendix D
52

---

7
Figure 2: Left. Visualization of sampling algorithm trajectories. Each row shows a sequence of images, y/. f 1,9, 17,
25, .... from the iterative sampling procedure, with different initializations, You and no added noise (8 = 1), demonstrating the way that the algorithm amplifies and "hallucinates" structure found in the initial (noise) images. Right. Sampling diversity. Inpainting examples generated using two BF-CNN denoisers. First column: original images. Second column: partially Restored examples.
measured images, with missing block Right three columns:
Solving deterministic linear inverse problems using the implicit prior
Many applications in signal processing be expressed deterministic linear inverse problems- deblurring, super-resolution, estimating missing pixels (e.g., inpainting), and compressive sensing are all examples. Given set of linear measurements of an image, 2 M x, where M is low-rank measurement matrix, one attempts to recover the original image. In Section we developed stochastic aaatientt-scentt algorithm for obtaining a high-probbbility sample from p(z)). Here, we generalize this algorithm to solve for a high-prober brgb-probbbility sample from the conditional density p(x|MMx x²). Geometrically, constrained sampling corresponds to drawing points which sit the intersection of the image manifold and the constrained hyperplane (see Fig. IT.. Note that for these problems, the injection of noise (3 < is particularly important, because points the intersection are not necessarily the closest points to the initial image.
3.1 Constrained sampling algorithm
Consider the distribution of noisy image, y, conditioned on the linear measurements, xf =M² x. Without loss generality, we assume the columns of the matrix are orthogonal unit vectors We project y onto complementary subspaces spanned by the measurement matrix and its orthogonal complement M. We write the conditional density of the noisy image conditioned on the linear measurement
p(y|=*)= = p(y", 1²|=²)) pl x )(y*)>>) =
where === M y. y²th = Myy y. The last equality is obtained by considering that yf is equal to xe plus independent Gaussian noise. So given xf, yf does not provide any additional information about y" That is, y" is independent of of when conditioned on x². As with the algorithm of Section 2 wish to obtain a local maximum of this function using stochastic coarse-to- to-fine gradient ascent. Applying the operator o² o²V log(-) yields
2 logl())=") o'V, log (oog((y"aar (x=" + a V²V,, log
The second term is the gradient of the log of the observation noise distribution that lies within the measurement subspace (column space of M). For Gaussian noise with variance o², it reduces to M x²). The first term is the gradient of log of the noisy image distribution in the subspace orthogonal to the measurement subspace, conditioned by projecting the measurement subspace out of the full gradient given by the denoiser residual. Specifically, project ffy onto the orthogonal complement of M using the matrix I - MMM. Combining these gives:
the measurements. This can be computed
logr(v)= (T- MM log M(x v²) (I )f + M(x M(x- M²
Thus, we see that the gradient of the conditional density partitioned into two orthogonal components, capturing the gradient of the (log) noisy density, and the deviation from the constraints, respectively.
'If not, we can re- parameterize to an equivalent constraint using the SVD. In this case, M is the pseudo- inverse of M and the matrix M M/ projects image onto the measurement subspace.

---

cropped
low res
DIP R
DeepRED 20
Ours
Ours:avg
Figure 3: Spatial super-resolution. First column shows cropped portion of original images from Set14 (face: 272 x 272 and, Barbara: 720 576). The algorithm can be applied to images of any resolution we show cropped portions to facilitate visual inspection. Second column shows cropped portion with resolution reduced by averaging over 4x4 blocks (dimensionality reduction 6. 6.25%). Next three columns show reconstruction results obtained using DIP [. DeepRED 20). and our method. In all cases, our method produces an image that sharper with less noticeable artifacts (c.g.. note blocking/aliasinn artifacts along diagonal contours of lower image). The last column shows an average over 10 samples obtained by our method.
To draw prob-brobybility sample from p(x(==), we use the same algorithm described in Section 2 substituting Eq. 7 for the deterministic update vector, de (see Algorithm 2 in Appendix). Note that without any measurements (i.e., M = 0) Algorithm 2 reduces to Algorithm |
3.2 Linear inverse examples
We evaluate our method on three linear inverse problems (and provide two additional problems in Appendix B The same algorithm and parameters are used on all problems only the measurement matrix M and measured values M are altered. In particular, as in section 2.2 we used BF-CNN 42 and chose parameters σg 7 0.01, ho 0.01, 8 0.01. For each example, we show original images (z), the direct least-squares reconstruction (MM z), and restored images. Subjective assessment of perceptual quality is particularly important in cases where the measurement matrix of very low rank, and the distribution of solutions is diverse (the synthesis examples of the previous section correspond to limiting case, with measurements rank zero). We also provide numerical comparisons with other unsupervised methods, in terms of both PSNR and SSIM (an approximate measure of perceptual quality). Since our method stochastic, we provide standard deviations of these performance values across 10 realizations.
Spatial super-resolution Here, one aims reconstruct high resolution image from a low resolution (i.c. downsampled) image. Downsampling is typically performed after lowpass filtering, and the downsampling factor and filter kernel determine the measurement model, M. Here, we use a x constant filter, and 4 x 4 downsampling (i.e., measurements are averages over non-overlapping blocks). We compare two recent unsupervised methods Deep image Prior (DIP) [8] and DeepRED 20 DIP chooses random input vector, and adjusts the weights of CNN to minimize the mean square error between the output and the corrupted image. Regularization by denoising (RED) is recent successful method closely related to P&P [15]. DeepRED 20 combines DIP and RED,
obtaining better performance than either method alone.
Inspection of results on two example images demonstrates that our method produces results that are sharper with less noticeable artifacts (Fig. 33). Despite this, the PSNR and SSIM values are slightly worse than both DIP and DeepRED (Table Ib. These can be improved by averaging over realizations
(last column of Table producing superior PSNR and SSIM values the expense of some blurring (last column of Fig. This is expected: the algorithm produces high- gigh-probbbility samples of the prior subject to the measurement constraint, but least-squares optimal solution is the mean of the posterior distribution. If the samples are drawn from curved manifold, their average (a convex combination those points) will lie off the manifold. Finally, note that our method is more than two orders of magnitude faster than either DIP or DeepRED (bottom row, Table I
7

---

Table 1: Spatial super-resolution performance over Set5 (top 2 rows) and Set14 (second 2 rows).
Values indicate YCbCr-PSNR (SSIM). Last row shows average S runtime on DGX GPU.
MM"= x
DIP
DeepRED 20
Ours
Ours:vy
4:1
26.35 (0.826)
30.04 (0.902)
30.22 (0.904)
29.47 (0.894110.001)
31.20 (0.913)
8:1
23.02 (0.673)
24.98 (0.760)
24.95 (0.760)
25.07 (0.767110.003)
25.64 (0.792)
4:1
24.65 (0.765)
26,88 (0.815)
27.01 (0.817)
26.56=#0.09 (0.808=t0000)
27.14 (0.826)
8:1
22.06 (0.628)
23.33 (0.685)
23.34 (0.685)
23.32:+0.11 (0.681++0.002)
23.78 (0.703)
runtime (sec):
1,190
1,584
9
10 x
cropped
blurry
DIP g
DeepRED 20
Ours
Ours:avg
Figure Deblurring (spectral super-resolution) Measurements correspond to 5% of low frequencies. Original images are from Set5 (butterfly of size 256 x 256 and woman of size 224 336).
Deblurring (spectral super-resolution). The applications described above (and in Appendix B are based on partial measurements in the pixel domain. Here, we consider a blurring operator that retains set of low-frequency coefficient in the Fourier domain, discarding the rest. This is equivalent to convolving the image with a sine kernel. this case, M consists of the preserved low-frequency columns of the discrete Fourier transform, and M M is the blurred version of x. Example images are shown in Fig. 3 and numerical comparisons are shown in Table 2 Our method produces strong results, both perceptually and numerically.
Compressive sensing. Compressive sensing 56 aims recover signals from a small number of (typically, random) linear measurements. In brief, one acquires measurements with a sensing matrix containing set of << N random orthogonal axes, and solves the inverse problem by assuming sparse prior. Photographic images are not truly sparse in any fixed linear basis, but they can be reasonably approximated by low-dimensional subsets of Fourier or wavelet basis functions, and compressive sensing results are typically demonstrated using one of these. The manifold prior embedded within our CNN denoiser corresponds to nonlinear form of sparsity, and our stochastic coarse-to-fine ascent algorithm can be used to recover an image from the measured linear projections onto the random basis. We compare to four other methods. TVAL3 [53] an optimization algorithm using total variation regularization, ISTA-Net [54] is block-based supervised CNN method trained to reconstruct images from measurements obtained from a single pre-specified measurement matrix. BNN (55) is unsupervised Bayesian method for solving compressive sensing problems. DIP g was previously described. Fig. 55 results for two example images, and Table 3 summarizes numerical performance. All values taken from [55] except for ISTA-Net which were obtained by running the open-source code. Our method generally outperforms all other methods, even those that specialized for Compressive Sensing (ISTA-NET, BNN).
Table 2: Spectral super-resolution (deblurring) performance over Set5, in YCbCr-PSNR (SSIM).
Ratio
MM3r x
DIP g
DeepRED 20
Ours
Ours:avg
10%
30.2 (0.91)
32.54 (0.93)
32.63 (0.93)
31.82++0.08 (0.93+00.00))
32.78 (0.94)
5%
27.77 (0.85)
29.88 (0.89)
29.91 (0.89)
29.22-t0.14 (0.89+00.002)
30.07 (0.90)
8

---

Table Compressive sensing performance over Set68 [51].
Values indicate PSNR (SSIM).
Ratio
TVAL3 66
ISTA-Net 5d
DIP g
BNN 65
Ours:ttt
Ours:vv
25%
26.48 (0.77)
29.07 (0.84)
27.78 (0.80)
28.63 (0.84)
29.16110.033 (0.88_10.001)
29.74(0.89)
10%
22.49 (0.58)
25.23 (0.69)
24.82 (0.69)
25.24 (0.71)
25.47 (0.78=0.001)
25,84 (0.80)
4%
19.10 (0.42)
22.02 (0.54)
22.51 (0.58)
22.52 (0.58)
22.07=00.55 (0.68=0.002)
22.29 (0.69)
original
measured
ISTA-Net -Net
54
Ours
Figure 5: Compressive sensing. Measurement matrix M contains random orthogonal unit vectors, whose number equal 10% of the number of image pixels (300 300). Third column: images recovered using ISTA-Net (trained with supervision for a particular measurement matrix). Fourth column: Our method, which again is seen exhibit fewer artifacts.
4 Related work
Our method is conceptually similar to P&P[44 in that it uses denoiser solve linear inverse problems. But it differs a number of important ways: (1) direct relationship between prior and denoiser mapping. The rationale for using denoiser
regularizer in P&P frameworks arises from interpreting the proximal operator of the regularizer as a MAP solution of a denoising problem, providing an indirect connection prior; We derive our method from Miyasawa's 's relationship between an MMSE denoiser and the prior, which is exact and explicit, and makes the algorithm interpretable. This connection has also been made 2 2 (2) sampling vs. MAP estimation. We obtain a high probability image from the implicit prior that is consistent with a lincar constraint. Our solution is stochastic, and does not minimize MSE, but has high perceptual quality. RED and other P&P methods are derived as MAP solutions, and although this is not equivalent to minimizing MSE (maximizing PSNR), the results generally have better PSNR than our sampling results, but are visually more blurred (see Figs. and 4 : (3) automatic step-size selection and convergence. selection that are critical for convergence (as discussed extensively in (15)), ). Our algorithm adjusts step-size automatically using the blind denoiser, with only two primary haper-parameters (hg and B), and convergence robust to choices of these (Fig. D.
P&P, which uses ADMM for optimization, relies on
Our method is also closely related to recent work that uses Score Matching [10] to draw samples from an implicit prior. This line of work is rooted in the connection between Denoising Autoencoders and Score Matching, first described in [32) Denoising autoencoders learn the gradient of log density,
and their connection to an underlying data manifold has been explored in 22 43 B1 More recently,
Refs. 44 26 trained a neural network to directly estimate energy (the negative log prior) and interpreted its gradient as the Score of the data. Finally, in a breakthrough paper that partially inspired our approach, Song and Ermon 29 trained a sequence of denoisers with decreasing levels of noise and used Langevin dynamics to sample from the underlying probability distributions in succession. Our work differs from these in several important ways: (1) Direct derivation. Our method based on Miyasawa's is explicit relationship between the denoiser mapping and implicit density [9, which can

---

L.3). The relationship of this result to Score Matching M [2) and the relationship between Score Matching and Denoising Autoencoders [32 are both significantly more nuanced and complex. (2) Automatic stepsize adjustment and efficiency. The algorithm presented in [29 uses a discrete sequence of denoisers, each trained for a single noise level, and embedding a distribution of noisy images with that specific noise level. Langevin dynamics are used to sample from each of these distributions in succession. Langevin sampling guarantees (asymptotically) convergence to each successive distribution, although these intermediate samples are used only for initialization of the subsequent stage. In addition to the Langevin sampling itself, the method requires choices of the schedule of noise levels (standard deviations, and number of iterations used at each level).
be proven with few lines of math (Section
In contrast, our formulation relies on a single universal blind denoiser which implicitly embeds an infinite family of distributions of noisy images (corresponding to continuous range of noise levels). The gradient provided by the denoiser is not used for sampling, but for coarse-to-fine ascent of the noisy distribution, removing a fraction of the noise at each step. The universal denoiser adapts to this updated implicit noisy density, both in terms of gradient direction and magnitude. We use the denoiser magnitude as an estimate of the implicit noise level 6 and this is used to control the amplitude of injected noise. Note that this differs markedly from the control of noise injection in the Langevin method, which is of constant magnitude in each stage. Under assumptions about the denoiser (which hold empirically), our method converges rapidly to mode of the prior and is robust to parameter choices [T] The continuous maximization of probability results in substantial gains in efficiency. For comparison, Song&Ermon (29 report 1000 iteration to synthesize 32 x 32 image, whereas we synthesize 40 x 40 images in roughly 35 iteration (for 8 = 1). (3) Image synthesis vs. universal solver for linear linear inverse problems. Finally, we' ve focused the of our method
inverse problems, whereas Score Matching methods have generally been focused on unconditional image synthesis.
5 Discussion
We've described a framework for using the prior embedded in a denoiser to solve inverse problems. Specifically, we developed a stochastic coarse-to-fine gradient ascent algorithm that uses the denoiser to draw high-probability samples from its implicit prior, and constrained variant that can be used to solve any deterministic linear inverse problem. To demonstrate the generality of our algorithm, we showed solving five inverse problems using the same algorithm and parameters, without any additional training. The derivation is based on a simple and direct expression relating denoising to priors, and few basic empirical facts about universal CNN denoisers. Finally, we empirically demonstrated its efficiency in Fig. and Table ! is worth noting that the assumed use of Gaussian additive noise and MSE objective in training the denoiser is necessary only justify the use of Miyasawa's 's expression (Eq. G). but does not impose any such restrictions on use of the trained denoiser for sampling from the implicit prior solving inverse problems. Denoisers be trained supervised on unlimited amounts of unlabeled data, and as such, our method extends the power of supervised learning much broader set of problems, with no additional training.
As discussed following Eq. 5 the control step sizes and noise injection in our algorithm is chosen by assuming that the denoiser residual exactly cancels the noise. We showed empirically (Fig. D that this idcalization does not disrupt convergence for BF-CNN, but we cannot guarantee this would hold for all other denoisers. More generally, although the method can be used with any universal least-squares denoiser designed or trained to remove Gaussian noise, the complexity of the embedded prior, and thus the quality of our linear inverse results, relies heavily the expressive power of I the denoiser (BF-CNN in our case), as well as the diversity of the training set. Finally, the current algorithm is designed to solve deterministic linear inverse problems. We are currently exploring extensions to inverse problems based on stochastic or nonlinear measurements.
Acknowledgments and Disclosure of Funding
We gratefully acknowledge financial support from the Simons Foundation, the Howard Hughes Medical Institute, and NSF NRT HDR Award 1922658 to the Center for Data Science at NYU. High performance computing resources were provided by the NYU HPC center and the Flatiron Institute of the Simons Foundation.
10

---

References
[1] Donoho. Denoising by soft-thresholding. IEEE Trans Info Theory, 43:613-627, 1995.
[2] P Simoncelli and E H Adelson. Noise removal via Bayesian wavelet coring. In Proc 3rd
IEEE Int'l. I, Conf. on Image Processing (ICIP), pages 379-382, 1996.
[3] P Moulin and J Liu. Analysis of multiresolution image denoising schemes using a generalized Gaussian and complexity priors. IEEE Trans Info Theory, 45:909-919 1999.
[4] Hyvärinen. Sparse code shrinkage: Denoising of nonGaussia data by maximum likelihood estimation. Neural Computation, 11(7):139-1776. 1999.
[5] J. Romberg, H. Choi, and R. Baraniuk. Bayesian tree-structured image modeling using wavelet- domain hidden Markov models. IEEE Trans Image Proc, 10(7), July 2001.
[6] L Şendur and I W Selesnick. Bivariate shrinkage functions for wavelet-based denoising exploiting interscale dependency. IEEE Trans Signal Proc, 50(1 1):2744 2756, November 2002.
[7] J Portilla, Strela, M Wainwright, and E Simoncelli. Image denoising using scale mixtures of Gaussians in the wavelet domain. IEEE Trans Image Proc, 1331))3331351,, Nov 2003.
[8] D Ulyanov, A Vedaldi, and V Lempitsky. Deep image prior. Proc Int'l J Computer Vision, pages 1867-1888, April 2020.
[9] K Miyasawa. An empirical Bayes estimator of the mean of a normal population. Bull. Inst. Internat. Statist., 38: 181-188 1961.
[10] Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4). 2005.
[11] M Raphan and E Simoncelli. Least squares estimation without priors or supervision. Neural Computation, 23(2):374 420, Feb 2011.
[12] E. T. Rechorst and P. Schniter. Regularization by denoising: Clarifications and new interpreta- tions. IEEE Transactions on Computational Imaging, 5(1):52-67, 2019.
[13] Zahra Kadkhodaie and Eero P Simoncelli. Solving linear inverse problems using the prior implicit in a denoiser. arXiv preprint arXiv:2007. 13640, July 2020.
[14] Singanallur V Venkatakrishhan, Charles A Bouman, and Brendt Wohlberg. Plug-and-play priors for model based reconstruction. In 2013 IEEE Global Conference on Signal and Information Processing, pages 945-948. IEEE, 2013.
[15] Yaniy Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denoising (RED). CoRR, abs/1 11221611.020000 1.02862, 2017.
[16] Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. Learning deep enn denoiser prior for image restoration. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), volume 2, 2017.
[17] Ulugbek S Kamilov, Hassan Mansour, and Brendt Wohlberg. A plug-and-play priors approach for solving nonlinear imaging inverse problems. IEEE Signal Processing Letters, 14(12):1872- 1876, 2017.
[18] T. Meinhardt, M. Moeller, C. Hazirbas, and D. Cremers. Learning proximal operators: Using denoising networks for regularizing inverse imaging problems. In 2017 IEEE International Conference Computer Vision (ICCV), pages 1799-1808, 2017.
[19] S. H. Chan, X. Wang, and O. A. Elgendy. Plug-and- admm for image restoration: Fixed- point convergence and applications. IEEE Transactions on Computational Imaging, 3(1):84-98, 2017.
[20] Gary Mataev, Peyman Milanfar, and Michael Elad. Deepred: Deep image prior powered by red. In Proceedings of the IEFE International Conference on Computer Vision Workshops, pages 0-0, 2019.
11

---

[21] Afonso M Teodoro, José M Bioucas-Diss and Mário AT Figueiredo. Image restoration and reconstruction using targeted plug-and-play priors. IEEE Transactions on Computational Imaging, 5(4):675-666, 2019.
[22] Y. Sun, B. Wohlberg, and U. S. Kamilov. An online plug-and-play algorithm for regularized image reconstruction. IEEE Transactions on Computational Imaging. 5(3):395-400) 2019.
[23] Xiaojian Xu, Yu Sun, Jiaming Liut, Brendt Wohlberg, and Ulugbek S. Kamilov. Provable convergence of plug-and-play priors with MMSE denoisers. arXiv preprini arXiv:2005.076635/, May 2020.
[24] Rèmi Laumont, Valentin De Bortoli, Andrés Almansa, Julie Delon, Alain Durmus, and Marcelo Pereyra. Bayesian imaging using plug & play priors: When Langevin meets Tweedie. arXiv preprint arXiv:22033.047554, March 2021.
[25] Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data- generating distribution. The Journal of Machine Learning Research, 15()5335333353,, 2014.
[26] Saeed Saremi, Bernhard Schölkopf, Arash Mehrjou, and Aapo Hyvärinen. Deep energy estimator networks. ArXiv e-prints (arXiv.org), 1805.08306, May 2018.
[27] Bichuan Guo, Yuxing Han, and Jiangtao Wen. Agem: Solving linear inverse problems via deep priors and sampling. Advances in Neural Information Processing Systems, 32:547-558, 2019.
[28] Zengyi Li, Yubei Chen, and Friedrich T. Sommer. Learning energy-based models in high- dimensional spaces with multi-scale denoising score matching, 2019.
[29] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems 32, pages 11918-11930. Curran Associates, Inc., 2019.
[30] Siavash A Bigdeli, Geng Lin, Tiziano Portenier, L Andrea Dunbar, and Matthias Zwicker. Learn- ing generative models using denoising density estimators. arXiv preprint arXiv:2001.02728, 2020.
[31] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096-1103, 2008
Manzagol. Extracting and
[32] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7): 2011.
[33] JJ Koenderink. The structure of images. Biological Cybernetics, 50:363-370, 1984.
[34] T Lindeberg. Scale-space theory: A basic tool for analysing structures at different scales. Journal of Applied Statistics, 21(2):224-270, 1994.
[35] Y. Hel-Or and D. Shaked. A discriminative approach for wavelet shrinkage denoising. IEEE Trans. Image Processing, 17(4), April 2008
[36] Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations over learned dictionaries. JEEE Trans. on Image processing, 15(12):736-3745 2006.
[37] Viren Jain and Sebastian Seung. Natural image denoising with convolutional networks. In Advances in neural information processing systems, pages 769-776, 2009.
[38] Harold C Burger, Christian Schuler, and Stefan Harmeling. Image denoising: Can plain neural networks compete with BM3D? In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pages 2392-2399, IEEE, 2012.
[39] Kai Zhang. Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Trans. Image Processing, 26(7):3142-3153 2017.
12

---

[40] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pages 4700-4708, 2017.
[41] Yunjin Chen and Thomas Pock. Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration. IEEE Trans. Patt. Analysis and Machine Intelligence, 39(6): 1272222262222 2017.
[42] S Mohan ; Z Kadkhodaie*, E Simoncelli, and C Granda. Robust and interpretable blind image denoising via bias-free convolutional neural networks. In Inl'l. Conf. on Learning Representations (ICLR), Addis Ababa, Ethiopia, April 2020.
[43] Bengio, Yao, G Alain, and P Vincent. Generalized denoising auto-encoders as generative models. In Adv. Neural Information Processing Systems (NIPS* pages 907. MIT Press, 2013.
[44] Saeed Saremi and Aapo Hyvarinen. Neural empirical Bayes. Journal of Machine Learning Research, 20:1-23, 2019.
[45] Zengyi Li, Yubei Chen, and Friedrich T Sommer. Annealed denoising score matching: Learning energy-based models in high- high-dimensional spaces. arXiv preprint 1rXiv:1910.0776,, 2019.
[46] H Robbins. An empirical Bayes approach to statistics. Proc. Third Berkley Symposium on Mathemalcal Statistics, 1:157-163, 1956.
[47] Lucas and T Kanade. An iterative image registration technique with an application to stereo vision. In Proc. 7th Int't l Joint Conf. on Artificial Intelligence, pages 674 679, Vancouver, 1981.
[48] S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi. Optimization by simulated annealing. Science, 22014598)x611-680, 1983.
[49] S. Geman and D. Geman. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Trans Pattern Analysis and Machine Intelligence, 6(6):711-741, 1984.
[50] A. Blake and A. Zisserman. Visual Reconstruction. MIT Press, 1987.
[51] D. Martin, C. Fowlkes, D. Tal, and Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int'l Conf. Computer Vision, volume pages 416-423, July 2001.
[52] Yann LeCun and Corinna Cortes. http://yammleccm.co//eb/////m//
MNIST
handwritten
digit database.
2010.
[53] Chengbo Li, Wotao Yin, Hong Jiang, and Yin Zhang. An efficient augmented lagrangian method with applications to total variation minimization. Computational Optimization and Applications, 56(3):507-530, 2013.
[54] Jian Zhang and Bernard Ghanem. Ista-net: Interpretable optimization-inspired deep network for image compressive sensing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1828-1837, 2018.
[55] Tongyao Pang, Yuhui Quan, and Hui Ji. Self-supervised bayesian deep learning for image recovery with applications to compressive sensing. 2020.
[56] Emmanuel J Candès, Justin K Romberg, and Terence Tao. Stable signal recovery from in- complete and inaccurate measurements. Communications on Pure and Applied Mathematics, 1207-1327,2 2006.
[57] D. L. Donoho. Compressed sensing. IEEE Trans Info Theory, 52(4): 2006.
[58] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecmm www.n. cot.emm.com/////nnniiss... 2, 2010.
13


Reviews:
[{"review_id": "q07hpzwJ_id", "reviewer": "Reviewer_9hAm", "summary": "The paper derives an algorithm for solving linear inverse problems using the prior that is implicit in an end-to-end denoiser in the form of a CNN.\n\nConsider a denoising algorithm in form of a CNN that takes a noisy observation $y = x+z$ as input and yields an estimate of the signal $x$. CNNs trained end-to-end yield state-of-the-art denoising performance. After training, the image/signal prior is implicit in the CNN. The paper notes that the least-squares-estimate of the signal given a prior $p(x)$ is \n\n$$\n\\hat x(y) = y + \\sigma^2 \\nabla_y \\log p(y),\n$$\n\nwhere $p(y)$ is the noisy observation density. Based on this relation, the paper proposes a method to obtain a high-probability image (sample) from the prior implicit in a denoiser. The idea is as follows: We can denoise a given image by evaluating $\\hat x(y)$, and by computing the residual $f(y) = \\hat x(y) - y$ we obtain an image that is proportional to $\\nabla_y \\log p(y)$. Based on this insight, the paper proposes to sample from the prior implicit in a denoiser by running stochastic gradient ascent through iterating $y_{t+1} = y_t + h_t f(y_t) + \\gamma_t z_t$, where $z_t$ is Gaussian noise, and $h_t$ and $\\gamma_t$ are tuning parameters. The paper demonstrates (Fig. 2) that this method can yield realistic samples of low-dimensional images. \n\nNext, the paper then modifies this approach to solve linear inverse problems by obtaining a high-probability sample based on partial linear measurements of a signal/image. The paper evaluates the method on standard inverse problems, including super-resolution, deblurring, and compressive sensing. The paper shows that the method works well, both in numbers and by providing example images. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper introduces an original way to solve linear inverse problems and demonstrates with simulations that it works well, both computationally and in terms of imaging quality. The idea is well explained, the paper is well written, and the experiments support the claims. I would be happy to see the paper at the conference.\n\nHere are a few comments/experiments that would make the paper even stronger:\n- The paper uses as an denoiser the bias-free CNN from [36], and writes it obtains similar results with other-bias free denoisers. I would be curious to also see results for other standard CNN-denoisers, in particular for U-net, because U-net or similar encoder-decoder architectures work very well for denoising and other inverse problem. If the proposed approach does not work well with U-net, that wouldn't make the paper any less interesting of course. I would also be interesting to show results for standard CNN denoisers that are not bias-free, and potentially add a brief discussion whether bias-freeness ist important or not for this method to work. \n- For the numbers in Table 1 ``ours - avg'', it would be great to also give confidence intervals or a standard deviation, to get intuition on the variation across samples.\n- Another interesting baseline that is also easy to train for table 3 is a U-net trained end-to-end for compressive sensing. Runtimes for CS would also be great to add.\n- Can the method also give realistic samples of high-resolution images? I didn't understand from the paper what the resolution of the images in figure 4 or 5 is.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Overall the paper adequately discusses limitations. One interesting comparison to add is U-net-based reconstruction in Table 3 (so U-net applied to M^Ty, where y is the measurement), along with corresponding computational costs. I expect that this will show that U-net-based reconstruction is at least on par in terms of image quality, while being faster. ", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "FpBuYOPRkz", "reviewer": "Reviewer_N12S", "summary": "The authors proposed a stochastic coarse-to-fine gradient ascent method for drawing high-probability samples from the implicit prior to denoise the noisy image and generalizes it to more broad linear inverse problems.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea is quite interesting, i.e., building a plug-and-play denoiser to inverse problems.\nHowever, the proposed approach is not very clearly explained. There are many incorrect or unclear claims, which are detailed in the below box.\nThe paper is also not very well organized, e.g., methodology and experiments are mixed in one section. The authors' contribution in methodology is not clearly separated from existing works.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Paper organization: \nThe proposed method and experimental results are mixed in Section 3, which does not help readers understand the proposed solution better. It is more readable to separate the methodology and experiments into two sections and clearly present the methodology contribution. \n\nThere is only a subsection in Section 2. Also, in Section 2.1, the title Image synthesis is not logically connected with the denoiser, which should be explicitly explained. Also, this section is about experiments and should be merged with Section 3.2.\n\nTechnical issues \nLine 173-174: This part is not clear since \\bar{M} is not clearly defined. How y is decomposed as (y^c, y^u) should be explicitly explained using the property of orthogonal complement. Also, why y^c can be omitted conditional on x^c in p(y^u|y^c, x^c)? \nLast but not least, the dimension of y and x is very likely not the same in super-resolution, inpainting, compressive sensing, etc. Let’s say, x in R^n and y in R^m, thus m<n. If M^T can be left multiplied to x, how can it be also left multiplied to y as well?\n\nLine 74: The authors’ claim that (2) is least square estimation is not correct. The conditional mean of the posterior is the MMSE estimator.\n\nIn Fig 6, the notation of denoiser D is not consistent with the text and Algo 1, which can easily confuse readers.\n\nNote that the proposed method requires Monte Carlo sampling, which might be quite time-consuming. The computation complexity needs to be analyzed and in experiments, the running time for different methods should be compared.\n\nMinor issues:\n\nLine 78: Solves—> solves it.\n\nIn Fig 3, the reference for DeepRED is not correct.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "AitVFp3zrKG", "reviewer": "Reviewer_tt2X", "summary": "Classical result by Miyasawa [25]  shows that the gradient of the log of the noisy signal density  can be expressed in terms of least-squares optimal denoisers for additive Gaussian noise.  Using this result, the authors develop an algorithm which uses  stochastic gradient ascent  to sample from the implicit prior of the denoiser, where the gradients  are computed using the residual of a trained bias-free universal CNN denoiser.  The algorithm includes noise injection which allows stochastic sampling of the manifold. They subsequently modify this algorithm to account for constraints from noise-less linear measurement model to solve linear inverse problems. The authors perform different linear image reconstruction tasks including spatial super-resolution, deblurring, compressive sensing, inpainting and reconstruction from random missing pixels. Experimentally, they demonstrate faster and visually better reconstructions as compared to the considered baselines.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is nicely motivated, well-written and clear. The overall topic is highly relevant and the numerical results are encouraging. Yet, despite the detailed explanation given in lines 255 to 272, the difference to [20] did not become perfectly clear to me from an algorithmic perspective and thus raises questions about the originality of the research in my oppinion. The authors state that their method is not based on Langevin dynamics. The update equation for y in eq.(4) is, however, identical to the update equation of x in [20] except for the way the scaling factors / step sizes are computed. Both approaches update the variable with the help of (differently weighted) combinations of the denoiser residual and random noise. Thus, I believe it is crucial to discuss the difference of the proposed approach with respect to Langevin dynamics in more detail. In particular, the Langevin dynamics of [20] could be used / compared to in the numerical results. As the paper is validated empirically (with no convergence proof of the proposed scheme), I believe it is crucial to describe why the two approaches are different despite their very similar appearance of the algorithm. \n\nTo better describe the relation of the proposed work in the overall context of reserach on using denoising priors, I am attaching some related works below, which could be of interest. (Some of them are preprints only and of course to not have to be considered, yet could be of general interest to the authors). \n\nTypos:\n - line 118: maniofld >> manifold\n - line 126- 127: residual of the denoising function proportional to the gradient of p(y) >>  residual of the denoising function proportional to the gradient of log p(y) \n\n\n\n[a] Guo, Bichuan, Yuxing Han, and Jiangtao Wen. \"Agem: Solving linear inverse problems via deep priors and sampling.\" Advances in Neural Information Processing Systems 32 (2019).\n\n[b] Ramzi, Zaccharie, Benjamin Rémy, Francois Lanusse, Jean-Luc Starck, and Philippe Ciuciu. \"Denoising Score-Matching for Uncertainty Quantification in Inverse Problems.\" In NeurIPS 2020-34th Conference on Neural Information Processing Systems/Workshop on Deep Learning and Inverse Problems. 2020\n\n[c] Regev Cohen, Michael Elad, and Peyman Milanfar. \"Regularization by denoising via fixed-point projection (red-pro).\" (2020)\n\n[d] Kawar, Bahjat, Gregory Vaksman, and Michael Elad. \"SNIPS: Solving Noisy Inverse Problems Stochastically.\" arXiv preprint arXiv:2105.14951 (2021)\n\n[e] Saharia, Chitwan, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. \"Image super-resolution via iterative refinement.\" arXiv preprint arXiv:2104.07636 (2021).\n\n[f] Laumont, Rémi, Valentin De Bortoli, Andrés Almansa, Julie Delon, Alain Durmus, and Marcelo Pereyra. \"Bayesian imaging using Plug & Play priors: when Langevin meets Tweedie.\" arXiv preprint arXiv:2103.04715 (2021)\n\n-----------------------------\nI would like to thank the authors for their explanation which helped me understand the difference to Langevin dynamics better. Due to the large algorithmic similarity to Langevin dynamics I would have still appreciated a comparison to (or at least a small test on) using an algorithm like [20] in the same fashion the authors are solving inverse problems based on the idea of algorithm 1. This would contribute to my understanding whether it is the use of stochasticity or the specific way it is introduced (i.e. controlling the amount of noise via the magnitude of the residual predicted by the denoiser) that leads to the superior results. Although it at least seems like [20] could easily be adapted to solve inverse problems the same way as proposed in this paper, I understand that [20] addressed generative tasks only (counting inpainting as a generative task). I will therefore raise my score to 7. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are adequately discussed. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "3B06cJO3i2Z", "reviewer": "Reviewer_TGbx", "summary": "This paper proposed a score-based method that solves deterministic linear inverse problems by drawing samples from the denoising priors. By iteratively maximizing the score $\\textsf{log}p(y)$ using Langevin process, high-probability samples can be drawn from the distribution $p(x)$ embedded in the denoiser. However, in order to perform this, $\\nabla_y\\textsf{log}p(y)$ needs to be effectively computed. By leveraging the classic Miyasawa’s equation for MSE denoisers, the authors are able to use a residual CNN denoiser trained for minimizing MSE to approximate the term $\\textsf{CNN}(y) = \\hat{x}(y)-y = \\nabla_y\\textsf{log}p(y)$. \nThe authors also extend their method to linear deterministic linear inverse problems by expressing $\\textsf{log}p(y)$ into two orthogonal components accounting for noise density and constraint, respectively.\n\nThe method is related to the recent Plug-and-play (RED) and Regularization-by-denoising (RED) frameworks which take an optimization-centric view of using denoisers. However, it differs from the former in the sense that it explicitly relates the denoiser with the prior distribution. In the experiments, one self-supervised neural network and one RED algorithm are used as baselines, achieving inferior performance compared with the proposed method in single image super-resolution, denoising, deblurring, and compressive sensing. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:** This work is novel in relating the MSE denoisers to priors in an explicit way by using the classic result. The difference between PnP/RED and the proposed method is clearly stated. However, I found a concurrent paper [1] that seems to propose a similar method. I believe a discussion between your and their methods can avoid future confusion. Moreover, investigations on the relationship between denoisers and explicit regularizers have also been conducted for MMSE denoisers within the PnP framework [2].\n\n[1] Laumont et al. 'Bayesian imaging using Plug & Play priors: when Langevin meets Tweedie\\\n[2] Xu. et al. Provable Convergence of Plug-and-Play Priors With MMSE Denoisers\n\n**Quality:** This paper conducts adequate experiments to support their claims. The results are technically sound and convincible. For questions please see my comments below. Also, the limitations of the proposed method are well-discussed.\n\n**Clarity:** Since I am not very familiar with the score-match field, the paper is a little bit difficult for me to follow. A supplement with some introductory details (e.g. derivation of Langevin process) will be welcome for readers like me.\n\n**Significance:** This paper provides interesting insights from a Bayesian perspective, which can be useful for the theoretical studies of PnP/RED. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. While reading the manuscript, I encountered some confusions that need authors' explanation: 1) Throughout the paper, it seems to me that the final output is $\\hat{x}(y)$. However, the algorithm summarized in Alg. 1 seeks to update $y$, which I assume to be related to the noisy observation. Is there a hidden step $\\hat{x}(y)=f(y)+y$ for dumping the final output? or am I misinterpreting the variables? 2) Sampling $y$ seems quite unusual if my interpretation of $y_t$ is correct. In [1], I saw their formulations are for $x$. 3) I did not quite see why from the equation above eq 7 we can have eq 7. Could the authors provide some intuitive explanations? Derivations are also welcome.\n\n2. A discussion between your method and [1] is needed.\n\n3. Is it possible to construct a simple toy problem such that the oracle MMSE denoiser is reachable. In this way, we can compare CNN with this oracle denoiser, which may bridge the gap between theory and experiment. Can the experimental setup (Compressive Sensing with Bernoulli-Gaussian distribution) in [2] directly reusable to yours? \n\n4. In experiments, deepRED is treated as the PnP baseline. Although RED and PnP are closely related, the exact equivalence between the two is still an open question. I suggest including one exact PnP baseline (PnP-ISTA or PnP-ADMM) in the experiments.\n\n5. Minor: In line 245 Section 4, the sentence implies that PnP is coupled with ADMM. However, this is a long-standing misunderstanding of the PnP framework. Many works have shown that PnP is compatible with gradient-based [3], primal-dual [4], and message passing algorithms [5]. 2) \n\n\n[3] Kamilov et al. A Plug-and-Play Priors Approach for Solving Nonlinear Imaging Inverse Problems\\\n[4] Ono et al. Primal-Dual Plug-and-Play Image Restoration\\\n[5] Metzler et al. Learned D-AMP: Principled Neural Network-Based Compressive Image Recovery\n\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours for writing response. 3 hours for reading and evaluation"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides an original way to solve linear inverse problems when given a denoiser.  The method uses forward calls to the denoiser in a way that leverages the signal prior implicit to it.  The paper establishes strong quantitative and qualitative recovery performance.  In the camera ready version of the paper, the authors should make the modifications discussed with the reviewers.  In particular, the authors should clarify the relationship between the proposed method and Langevin Dynamics.  The authors say the method draws "high probability samples"; the authors should clarify what these terms mean and whether or not it is the same as directly sampling from the relevant distribution.  
Decision: Accept (Poster)