Stochastic Solutions for Linear Inverse Problems using the Prior Implicit in a Denoiser

1 Introduction . Many problems in image processing and computer vision rely , explicitly or implicitly , on prior probability models . Describing the full density of natural images is a daunting problem , given the high dimensionality of the signal space . Traditionally , models have been developed by combining assumed symmetry properties ( e.g. , translation-invariance , dilation-invariance ) , with simple parametric forms ( e.g. , Gaussian , exponential , Gaussian mixtures ) , often within pre-specified transformed coordinate systems ( e.g. , Fourier transform , multi-scale wavelets ) . While these models have led to steady advances in problems such as denoising ( e.g. , [ 1–7 ] ) , they are too simplistic to generate complex features that occur in our visual world , or to solve more demanding statistical inference problems . In recent years , nearly all problems in image processing and computer vision have been revolutionalized by the use of deep Convolutional Neural Networks ( CNNs ) . These networks are generally optimized in supervised fashion to obtain a direct input-output mapping for a specific task . This approach does not explicitly rely on a known prior , and offers performance far superior to prior-based methods . The downside , however , is that the learned mappings are intertwined with the task for which they are optimized , and in most cases require training a separate network for each new application . In contrast , a prior probability model can provide a universal substrate for solving inference 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . problems . The superior performance of CNNs suggests that they embed , implicitly , sophisticated prior knowledge of images . These implicit priors arise from a combination of the distribution of the training data , the architecture of the network [ 8 ] , regularization terms included in the optimization objective , and the optimization algorithm . Here , our goal is to extract the implicit prior from a network trained for denoising and use it to solve other inverse problems without further training . We choose denoising not because it is of particular importance or interest , but because we can make the relationship between mapping of a denoiser and a prior explicit . We combine the advantages of prior-based and mapping-based approaches , deriving a general algorithm for solving linear inverse problems using the prior implicit in a trained denoiser . We start with a result from classical statistics [ 9 ] that states that a denoiser that aims to minimize squared error of images corrupted by additive Gaussian noise may be interpreted as computing the gradient of the log of the density of noisy images . This result is related to Score Matching [ 10 ] , but provides a more direct relationship between least-squares optimal denoising and the embedded prior [ 11 , 12 ] . We develop a stochastic ascent algorithm that uses this denoiser-estimated gradient to draw high-probability samples from the embedded prior . Importantly , we use a blind denoiser that can handle noise contamination of unknown amplitude , which provides a means of adaptively controlling the gradient step sizes and the amplitude of injected noise , enabling robust and efficient convergence . We then modify the algorithm to incorporate constraints arising from any deterministic linear measurement of an image . The resulting procedure generates high-probability samples from the prior conditioned on the measurements , thus providing a general stochastic solution for any deterministic linear inverse problem . We demonstrate that our method produces visually high-quality results in recovering missing pixels , and state-of-the-art levels of unsupervised performance on superresolution , deblurring and compressive sensing.1 Earlier versions of this work were presented in [ 13 ] . This work is closely related to two lines of research . Nearly a decade ago , a strategy known as Plug-and-Play ( P & P ) was proposed for using a denoiser as a regularizer in solving other inverse problems [ 14 ] , and a number of recent extensions have used this concept to develop MAP solutions for linear inverse problems [ 15–22 , 12 ] . Generally , the objective is decoupled into data fidelity and regularization terms , introducing a slack variable for use in a proximal optimization algorithm ( e.g. , ADMM ) . The proximal operator of the regularization term is interpreted as the MAP solution of a denoising problem , and is replaced by a denoiser . Of particular relevance to our work , recent publications have proven convergence of such algorithms when used in conjunction with MMSE denoisers [ 23 , 24 ] . A parallel line of research has focused on the use of generative models based on Score Matching [ 25–30 ] . The connection between Score Matching [ 10 ] and denoising autoencoders [ 31 ] was first shown in [ 32 ] , by proving that the training criterion of a denoising autoencoder is equivalent to matching the score of the model and a Parzan density estimate of the data . Most recently , this idea has been used as the basis for an MCMC algorithm for sampling from the prior implicit in a CNN denoiser [ 29 ] . In Section 4 we elaborate on how these methods are related to our results . 1.1 Image priors , manifolds , and noisy observations . Digital photographic images lie in a high-dimensional space ( RN , where N is the number of pixels ) , and simple thought experiments suggest that they are concentrated on or near low-dimensional manifolds whose local coordinates represent continuous deformations and intensity variations . In contrast , images generated with random pixels are almost always feature and content free , and thus not considered to be part of this manifold . We can associate with this manifold a prior probability model , p ( x ) , by assuming that images within the manifold have constant or slowly-varying probability , while unnatural or distorted images ( which lie off the manifold ) have low or zero probability . Suppose we make a noisy observation of an image , y = x + z , where x 2 RN is the original image drawn from p ( x ) , and z ⇠ N ( 0 , 2IN ) is a sample of Gaussian white noise . The observation density p ( y ) is related to the prior p ( x ) via marginalization : p ( y ) = Z p ( y|x ) p ( x ) dx = Z g ( y x ) p ( x ) dx , ( 1 ) where g ( z ) is the Gaussian noise distribution . Equation ( 1 ) is in the form of a convolution , and thus p ( y ) is a Gaussian-blurred version of the signal prior , p ( x ) . Moreover , the family of observation 1A software implementation of the sampling and linear inverse algorithms is available at https : //github . com/LabForComputationalVision/universal_inverse_problem densities over different noise variances , p ( y ) , forms a Gaussian scale-space representation of the prior [ 33 , 34 ] , analogous to the temporal evolution of a diffusion process . 1.2 Least squares denoising and CNNs . Given a noisy observation , y , the minimum mean squared error ( MMSE ) estimate of the true signal is well known to be the conditional mean of the posterior density : x̂ ( y ) = Z xp ( x|y ) dx = Z x p ( y|x ) p ( x ) p ( y ) dx ( 2 ) The structure of the equation mirrors the traditional approach to the problem : one chooses a prior probability model , p ( x ) , combines it with a likelihood function describing the noisy measurement process , p ( y|x ) , and solves . Modern denoising solutions , on the other hand , are often based on supervised learning of a direct mapping from noisy to denoised images . One expresses the estimation function ( as opposed to the prior ) in parametric form , and sets the parameters by minimizing the denoising MSE over a large training set of example signals and their noise-corrupted counterparts [ 35– 38 ] . Current state-of-the-art denoising results using CNNs obtained with this supervised approach are far superior to results of previous methods [ 39–41 ] . Recent analysis of these networks demonstrates that when they are trained to handle a broad range of noise levels , they perform an approximate projection onto a low-dimensional subspace [ 42 ] . In our context , we interpret this subspace as a tangent hyperplane of the image manifold . 1.3 Exposing the implicit prior through Empirical Bayes estimation . Trained CNN denoisers contain detailed prior knowledge of image structure , but Eq . ( 2 ) suggests that it is embedded within a high-dimensional integral . How can we make use of this implicit prior ? Recent results have derived relationships between Score Matching density estimates and denoising , and have used these relationships to make use of implicit prior information [ 43 , 44 , 29 , 45 ] . Here , we exploit a more direct but less-known result from the literature on Empirical Bayesian estimation . The idea was introduced in [ 46 ] , extended to the case of Gaussian additive noise in [ 9 ] ( see also [ 12 ] ) , and generalized to many other measurement models [ 11 ] . In the case of additive Gaussian noise , one can rewrite the estimator of Eq . ( 2 ) as : x̂ ( y ) = y + 2ry log p ( y ) . ( 3 ) The proof is relatively straightforward . The gradient of the observation density of Eq . ( 1 ) is : ry p ( y ) = 1 2 Z ( x y ) g ( y x ) p ( x ) dx = 1 2 Z ( x y ) p ( y , x ) dx . Multiplying both sides by 2/p ( y ) and separating the right side into two terms gives : 2 ry p ( y ) p ( y ) = Z xp ( x|y ) dx Z yp ( x|y ) dx = x̂ ( y ) y . Rearranging terms and using the chain rule to compute the gradient of the log gives Eq . ( 3 ) . This remarkable result re-expresses the integral over the prior and likelihood of Eq . ( 2 ) in terms of a gradient . Note that 1 ) the relevant density is not the prior , p ( x ) , but the noisy observation density , p ( y ) ; 2 ) the gradient is computed on the log density ( the associated “ energy function ” ) ; and 3 ) the gradient adjustment is not iterative - the estimate is achieved in a single step , and holds for any noise level , . 2 Drawing high-probability samples from the implicit prior . Suppose we wish to draw a sample from the prior implicit in a denoiser . Equation ( 3 ) allows us to generate an image proportional to the gradient of log p ( y ) by computing the denoiser residual , f ( y ) = x̂ ( y ) y . Song and Ermon [ 29 ] developed a Markov chain Monte Carlo ( MCMC ) scheme , combining gradient steps derived from Score Matching and injected noise in a Langevin sampling algorithm to draw samples from a sequence of densities p ( y ) , while reducing in a sequence of discrete steps , each associated with an appropriately trained denoiser . In contrast , starting from a random initialization , y0 , we aim to find a high-probability image ( i.e. , an image from the manifold ) using a more direct and efficient stochastic gradient ascent procedure .

Summary: The paper derives an algorithm for solving linear inverse problems using the prior that is implicit in an end-to-end denoiser in the form of a CNN.  Consider a denoising algorithm in form of a CNN that takes a noisy observation $y = x+z$ as input and yields an estimate of the signal $x$. CNNs trained end-to-end yield state-of-the-art denoising performance. After training, the image/signal prior is implicit in the CNN. The paper notes that the least-squares-estimate of the signal given a prior $p(x)$ is   $$ \hat x(y) = y + \sigma^2 \nabla_y \log p(y), $$  where $p(y)$ is the noisy observation density. Based on this relation, the paper proposes a method to obtain a high-probability image (sample) from the prior implicit in a denoiser. The idea is as follows: We can denoise a given image by evaluating $\hat x(y)$, and by computing the residual $f(y) = \hat x(y) - y$ we obtain an image that is proportional to $\nabla_y \log p(y)$. Based on this insight, the paper proposes to sample from the prior implicit in a denoiser by running stochastic gradient ascent through iterating $y_{t+1} = y_t + h_t f(y_t) + \gamma_t z_t$, where $z_t$ is Gaussian noise, and $h_t$ and $\gamma_t$ are tuning parameters. The paper demonstrates (Fig. 2) that this method can yield realistic samples of low-dimensional images.   Next, the paper then modifies this approach to solve linear inverse problems by obtaining a high-probability sample based on partial linear measurements of a signal/image. The paper evaluates the method on standard inverse problems, including super-resolution, deblurring, and compressive sensing. The paper shows that the method works well, both in numbers and by providing example images.