Stochastic Solutions for Linear Inverse Problems using the Prior Implicit in a Denoiser

1 Introduction . Many problems in image processing and computer vision rely , explicitly or implicitly , on prior probability models . Describing the full density of natural images is a daunting problem , given the high dimensionality of the signal space . Traditionally , models have been developed by combining assumed symmetry properties ( e.g. , translation-invariance , dilation-invariance ) , with simple parametric forms ( e.g. , Gaussian , exponential , Gaussian mixtures ) , often within pre-specified transformed coordinate systems ( e.g. , Fourier transform , multi-scale wavelets ) . While these models have led to steady advances in problems such as denoising ( e.g. , [ 1–7 ] ) , they are too simplistic to generate complex features that occur in our visual world , or to solve more demanding statistical inference problems . In recent years , nearly all problems in image processing and computer vision have been revolutionalized by the use of deep Convolutional Neural Networks ( CNNs ) . These networks are generally optimized in supervised fashion to obtain a direct input-output mapping for a specific task . This approach does not explicitly rely on a known prior , and offers performance far superior to prior-based methods . The downside , however , is that the learned mappings are intertwined with the task for which they are optimized , and in most cases require training a separate network for each new application . In contrast , a prior probability model can provide a universal substrate for solving inference 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . problems . The superior performance of CNNs suggests that they embed , implicitly , sophisticated prior knowledge of images . These implicit priors arise from a combination of the distribution of the training data , the architecture of the network [ 8 ] , regularization terms included in the optimization objective , and the optimization algorithm . Here , our goal is to extract the implicit prior from a network trained for denoising and use it to solve other inverse problems without further training . We choose denoising not because it is of particular importance or interest , but because we can make the relationship between mapping of a denoiser and a prior explicit . We combine the advantages of prior-based and mapping-based approaches , deriving a general algorithm for solving linear inverse problems using the prior implicit in a trained denoiser . We start with a result from classical statistics [ 9 ] that states that a denoiser that aims to minimize squared error of images corrupted by additive Gaussian noise may be interpreted as computing the gradient of the log of the density of noisy images . This result is related to Score Matching [ 10 ] , but provides a more direct relationship between least-squares optimal denoising and the embedded prior [ 11 , 12 ] . We develop a stochastic ascent algorithm that uses this denoiser-estimated gradient to draw high-probability samples from the embedded prior . Importantly , we use a blind denoiser that can handle noise contamination of unknown amplitude , which provides a means of adaptively controlling the gradient step sizes and the amplitude of injected noise , enabling robust and efficient convergence . We then modify the algorithm to incorporate constraints arising from any deterministic linear measurement of an image . The resulting procedure generates high-probability samples from the prior conditioned on the measurements , thus providing a general stochastic solution for any deterministic linear inverse problem . We demonstrate that our method produces visually high-quality results in recovering missing pixels , and state-of-the-art levels of unsupervised performance on superresolution , deblurring and compressive sensing.1 Earlier versions of this work were presented in [ 13 ] . This work is closely related to two lines of research . Nearly a decade ago , a strategy known as Plug-and-Play ( P & P ) was proposed for using a denoiser as a regularizer in solving other inverse problems [ 14 ] , and a number of recent extensions have used this concept to develop MAP solutions for linear inverse problems [ 15–22 , 12 ] . Generally , the objective is decoupled into data fidelity and regularization terms , introducing a slack variable for use in a proximal optimization algorithm ( e.g. , ADMM ) . The proximal operator of the regularization term is interpreted as the MAP solution of a denoising problem , and is replaced by a denoiser . Of particular relevance to our work , recent publications have proven convergence of such algorithms when used in conjunction with MMSE denoisers [ 23 , 24 ] . A parallel line of research has focused on the use of generative models based on Score Matching [ 25–30 ] . The connection between Score Matching [ 10 ] and denoising autoencoders [ 31 ] was first shown in [ 32 ] , by proving that the training criterion of a denoising autoencoder is equivalent to matching the score of the model and a Parzan density estimate of the data . Most recently , this idea has been used as the basis for an MCMC algorithm for sampling from the prior implicit in a CNN denoiser [ 29 ] . In Section 4 we elaborate on how these methods are related to our results . 1.1 Image priors , manifolds , and noisy observations . Digital photographic images lie in a high-dimensional space ( RN , where N is the number of pixels ) , and simple thought experiments suggest that they are concentrated on or near low-dimensional manifolds whose local coordinates represent continuous deformations and intensity variations . In contrast , images generated with random pixels are almost always feature and content free , and thus not considered to be part of this manifold . We can associate with this manifold a prior probability model , p ( x ) , by assuming that images within the manifold have constant or slowly-varying probability , while unnatural or distorted images ( which lie off the manifold ) have low or zero probability . Suppose we make a noisy observation of an image , y = x + z , where x 2 RN is the original image drawn from p ( x ) , and z ⇠ N ( 0 , 2IN ) is a sample of Gaussian white noise . The observation density p ( y ) is related to the prior p ( x ) via marginalization : p ( y ) = Z p ( y|x ) p ( x ) dx = Z g ( y x ) p ( x ) dx , ( 1 ) where g ( z ) is the Gaussian noise distribution . Equation ( 1 ) is in the form of a convolution , and thus p ( y ) is a Gaussian-blurred version of the signal prior , p ( x ) . Moreover , the family of observation 1A software implementation of the sampling and linear inverse algorithms is available at https : //github . com/LabForComputationalVision/universal_inverse_problem densities over different noise variances , p ( y ) , forms a Gaussian scale-space representation of the prior [ 33 , 34 ] , analogous to the temporal evolution of a diffusion process . 1.2 Least squares denoising and CNNs . Given a noisy observation , y , the minimum mean squared error ( MMSE ) estimate of the true signal is well known to be the conditional mean of the posterior density : x̂ ( y ) = Z xp ( x|y ) dx = Z x p ( y|x ) p ( x ) p ( y ) dx ( 2 ) The structure of the equation mirrors the traditional approach to the problem : one chooses a prior probability model , p ( x ) , combines it with a likelihood function describing the noisy measurement process , p ( y|x ) , and solves . Modern denoising solutions , on the other hand , are often based on supervised learning of a direct mapping from noisy to denoised images . One expresses the estimation function ( as opposed to the prior ) in parametric form , and sets the parameters by minimizing the denoising MSE over a large training set of example signals and their noise-corrupted counterparts [ 35– 38 ] . Current state-of-the-art denoising results using CNNs obtained with this supervised approach are far superior to results of previous methods [ 39–41 ] . Recent analysis of these networks demonstrates that when they are trained to handle a broad range of noise levels , they perform an approximate projection onto a low-dimensional subspace [ 42 ] . In our context , we interpret this subspace as a tangent hyperplane of the image manifold . 1.3 Exposing the implicit prior through Empirical Bayes estimation . Trained CNN denoisers contain detailed prior knowledge of image structure , but Eq . ( 2 ) suggests that it is embedded within a high-dimensional integral . How can we make use of this implicit prior ? Recent results have derived relationships between Score Matching density estimates and denoising , and have used these relationships to make use of implicit prior information [ 43 , 44 , 29 , 45 ] . Here , we exploit a more direct but less-known result from the literature on Empirical Bayesian estimation . The idea was introduced in [ 46 ] , extended to the case of Gaussian additive noise in [ 9 ] ( see also [ 12 ] ) , and generalized to many other measurement models [ 11 ] . In the case of additive Gaussian noise , one can rewrite the estimator of Eq . ( 2 ) as : x̂ ( y ) = y + 2ry log p ( y ) . ( 3 ) The proof is relatively straightforward . The gradient of the observation density of Eq . ( 1 ) is : ry p ( y ) = 1 2 Z ( x y ) g ( y x ) p ( x ) dx = 1 2 Z ( x y ) p ( y , x ) dx . Multiplying both sides by 2/p ( y ) and separating the right side into two terms gives : 2 ry p ( y ) p ( y ) = Z xp ( x|y ) dx Z yp ( x|y ) dx = x̂ ( y ) y . Rearranging terms and using the chain rule to compute the gradient of the log gives Eq . ( 3 ) . This remarkable result re-expresses the integral over the prior and likelihood of Eq . ( 2 ) in terms of a gradient . Note that 1 ) the relevant density is not the prior , p ( x ) , but the noisy observation density , p ( y ) ; 2 ) the gradient is computed on the log density ( the associated “ energy function ” ) ; and 3 ) the gradient adjustment is not iterative - the estimate is achieved in a single step , and holds for any noise level , . 2 Drawing high-probability samples from the implicit prior . Suppose we wish to draw a sample from the prior implicit in a denoiser . Equation ( 3 ) allows us to generate an image proportional to the gradient of log p ( y ) by computing the denoiser residual , f ( y ) = x̂ ( y ) y . Song and Ermon [ 29 ] developed a Markov chain Monte Carlo ( MCMC ) scheme , combining gradient steps derived from Score Matching and injected noise in a Langevin sampling algorithm to draw samples from a sequence of densities p ( y ) , while reducing in a sequence of discrete steps , each associated with an appropriately trained denoiser . In contrast , starting from a random initialization , y0 , we aim to find a high-probability image ( i.e. , an image from the manifold ) using a more direct and efficient stochastic gradient ascent procedure .

Summary: Classical result by Miyasawa [25]  shows that the gradient of the log of the noisy signal density  can be expressed in terms of least-squares optimal denoisers for additive Gaussian noise.  Using this result, the authors develop an algorithm which uses  stochastic gradient ascent  to sample from the implicit prior of the denoiser, where the gradients  are computed using the residual of a trained bias-free universal CNN denoiser.  The algorithm includes noise injection which allows stochastic sampling of the manifold. They subsequently modify this algorithm to account for constraints from noise-less linear measurement model to solve linear inverse problems. The authors perform different linear image reconstruction tasks including spatial super-resolution, deblurring, compressive sensing, inpainting and reconstruction from random missing pixels. Experimentally, they demonstrate faster and visually better reconstructions as compared to the considered baselines.