Deep Denoising: Rate-Optimal Recovery of Structured Signals with a Deep Prior

1 INTRODUCTION . We consider the image or signal denoising problem , where the goal is to remove noise from an unknown image or signal . In more detail , our goal is to obtain an estimate of an image or signal y˚ P Rn from y “ y˚ ` η , where η is unknown noise , often modeled as a zero-mean white Gaussian random variable with covariance matrix σ2 { nI . Image denoising relies on modeling or prior assumptions on the image y˚ . For example , suppose that the image y˚ lies in a k-dimensional subspace of Rn denoted by Y . Then we can estimate the original image by finding the closest point in ` 2-distance to the noisy observation y on the subspace Y . The corresponding estimate , denoted by ŷ , obeys } ŷ ´ y˚ } 2 À σ2 k n , ( 1 ) with high probability ( throughout , } ¨ } denotes the ` 2-norm ) . Thus , the noise energy is reduced by a factor of k { n over the trivial estimate ŷ “ y which does not use any prior knowledge of the signal . The denoising rate ( 1 ) shows that the more concise the image prior or image representation ( i.e. , the smaller k ) , the more noise can be removed . If on the other hand the prior ( the subspace , in this example ) does not include the original image y˚ , then the error bound ( 1 ) increases as we would remove a significant part of the signal along with noise when projecting onto the range of the signal prior . Thus a concise and accurate prior is crucial for denoising . Real world signals rarely lie in a priori known subspaces , and the last few decades of image denoising research have developed sophisticated and accurate image models or priors and algorithms . Examples include models based on sparse representations in overcomplete dictionaries such as wavelets ( Donoho , 1995 ) and curvelets ( Starck et al. , 2002 ) , and algorithms based on exploiting self-similarity within images ( Dabov et al. , 2007 ) . A prominent example of the former class of algorithms is the BM3D ( Dabov et al. , 2007 ) algorithm , which achieves state-of-the-art performance for certain denoising problems . However , the nuances of real world images are difficult to describe with handcrafted models . Thus , starting with the paper ( Elad & Aharon , 2006 ) that proposes to learn sparse representation based on training data , it has become common to learn concise representation for denoising ( and other inverse problems ) from a set of training images . In 2012 , Burger et al . ( Burger et al. , 2012 ) applied deep networks to the denoising problem , by training a deep network on a large set of images . Since then , deep learning based denoisers ( Zhang et al. , 2017 ) have set the standard for denoising . The success of deep network priors can be attributed to their ability to efficiently represent and learn realistic image priors , for example via autodecoders ( Hinton & Salakhutdinov , 2006 ) and generative adversarial models ( Goodfellow et al. , 2014 ) . Over the last few years , the quality of deep priors has significantly improved ( Karras et al. , 2017 ; Ulyanov et al. , 2017 ) . As this field matures , priors will be developed with even smaller latent code dimensionality and more accurate approximation of natural signal manifolds . Consequently , the representation error from deep priors will decrease , and thereby enable even more powerful denoisers . As the influence of deep networks in inverse problems grows , it becomes increasingly important to understand their performance at a theoretical level . Given that most optimization approaches for deep learning are first order gradient methods , a justification is needed for why they do not get stuck in local minima . The closest theoretical work to this question is Bora et al . ( 2017 ) , which solves a noisy compressive sensing problem with generative priors by minimizing empirical risk . Under the assumption that the network is Lipschitz , they show that if the global optimizer can be found , which is in principle NP-hard , then a signal estimate is recovered to within the noise level . While the Lipschitzness assumption is quite mild , the resulting theory does not provide justification for why global optimality can be reached . The most related work that establishes theoretical reasons for why gradient methods would not get stuck in local minima , when using deep generative priors for solving inverse problems , is Hand & Voroninski ( 2018 ) . In it , the authors establish global favorability for optimization of the noiseless empirical risk function . Specifically , they show existence of a descent direction outside a ball around the global optimizer and a negative multiple of it in the latent space of the generative model . This work does not provide a specific algorithm which provably estimates the global minimizer , nor does it provide an analysis of the robustness of the problem with respect to noise . In this paper , we propose the first algorithm for solving denoising with deep generative priors that provably finds an approximation of the underlying image . Following the lead of Hand & Voroninski ( 2018 ) , we assume an expansive Gaussian model for the deep generative network in order to establish this result . Contributions : The goal of this paper is to analytically quantify the denoising performance of deep-prior based denoisers . Specifically , we characterize the performance of a simple and efficient algorithm for denoising based on a d-layer generative neural network G : Rk Ñ Rn , with k ă n , and random weights . In more detail , we propose a gradient method with a tweak that attempts to minimize the least-squares loss fpxq “ 12 } Gpxq ´ y } 2 between the noisy image y and an image in the range of the prior , Gpxq . While f is non-convex , we show that the gradient method yields an estimate x̂ obeying } Gpx̂q ´ y˚ } 2 À σ2 k n , with high probability , where the notation À absorbs a constant factor depending on the number of layers of the network , and its expansitivity , as discussed in more detail later . Our result shows that the denoising rate of a deep prior based denoiser is determined by the dimension of the latent representation . We also show in numerical experiments , that this rate—shown to be analytically achieved for random priors—is also experimentally achieved for priors learned from real imaging data . 2 PROBLEM FORMULATION . We consider the problem of estimating a vector y˚ P Rn from a noisy observation y “ y˚ ` η . We assume that the vector y˚ belongs to the range of a d-layer generative neural network G : Rk Ñ Rn , with k ă n. That is , y˚ “ Gpx˚q for some x˚ P Rk . We consider a generative network of the form Gpxq “ relupWd . . . relupW2 relupW1x˚qq . . .q , where relupxq “ maxpx , 0q applies entrywise , Wi P Rniˆni´1 , are the weights in the i-th layer , ni is the number of neurons in the ith layer , and the network is expansive in the sense that k “ n0 ă n1 ă ¨ ¨ ¨ ă nd “ n. The problem at hand is : Given the weights of the network W1 . . .Wd and a noisy observation y , obtain an estimate ŷ of the original image y˚ such that } ŷ ´ y˚ } is small and ŷ is in the range of G . 3 DENOISING VIA EMPIRICAL RISK MINIMIZATION . As a way to solve the above problem , we first obtain an estimate of x˚ , denoted by x̂ , and then estimate y˚ as Gpx̂q . In order to estimate x˚ , we minimize the empirical risk objective fpxq : “ 1 2 } Gpxq ´ y } 2 . Since this objective is nonconvex , there is no a priori guarantee of efficiently finding the global minimum . Approaches such as gradient methods could in principle get stuck in local minima , instead of finding a global minimizer that is close to x˚ . However , as we show in this paper , under appropriate conditions , a gradient method with a tweak— introduced next—finds a point that is very close to the original latent parameter x˚ , with the distance to the parameter x˚ controlled by the noise . In order to state the algorithm , we first introduce a useful quantity . For analyzing which rows of a matrix W are active when computing relupWxq , we let W ` , x “ diagpWx ą 0qW . For a fixed weight matrix W , the matrix W ` , x zeros out the rows of W that do not have a positive dot product with x. Alternatively put , W ` , x contains weights from only the neurons that are active for the input x . We also define W1 , ` , x “ pW1q ` , x “ diagpW1x ą 0qW1 and Wi , ` , x “ diagpWiWi´1 , ` , x ¨ ¨ ¨W2 , ` , xW1 , ` , xx ą 0qWi . The matrix Wi , ` , x consists only of the weights of the neurons in the ith layer that are active if the input to the first layer is x . We are now ready to state our algorithm : a gradient method with a tweak informed by the loss surface of the function to be minimized . Given a noisy observation y , the algorithm starts with an arbitrary initial point x0 ‰ 0 . At each iteration i “ 0 , 1 , . . . , the algorithm computes the step direction ṽxi “ pΠ1i “ dWi , ` , xiqtpGpxiq ´ yq , which is equal to the gradient of f if f is differentiable at xi . It then takes a small step opposite to ṽxi . The tweak is that before each iteration , the algorithm checks whether fp´xiq is smaller than fpxiq , and if so , negates the sign of the current iterate xi . This tweak is informed by the loss surface . To understand this step , it is instructive to examine the loss surface for the noiseless case in Figure 1 . It can be seen that while the loss function has a global minimum at x˚ , it is relatively flat close to ´x˚ . In expectation , there is a critical point that is a negative multiple of x˚ with the property that the curvature in the ˘x˚ direction is positive , and the curvature in the orthogonal directions is zero . Further , around approximately ´x˚ , the loss function is larger than around the optimum x˚ . As a simple gradient descent method ( without the tweak ) could potentially get stuck in this region , the negation check provides a way to avoid converging to this region . Our algorithm is formally summarized as Algorithm 1 below . Algorithm 1 Gradient method Require : Weights of the network Wi , noisy observation y , and step size α ą 0 1 : Choose an arbitrary initial point x0 P Rkzt0u 2 : for i “ 0 , 1 , . . . do 3 : if fp´xiq ă fpxiq then 4 : xi Ð ´xi ; 5 : end if 6 : Compute ṽxi “ pΠ1i “ dWi , ` , xiqtpGpxiq ´ yq 7 : xi ` 1 “ xi ´ αṽxi 8 : end for Other variations of the tweak are also possible . For example , the negation check in Step 2 could be performed after a convergence criterion is satisfied , and if a lower objective is achieved by negating the latent code , then the gradient descent can be continued again until a convergence criterion is again satisfied .

Summary: The paper analyzes the recovery accuracy of a "tweaked" gradient descent algorithm for imaging denoising and compressive sensing under deep generative priors. In particular, when assuming Gaussian randomness of the network weights and extremely stringent conditions of network sizes, they demonstrate a specific denoising rate of O(k/n), with k and n being the input and output dimension of the generative network. This is seemingly optimal in terms of the dependence on the latent code dimensionality and the signal dimensionality and is the first result of this kind.