Deep Denoising: Rate-Optimal Recovery of Structured Signals with a Deep Prior

1 INTRODUCTION . We consider the image or signal denoising problem , where the goal is to remove noise from an unknown image or signal . In more detail , our goal is to obtain an estimate of an image or signal y˚ P Rn from y “ y˚ ` η , where η is unknown noise , often modeled as a zero-mean white Gaussian random variable with covariance matrix σ2 { nI . Image denoising relies on modeling or prior assumptions on the image y˚ . For example , suppose that the image y˚ lies in a k-dimensional subspace of Rn denoted by Y . Then we can estimate the original image by finding the closest point in ` 2-distance to the noisy observation y on the subspace Y . The corresponding estimate , denoted by ŷ , obeys } ŷ ´ y˚ } 2 À σ2 k n , ( 1 ) with high probability ( throughout , } ¨ } denotes the ` 2-norm ) . Thus , the noise energy is reduced by a factor of k { n over the trivial estimate ŷ “ y which does not use any prior knowledge of the signal . The denoising rate ( 1 ) shows that the more concise the image prior or image representation ( i.e. , the smaller k ) , the more noise can be removed . If on the other hand the prior ( the subspace , in this example ) does not include the original image y˚ , then the error bound ( 1 ) increases as we would remove a significant part of the signal along with noise when projecting onto the range of the signal prior . Thus a concise and accurate prior is crucial for denoising . Real world signals rarely lie in a priori known subspaces , and the last few decades of image denoising research have developed sophisticated and accurate image models or priors and algorithms . Examples include models based on sparse representations in overcomplete dictionaries such as wavelets ( Donoho , 1995 ) and curvelets ( Starck et al. , 2002 ) , and algorithms based on exploiting self-similarity within images ( Dabov et al. , 2007 ) . A prominent example of the former class of algorithms is the BM3D ( Dabov et al. , 2007 ) algorithm , which achieves state-of-the-art performance for certain denoising problems . However , the nuances of real world images are difficult to describe with handcrafted models . Thus , starting with the paper ( Elad & Aharon , 2006 ) that proposes to learn sparse representation based on training data , it has become common to learn concise representation for denoising ( and other inverse problems ) from a set of training images . In 2012 , Burger et al . ( Burger et al. , 2012 ) applied deep networks to the denoising problem , by training a deep network on a large set of images . Since then , deep learning based denoisers ( Zhang et al. , 2017 ) have set the standard for denoising . The success of deep network priors can be attributed to their ability to efficiently represent and learn realistic image priors , for example via autodecoders ( Hinton & Salakhutdinov , 2006 ) and generative adversarial models ( Goodfellow et al. , 2014 ) . Over the last few years , the quality of deep priors has significantly improved ( Karras et al. , 2017 ; Ulyanov et al. , 2017 ) . As this field matures , priors will be developed with even smaller latent code dimensionality and more accurate approximation of natural signal manifolds . Consequently , the representation error from deep priors will decrease , and thereby enable even more powerful denoisers . As the influence of deep networks in inverse problems grows , it becomes increasingly important to understand their performance at a theoretical level . Given that most optimization approaches for deep learning are first order gradient methods , a justification is needed for why they do not get stuck in local minima . The closest theoretical work to this question is Bora et al . ( 2017 ) , which solves a noisy compressive sensing problem with generative priors by minimizing empirical risk . Under the assumption that the network is Lipschitz , they show that if the global optimizer can be found , which is in principle NP-hard , then a signal estimate is recovered to within the noise level . While the Lipschitzness assumption is quite mild , the resulting theory does not provide justification for why global optimality can be reached . The most related work that establishes theoretical reasons for why gradient methods would not get stuck in local minima , when using deep generative priors for solving inverse problems , is Hand & Voroninski ( 2018 ) . In it , the authors establish global favorability for optimization of the noiseless empirical risk function . Specifically , they show existence of a descent direction outside a ball around the global optimizer and a negative multiple of it in the latent space of the generative model . This work does not provide a specific algorithm which provably estimates the global minimizer , nor does it provide an analysis of the robustness of the problem with respect to noise . In this paper , we propose the first algorithm for solving denoising with deep generative priors that provably finds an approximation of the underlying image . Following the lead of Hand & Voroninski ( 2018 ) , we assume an expansive Gaussian model for the deep generative network in order to establish this result . Contributions : The goal of this paper is to analytically quantify the denoising performance of deep-prior based denoisers . Specifically , we characterize the performance of a simple and efficient algorithm for denoising based on a d-layer generative neural network G : Rk Ñ Rn , with k ă n , and random weights . In more detail , we propose a gradient method with a tweak that attempts to minimize the least-squares loss fpxq “ 12 } Gpxq ´ y } 2 between the noisy image y and an image in the range of the prior , Gpxq . While f is non-convex , we show that the gradient method yields an estimate x̂ obeying } Gpx̂q ´ y˚ } 2 À σ2 k n , with high probability , where the notation À absorbs a constant factor depending on the number of layers of the network , and its expansitivity , as discussed in more detail later . Our result shows that the denoising rate of a deep prior based denoiser is determined by the dimension of the latent representation . We also show in numerical experiments , that this rate—shown to be analytically achieved for random priors—is also experimentally achieved for priors learned from real imaging data . 2 PROBLEM FORMULATION . We consider the problem of estimating a vector y˚ P Rn from a noisy observation y “ y˚ ` η . We assume that the vector y˚ belongs to the range of a d-layer generative neural network G : Rk Ñ Rn , with k ă n. That is , y˚ “ Gpx˚q for some x˚ P Rk . We consider a generative network of the form Gpxq “ relupWd . . . relupW2 relupW1x˚qq . . .q , where relupxq “ maxpx , 0q applies entrywise , Wi P Rniˆni´1 , are the weights in the i-th layer , ni is the number of neurons in the ith layer , and the network is expansive in the sense that k “ n0 ă n1 ă ¨ ¨ ¨ ă nd “ n. The problem at hand is : Given the weights of the network W1 . . .Wd and a noisy observation y , obtain an estimate ŷ of the original image y˚ such that } ŷ ´ y˚ } is small and ŷ is in the range of G . 3 DENOISING VIA EMPIRICAL RISK MINIMIZATION . As a way to solve the above problem , we first obtain an estimate of x˚ , denoted by x̂ , and then estimate y˚ as Gpx̂q . In order to estimate x˚ , we minimize the empirical risk objective fpxq : “ 1 2 } Gpxq ´ y } 2 . Since this objective is nonconvex , there is no a priori guarantee of efficiently finding the global minimum . Approaches such as gradient methods could in principle get stuck in local minima , instead of finding a global minimizer that is close to x˚ . However , as we show in this paper , under appropriate conditions , a gradient method with a tweak— introduced next—finds a point that is very close to the original latent parameter x˚ , with the distance to the parameter x˚ controlled by the noise . In order to state the algorithm , we first introduce a useful quantity . For analyzing which rows of a matrix W are active when computing relupWxq , we let W ` , x “ diagpWx ą 0qW . For a fixed weight matrix W , the matrix W ` , x zeros out the rows of W that do not have a positive dot product with x. Alternatively put , W ` , x contains weights from only the neurons that are active for the input x . We also define W1 , ` , x “ pW1q ` , x “ diagpW1x ą 0qW1 and Wi , ` , x “ diagpWiWi´1 , ` , x ¨ ¨ ¨W2 , ` , xW1 , ` , xx ą 0qWi . The matrix Wi , ` , x consists only of the weights of the neurons in the ith layer that are active if the input to the first layer is x . We are now ready to state our algorithm : a gradient method with a tweak informed by the loss surface of the function to be minimized . Given a noisy observation y , the algorithm starts with an arbitrary initial point x0 ‰ 0 . At each iteration i “ 0 , 1 , . . . , the algorithm computes the step direction ṽxi “ pΠ1i “ dWi , ` , xiqtpGpxiq ´ yq , which is equal to the gradient of f if f is differentiable at xi . It then takes a small step opposite to ṽxi . The tweak is that before each iteration , the algorithm checks whether fp´xiq is smaller than fpxiq , and if so , negates the sign of the current iterate xi . This tweak is informed by the loss surface . To understand this step , it is instructive to examine the loss surface for the noiseless case in Figure 1 . It can be seen that while the loss function has a global minimum at x˚ , it is relatively flat close to ´x˚ . In expectation , there is a critical point that is a negative multiple of x˚ with the property that the curvature in the ˘x˚ direction is positive , and the curvature in the orthogonal directions is zero . Further , around approximately ´x˚ , the loss function is larger than around the optimum x˚ . As a simple gradient descent method ( without the tweak ) could potentially get stuck in this region , the negation check provides a way to avoid converging to this region . Our algorithm is formally summarized as Algorithm 1 below . Algorithm 1 Gradient method Require : Weights of the network Wi , noisy observation y , and step size α ą 0 1 : Choose an arbitrary initial point x0 P Rkzt0u 2 : for i “ 0 , 1 , . . . do 3 : if fp´xiq ă fpxiq then 4 : xi Ð ´xi ; 5 : end if 6 : Compute ṽxi “ pΠ1i “ dWi , ` , xiqtpGpxiq ´ yq 7 : xi ` 1 “ xi ´ αṽxi 8 : end for Other variations of the tweak are also possible . For example , the negation check in Step 2 could be performed after a convergence criterion is satisfied , and if a lower objective is achieved by negating the latent code , then the gradient descent can be continued again until a convergence criterion is again satisfied .

Summary: The paper studies the standard denoising problem under the assumption that the unknown n-dimensional signal can be written as the output of a known d-layer neural network G mapping k dimensions to n dimensions. The paper specifies an algorithm to perform this denoising and the algorithm is based on a variant of the usual gradient method. Then, under additional assumptions on the neural network G, the paper proves that their algorithm produces a denoised signal that achieves a mean squared accuracy of k/n. Because the input signal has "effective" dimensionality k (as it can be written as G(x) for some k-dimensional x), it is nice that it can be recovered at the accuracy k/n by Gradient Descent despite the complicated nature of G. In this respect, the result is quite interesting. However, the underlying assumptions are too strong in my opinion as described below: