Denoising and Regularization via Exploiting the Structural Bias of Convolutional Generators

Convolutional Neural Networks ( CNNs ) have emerged as highly successful tools for image generation , recovery , and restoration . A major contributing factor to this success is that convolutional networks impose strong prior assumptions about natural images . A surprising experiment that highlights this architectural bias towards natural images is that one can remove noise and corruptions from a natural image without using any training data , by simply fitting ( via gradient descent ) a randomly initialized , over-parameterized convolutional generator to the corrupted image . While this over-parameterized network can fit the corrupted image perfectly , surprisingly after a few iterations of gradient descent it generates an almost uncorrupted image . This intriguing phenomena enables state-of-the-art CNNbased denoising and regularization of other inverse problems . In this paper we attribute this effect to a particular architectural choice of convolutional networks , namely convolutions with fixed interpolating filters . We then formally characterize the dynamics of fitting a two layer convolutional generator to a noisy signal and prove that early-stopped gradient descent denoises/regularizes . Our proof relies on showing that convolutional generators fit the structured part of an image significantly faster than the corrupted portion . 1 INTRODUCTION . Convolutional neural networks are extremely popular for image generation . The majority of image generating networks is convolutional , ranging from deep convolutional Generative Adversarial Networks ( DC-GANs ) ( Radford et al. , 2015 ) to the U-Net ( Ronneberger et al. , 2015 ) . It is well known that convolutional neural networks incorporate implicit assumptions about the signals they generate , such as pixels that are close being related . This makes them particularly well suited for modeling distributions of images . It is less well known , however , that those prior assumptions build into the architecture are so strong that convolutional neural networks are useful even without ever being exposed to training data . The latter was first shown in the Deep Image Prior ( DIP ) paper ( Ulyanov et al. , 2018 ) . Ulyanov et al . ( 2018 ) observed that when ‘ training ’ a standard convolutional auto-encoder such as the popular U-net ( Ronneberger et al. , 2015 ) on a single noisy image and regularizing by early stopping , the network denoises the image with excellent performance . This is based on the empirical observation that un-trained convolutional auto-decoders fit a single natural image faster when optimized with gradient descent than pure noise . Many elements of an auto-encoder , however are irrelevant for this effect : A more recent paper ( Heckel & Hand , 2019 ) proposed a much simpler image generating network , termed the deep decoder . This network can be seen as the relevant part of a convolutional generator architecture to function as an image prior , and can be obtained from a standard convolutional autoencoder by removing the encoder , the skip connections , and perhaps most notably , the trainable convolutional filters of spatial extent larger than one . Thus , the deep decoder does not use learned or trainable convolutional filters like conventional convolutional networks do , and instead only uses convolutions with fixed convolutional kernels to generate an image . In this paper , we study such a simple , untrained convolutional network theoretically . We consider the over-parameterized regime where the network has sufficiently many parameters to represent an arbitrary image ( including noise ) perfectly and show that : Fitting convolutional generators via early stopped gradient descent provably denoises “ natural ” images . To prove this statement , we characterize how the network architecture governs the dynamics of fitting over-parameterzed networks to a single ( noisy ) image . In particular we prove : Convolutional generators optimized with gradient descent fit natural images faster than noise . We depict this phenomena in Figure 1 where we fit a randomly initialized over-parameterized convolutional generator to an image via running gradient descent on the objective L ( C ) = ‖G ( C ) − y‖22 . Here , G ( C ) is the convolutional generator with weight parameters C , and y is either a noisy image , a clean image , or noise . This experiment demonstrates that an over-parameterized convolutional network fits a natural image ( Figure 1b ) much faster than noise ( Figure 1c ) . Thus , when fitting the noisy image ( Figure 1a ) , early stopping the optimization enables image denoising . This effect is so strong that it gives state-of-the-art denoising performance ( see Figure 2 and Appendix A ) . Beyond denoising , this effect also enables significant improvements in regularizing a variety of inverse problems such as compressive sensing ( Veen et al. , 2018 ; Heckel , 2019 ) . 1.1 CONTRIBUTIONS AND OVERVIEW OF RESULTS . In this paper we take a step towards understanding why fitting a convolutional generator via early stopped gradient descent leads to such surprising denoising and regularization capabilities . 1 2 6 21 2 CONVOLUTIONAL GENERATORS . A convolutional generator maps an input tensor B0 to an image only using upsampling and convolutional operations , followed by channel normalization ( a special case of batch normalization ) and applications of non-linearities , see Figure 4 . All previously mentioned convolutional generator networks ( Radford et al. , 2015 ; Ronneberger et al. , 2015 ) including the networks considered in the DIP paper ( Ulyanov et al. , 2018 ) primarily consist of those operations . For motivating the architecture of the convolutional generators studied in this paper , we first demonstrate in Section 2.1 that convolutions with fixed interpolation filters are critical to the denoising performance with early stopping . Specifically , we empirically show that convolutions with fixed convolutional kernels are critical for convolutional generators to fit natural images faster than noise . In Section 2.2 we formally introduce the class of convolutional generators studied in this paper , and finally , in Section 2.3 , we introduce a minimal convolutional architecture which is the focus of our theoretical results . 2.1 THE IMPORTANCE OF FIXED CONVOLUTIONAL FILTERS . Convolutions with fixed convolutional kernels are critical for denoising with early stopping , because they are critical for the phenomena that natural images are fitted significantly faster than noise . To see this , consider the experiment in Figure 4 in which we fit an image and noise by minimizing the least-squares loss via gradient descent with i ) a convolutional generator with only fixed convolutional filters ( see Section 2.2 below for a precise description ) and ii ) a conventional convolutional generator with trainable convolutional filters ( essentially the architecture from the popular DC-GAN generators , see Appendix B for details and additional numerical evidence ) . As illustrated in Figure 4 , the convolutional network with fixed filters fits the natural image much faster than noise , whereas the network with learned convolutional filters , only fits it slightly faster , and this effect vanishes as the network becomes highly overparameterized . Thus , fixed convolutional filters enable un-trained convolutional networks to function as highly effective image priors . We note that the upsampling operation present in most architectures implicitly incorporates a convolution with a fixed convolutional ( interpolation ) filter . 2.2 ARCHITECTURE OF CONVOLUTIONAL GENERATOR WITH FIXED CONVOLUTIONS . In this section , we describe the architecture of the deep decoder ( Heckel & Hand , 2019 ) , a convolutional network with fixed convolutional operators only . In this architecture , the channels in the ( i+ 1 ) -th layer are given by Bi+1 = cn ( ReLU ( UiBiCi ) ) , i = 0 , . . . , d− 1 , and finally , the output of the d-layer network is formed as x = BdCd+1 . Here , the coefficient matrices Ci ∈ Rk×k and Cd+1 ∈ Rk×kout contain the weights of the network . The number of channels , k , determines the number of weight parameters of the network , given by dk2 + koutk . Each column of the tensor BiCi ∈ Rni×k is formed by taking linear combinations of the channels of the tensor Bi in a way that is consistent across all pixels , and the ReLU activation function is given by ReLU ( t ) = max ( 0 , t ) . Then , cn ( · ) performs the channel normalization operation which normalizes each channel individually and can be viewed as a special case of the popular batch normalization operation ( Ioffe & Szegedy , 2015 ) . The operator Ui ∈ Rni+1×ni is a tensor implementing an upsampling and most importantly a convolution operation with a fixed kernel . This fixed kernel was chosen in all experiments above as a triangular kernel so that U performs bi-linear 2x upsampling ( this is the standard implementation in the popular software packages pytorch and tensorflow ) . As mentioned earlier this convolution with a fixed kernel is critical for fitting natural images faster than complex ones . 2.3 TWO LAYER CONVOLUTIONAL GENERATOR STUDIED THEORETICALLY IN THIS PAPER . The simplest model to study the denoising capability of convolutional generators and the phenomena that a natural image is fitted faster than a complex one theoretically is a network with only one hidden layers and one output channel i.e. , G ( C ) ∈ Rn . Then , the generator becomes G ( C ) = ReLU ( UB0C0 ) c1 , where U ∈ Rn×n is a circulant matrix that implements a convolution with a filter u . In this paper we consider the over-parameterized regime where k ≥ 2n . Note that scaling the i-th column of C0 with a non-negativ factor is equivalent to scaling the i-th entry of the output weights c1 with the same factor . We therefore fix the output weights to c1 = v , where v = [ 1 , . . . , 1 , −1 , . . . , −1 ] / √ k. Next , note that both the generators with the fixed output weights and non-fixed output weights have the same range in the regime k ≥ 2n . Next , because the matrix B0 is Gaussian , with probability one , it has full rank and thus spans Rn . It follows that optimizing over the parameter C0 in C = B0C0 is equivalent to optimizing over the matrix C ∈ Rn×k . We therefore consider the generator G ( C ) = ReLU ( UC ) v , ( 1 ) where v = [ 1 , . . . , 1 , −1 , . . . , −1 ] / √ k is fixed and C ∈ Rn×k is the new coefficient matrix we optimize over . Figure 11 in the appendix shows that even this simple two-layer convolutional network fits a simple image faster than noise . This is the simplest model in which the phenomena that a convolutional networks fits structure faster than noise can reliably be observed . As a consequence , the dynamics of training the model ( 1 ) are the focus of the remainder of this paper . 3 WARMUP : DYNAMICS OF GRADIENT DESCENT ON LEAST SQUARES . As a prelude for studying the dynamics of fitting convolutional generators via a non-linear least square problem , we study the dynamics of gradient descent applied to a linear least squares problem . We demonstrate how early stopping can lead to denoising capabilities even with a simple linear model . We consider a least-squares problem of the form L ( c ) = 1 2 ‖y − Jc‖22 , and study gradient descent with a constant step size η starting at c0 = 0 . The updates are given by cτ+1 = cτ − η∇L ( cτ ) , ∇L ( c ) = JT ( Jc− y ) . The following simple proposition characterizes the trajectory of gradient descent . Proposition 1 . Let J ∈ Rn×m be a matrix with left singular vectors w1 , . . . , wn ∈ Rn and corresponding singular values σ1 ≥ σ2 ≥ . . . ≥ σn . Then the residual after τ steps , rτ = y − Jcτ , of gradient descent starting at c0 = 0 is rτ = n∑ i=1 wi 〈wi , y〉 ( 1− ησ2i ) τ . Suppose that the signal y lies in the column span of J , and that the stepsize is chosen sufficiently small ( i.e. , η ≤ 1/ ‖J‖2 ) . Then , by Proposition 1 , gradient descent converges to a zero-loss solution and thus fits the signal perfectly . More importantly , gradient descent fits the components of y corresponding to large singular values faster than it fits the components corresponding to small singular values . To explicitly show how this observation enables regularization via early stopped gradient descent , suppose our goal is to find a good estimate of a signal x from a noisy observation y = x + z , where the signal x lies in a signal subspace that is spanned by the p leading left-singular vectors of J . Then , by Proposition 1 , the signal estimate after τ iterations , Jcτ , obeys ‖Jcτ − x‖2 ≤ ( 1− ησ 2 p ) τ ‖x‖2 + E ( z ) , E ( z ) : = √√√√ n∑ i=1 ( ( 1− ησ2i ) τ − 1 ) 2 〈wi , z〉 2 . ( 2 ) Thus , after a few iterations most of the signal has been fitted ( i.e. , ( 1−ησp ) τ is small ) . Furthermore , if we assume that the ratio σp/σp+1 is sufficiently large so that the spread between the two singular values separating the signal subspace from the rest is sufficiently large , most of the noise outside the signal subspace has not been fitted ( i.e. , ( ( 1− ησ2i ) τ − 1 ) 2 ≈ 0 for i = p+ 1 , . . . , n ) . In particular , suppose the noise vector has a Gaussian distribution given by z ∼ N ( 0 , ς 2 n I ) . Then E ( z ) ≈ ς √ p n so that after order τ = log ( ) / log ( 1− ησ 2 p ) iterations , with high probability , ‖Jcτ − x‖2 ≤ ‖x‖2 + cς √ p n . This demonstrates , that provided the signal lies in a subspace spanned by the leading singular vectors , early stoped gradient descent reaches the optimal denoising rate of ς √ p/n after a few iterations . See Figure 5 for a numerical example demonstrating this phenomena .

Summary: This paper studies the situation in which a two-layer CNN with RELU nonlinearity is fit to a single image and the observation that it is able to fit a "natural" image in fewer iterations than a "noisy" image. Theorems on the convergence of this fitting are discussed and proven in the appendices. Intermediate results study the convergence of fitting a linear model to an image plus noise and fitting a single-layer CNN. Denoising is demonstrated on two images with additive white Gaussian noise and the approach under study is shown to provide a better signal-to-noise ratio than BM3D, another untrained denoising approach. The main result is that the use of upsampling via a fixed interpolation filter provides an inductive bias towards "natural" images.