Robust And Interpretable Blind Image Denoising Via Bias-Free Convolutional Neural Networks

1 INTRODUCTION AND CONTRIBUTIONS . The problem of denoising consists of recovering a signal from measurements corrupted by noise , and is a canonical application of statistical estimation that has been studied since the 1950 ’ s . Achieving high-quality denoising results requires ( at least implicitly ) quantifying and exploiting the differences between signals and noise . In the case of photographic images , the denoising problem is both an important application , as well as a useful test-bed for our understanding of natural images . In the past decade , convolutional neural networks ( LeCun et al. , 2015 ) have achieved state-of-the-art results in image denoising ( Zhang et al. , 2017 ; Chen & Pock , 2017 ) . Despite their success , these solutions are mysterious : we lack both intuition and formal understanding of the mechanisms they implement . Network architecture and functional units are often borrowed from the image-recognition literature , and it is unclear which of these aspects contributes to , or limits , the denoising performance . The goal of this work is advance our understanding of deep-learning models for denoising . Our contributions are twofold : First , we study the generalization capabilities of deep-learning models across different noise levels . Second , we provide novel tools for analyzing the mechanisms implemented by neural networks to denoise natural images . An important advantage of deep-learning techniques over traditional methodology is that a single neural network can be trained to perform denoising at a wide range of noise levels . Currently , this is achieved by simulating the whole range of noise levels during training ( Zhang et al. , 2017 ) . Here , we show that this is not necessary . Neural networks can be made to generalize automatically across noise ∗Equal contribution . levels through a simple modification in the architecture : removing all additive constants . We find this holds for a variety of network architectures proposed in previous literature . We provide extensive empirical evidence that the main state-of-the-art denoising architectures systematically overfit to the noise levels in the training set , and that this is due to the presence of a net bias . Suppressing this bias makes it possible to attain state-of-the-art performance while training over a very limited range of noise levels . The data-driven mechanisms implemented by deep neural networks to perform denoising are almost completely unknown . It is unclear what priors are being learned by the models , and how they are affected by the choice of architecture and training strategies . Here , we provide novel linear-algebraic tools to visualize and interpret these strategies through a local analysis of the Jacobian of the denoising map . The analysis reveals locally adaptive properties of the learned models , akin to existing nonlinear filtering algorithms . In addition , we show that the deep networks implicitly perform a projection onto an adaptively-selected low-dimensional subspace capturing features of natural images . 2 RELATED WORK . The classical solution to the denoising problem is the Wiener filter ( Wiener , 1950 ) , which assumes a translation-invariant Gaussian signal model . The main limitation of Wiener filtering is that it over-smoothes , eliminating fine-scale details and textures . Modern filtering approaches address this issue by adapting the filters to the local structure of the noisy image ( e.g . Tomasi & Manduchi ( 1998 ) ; Milanfar ( 2012 ) ) . Here we show that neural networks implement such strategies implicitly , learning them directly from the data . In the 1990 ’ s powerful denoising techniques were developed based on multi-scale ( `` wavelet '' ) transforms . These transforms map natural images to a domain where they have sparser representations . This makes it possible to perform denoising by applying nonlinear thresholding operations in order to discard components that are small relative to the noise level ( Donoho & Johnstone , 1995 ; Simoncelli & Adelson , 1996 ; Chang et al. , 2000 ) . From a linear-algebraic perspective , these algorithms operate by projecting the noisy input onto a lower-dimensional subspace that contains plausible signal content . The projection eliminates the orthogonal complement of the subspace , which mostly contains noise . This general methodology laid the foundations for the state-of-the-art models in the 2000 ’ s ( e.g . ( Dabov et al. , 2006 ) ) , some of which added a data-driven perspective , learning sparsifying transforms ( Elad & Aharon , 2006 ) , and nonlinear shrinkage functions ( Hel-Or & Shaked , 2008 ; Raphan & Simoncelli , 2008 ) , directly from natural images . Here , we show that deep-learning models learn similar priors in the form of local linear subspaces capturing image features . In the past decade , purely data-driven models based on convolutional neural networks ( LeCun et al. , 2015 ) have come to dominate all previous methods in terms of performance . These models consist of cascades of convolutional filters , and rectifying nonlinearities , which are capable of representing a diverse and powerful set of functions . Training such architectures to minimize mean square error over large databases of noisy natural-image patches achieves current state-of-the-art results ( Zhang et al. , 2017 ; Huang et al. , 2017 ; Ronneberger et al. , 2015 ; Zhang et al. , 2018a ) . 3 NETWORK BIAS IMPAIRS GENERALIZATION . We assume a measurement model in which images are corrupted by additive noise : y = x+n , where x ∈ RN is the original image , containing N pixels , n is an image of i.i.d . samples of Gaussian noise with variance σ2 , and y is the noisy observation . The denoising problem consists of finding a function f : RN → RN , that provides a good estimate of the original image , x . Commonly , one minimizes the mean squared error : f = arg ming E||x − g ( y ) ||2 , where the expectation is taken over some distribution over images , x , as well as over the distribution of noise realizations . In deep learning , the denoising function g is parameterized by the weights of the network , so the optimization is over these parameters . If the noise standard deviation , σ , is unknown , the expectation must also be taken over a distribution of σ . This problem is often called blind denoising in the literature . In this work , we study the generalization performance of CNNs across noise levels σ , i.e . when they are tested on noise levels not included in the training set . Feedforward neural networks with rectified linear units ( ReLUs ) are piecewise affine : for a given activation pattern of the ReLUs , the effect of the network on the input is a cascade of linear transformations ( convolutional or fully connected layers , Wk ) , additive constants ( bk ) , and pointwise multiplications by a binary mask corresponding to the fixed activation pattern ( R ) . Since each of these is affine , the entire cascade implements a single affine transformation . For a fixed noisy input image y ∈ RN with N pixels , the function f : RN → RN computed by a denoising neural network may be written f ( y ) = WLR ( WL−1 ... R ( W1y + b1 ) + ... bL−1 ) + bL = Ayy + by , ( 1 ) where Ay ∈ RN×N is the Jacobian of f ( · ) evaluated at input y , and by ∈ RN represents the net bias . The subscripts on Ay and by serve as a reminder that both depend on the ReLU activation patterns , which in turn depend on the input vector y . Based on equation 1 we can perform a first-order decomposition of the error or residual of the neural network for a specific input : y−f ( y ) = ( I−Ay ) y−by . Figure 1 shows the magnitude of the residual and the constant , which is equal to the net bias by , for a range of noise levels . Over the training range , the net bias is small , implying that the linear term is responsible for most of the denoising ( see Figures 9 and 10 for a visualization of both components ) . However , when the network is evaluated at noise levels outside of the training range , the norm of the bias increases dramatically , and the residual is significantly smaller than the noise , suggesting a form of overfitting . Indeed , network performance generalizes very poorly to noise levels outside the training range . This is illustrated for an example image in Figure 2 , and demonstrated through extensive experiments in Section 5 . 4 PROPOSED METHODOLOGY : BIAS-FREE NETWORKS . Section 3 shows that CNNs overfit to the noise levels present in the training set , and that this is associated with wild fluctuations of the net bias by . This suggests that the overfitting might be ameliorated by removing additive ( bias ) terms from every stage of the network , resulting in a biasfree CNN ( BF-CNN ) . Note that bias terms are also removed from the batch-normalization used during training . This simple change in the architecture has an interesting consequence . If the CNN has ReLU activations the denoising map is locally homogeneous , and consequently invariant to scaling : rescaling the input by a constant value simply rescales the output by the same amount , just as it would for a linear system . Lemma 1 . Let fBF : RN → RN be a feedforward neural network with ReLU activation functions and no additive constant terms in any layer . For any input y ∈ R and any nonnegative constant α , fBF ( αy ) = αfBF ( y ) . ( 2 ) Proof . We can write the action of a bias-free neural network with L layers in terms of the weight matrix Wi , 1 ≤ i ≤ L , of each layer and a rectifying operator R , which sets to zero any negative entries in its input . Multiplying by a nonnegative constant does not change the sign of the entries of a vector , so for any z with the right dimension and any α > 0R ( αz ) = αR ( z ) , which implies fBF ( αy ) = WLR ( WL−1 · · ·R ( W1αy ) ) = αWLR ( WL−1 · · ·R ( W1y ) ) = αfBF ( y ) . ( 3 ) Note that networks with nonzero net bias are not scaling invariant because scaling the input may change the activation pattern of the ReLUs . Scaling invariance is intuitively desireable for a denoising method operating on natural images ; a rescaled image is still an image . Note that Lemma 1 holds for networks with skip connections where the feature maps are concatenated or added , because both of these operations are linear . In the following sections we demonstrate that removing all additive terms in CNN architectures has two important consequences : ( 1 ) the networks gain the ability to generalize to noise levels not encountered during training ( as illustrated by Figure 2 the improvement is striking ) , and ( 2 ) the denoising mechanism can be analyzed locally via linear-algebraic tools that reveal intriguing ties to more traditional denoising methodology such as nonlinear filtering and sparsity-based techniques . 5 BIAS-FREE NETWORKS GENERALIZE ACROSS NOISE LEVELS . In order to evaluate the effect of removing the net bias in denoising CNNs , we compare several state-ofthe-art architectures to their bias-free counterparts , which are exactly the same except for the absence of any additive constants within the networks ( note that this includes the batch-normalization additive parameter ) . These architectures include popular features of existing neural-network techniques in image processing : recurrence , multiscale filters , and skip connections . More specifically , we examine the following models ( see Section A for additional details ) : • DnCNN ( Zhang et al. , 2017 ) : A feedforward CNN with 20 convolutional layers , each consisting of 3 × 3 filters , 64 channels , batch normalization ( Ioffe & Szegedy , 2015 ) , a ReLU nonlinearity , and a skip connection from the initial layer to the final layer . • Recurrent CNN : A recurrent architecture inspired by Zhang et al . ( 2018a ) where the basic module is a CNN with 5 layers , 3× 3 filters and 64 channels in the intermediate layers . The order of the recurrence is 4 . • UNet ( Ronneberger et al. , 2015 ) : A multiscale architecture with 9 convolutional layers and skip connections between the different scales . • Simplified DenseNet : CNN with skip connections inspired by the DenseNet architecture ( Huang et al. , 2017 ; Zhang et al. , 2018b ) . We train each network to denoise images corrupted by i.i.d . Gaussian noise over a range of standard deviations ( the training range of the network ) . We then evaluate the network for noise levels that are both within and beyond the training range . Our experiments are carried out on 180 × 180 natural images from the Berkeley Segmentation Dataset ( Martin et al. , 2001 ) to be consistent with previous results ( Schmidt & Roth , 2014 ; Chen & Pock , 2017 ; Zhang et al. , 2017 ) . Additional details about the dataset and training procedure are provided in Section B . Figures 3 , 11 and 12 show our results . For a wide range of different training ranges , and for all architectures , we observe the same phenomenon : the performance of CNNs is good over the training range , but degrades dramatically at new noise levels ; in stark contrast , the corresponding BF-CNNs provide strong denoising performance over noise levels outside the training range . This holds for both PSNR and the more perceptually-meaningful Structural Similarity Index ( Wang et al. , 2004 ) ( see Figure 12 ) . Figure 2 shows an example image , demonstrating visually the striking difference in generalization performance between a CNN and its corresponding BF-CNN . Our results provide strong evidence that removing net bias in CNN architectures results in effective generalization to noise levels out of the training range .

Summary: This paper looks at how deep convolutional neural networks for image denoising can generalize across various noise levels. First, they argue that state-of-the-art denoising networks perform poorly outside of the training noise range. The authors empirically show that as denoising performance degrades on unseen noise levels, the network residual for a specific input is being increasingly dominated by the network bias (as opposed to the purely linear Jacobian term). Therefore, they propose using bias-free convolutional neural networks for better generalization performance in image denoising. Their experimental results show that bias-free denoisers significantly outperform their original counter-parts on unseen noise levels across various popular architectures. Then, they perform a local analysis of the bias-free network around an input image that is now a strictly linear function of the input. They empirically demonstrate that the Jacobian is approximately low-rank and symmetric, therefore the effect of the denoiser can be interpreted as a nonlinear adaptive filter that projects the noisy image onto a low-dimensional signal subspace. The authors show that most of the energy of the clean image falls into the signal subspace and the effective dimensionality of this subspace is inversely proportional to the noise level.