Robust And Interpretable Blind Image Denoising Via Bias-Free Convolutional Neural Networks
Authors: Sreyas Mohan
Zahra Kadkhodaie
Eero P. Simoncelli
Carlos Fernandez-Granda
Venue:  (2020)
Keywords: denoising
overfitting
generalization
robustness
interpretability
analysis of neural networks

Abstract: We study the generalization properties of deep convolutional neural networks for image denoising in the presence of varying noise levels. We provide extensive empirical evidence that current state-of-the-art architectures systematically overfit to the noise levels in the training set, performing very poorly at new noise levels. We show that strong generalization can be achieved through a simple architectural modification: removing all additive constants. The resulting "bias-free" networks attain state-of-the-art performance over a broad range of noise levels, even when trained over a limited range. They are also locally linear, which enables direct analysis with linear-algebraic tools.  We show that the denoising map can be visualized locally as a filter that adapts to both image structure and noise level. In addition, our analysis reveals that deep networks implicitly perform a projection onto an adaptively-selected low-dimensional subspace, with dimensionality inversely proportional to noise level, that captures features of natural images. 

Published as a conference paper at ICLR 2020
ROBUST AND INTERPRETABLE BLIND IMAGE DENOISING VIA BIAS-FREE CONVOLUTIONAL
NEURAL NETWORKS
Sreyas Mohan Center for Data Science New York University
Zahra Kadkhodaie"
Center for Data Science New York University
Sm7 5 82 82wnnn2nnuuucco.
zk3888nyu.ede edu
Eero P. Simoncelli
Carlos Fernandez-Granda
Center for Neural Science, and Howard Hughes Medical Institute New York University
Center for Data Science, and Courant Inst. of Mathematical Sciences
New York University
eero. s simoncelli@nynnieeeeeee.e edu
efgr rEgranda@ cimss nyu. edu
ABSTRACT
We study the generalization properties of deep convolutional neural networks for image denoising in the presence of varying noise levels. We provide extensive empirical evidence that current state-of-the-ar architectures systematically overfit to the noise levels in the training set, performing very poorly at new noise levels. We show that strong generalization can be achieved through a simple architectural modification: removing all additive constants. The resulting "bias-free" networks attain state-of-the-are performance over a broad range of noise levels, even when trained over narrow range. They are also locally linear, which enables direct anal- ysis with linear-algebraic tools. We show that the denoising map can be visualized locally filter that adapts to both image structure and noise level. In addi- tion, our analysis reveals that deep networks implicitly perform projection onto adaptively- selected cow-dimeniional subspace, with dimensionality inversely proportional to noise level, that captures features of natural images.
INTRODUCTION AND CONTRIBUTIONS
The problem of denoising consists of recovering a signal from measurements corrupted by noise, and canonical application of statistical estimation that has been studied since the 1950's. Achieving high-quality denoising results requires (at least implicitly) quantifying and exploiting the differences between signals and noise. In the of photographic images, the denoising problem is both an important application, as well as a useful test-bed for our understanding of natural images. In the past decade, convolutional neural networks (LeCun al. 2015) have achieved state-of-the-arn results in image denoising (Zhang et al.) 2017 Chen & Pock 2017). Despite their success, these solutions are mysterious: we lack both intuition and formal understanding of the mechanisms they implement. Network architecture and functional units are often borrowed from the renoggttcoonniionn literature, and unclear which of these aspects contributes to, or limits, the denoising performance. The goal of this work is advance our understanding of deep-learning models for denoising. Our contributions are twofold: First, we study the generalization capabilities of deep-learning models across different noise levels. Second, provide novel tools for analyzing the mechanisms implemented by neural networks to denoise natural images.
An important advantage of deep-learning techniques over traditional methodology is that single neural network can be trained to perform denoising at a wide range of noise levels. Currently, this is achieved by simulating the whole range of noise levels during training (Zhang et al. 2017]. Here, we show that this is not necessary. Neural networks can be made to generalize automatically across noise
"Equal contribution.

---

Published as a conference paper at ICLR 2020
DAS dMal y
TO/19,
ROTRIN
15-
25
25 35
Doise level (sd)
moise level (sd)
moise level (sd)
(a)
(b)
(c)
Figure I: First-order analysis of the residual of denoising convolutional neural network as function of noise level. The plots show the norms of the residual and the net bias averaged over 100 20 20 natural- natural-image patches for networks trained over different training ranges. The range of noises used for training is highlighted in blue. (a) When the network trained over the full range noise levels (a [0, 100)) the net bias is small, growing slightly as the noise increases. (b-c) When the network is trained over the smaller range (a E [0, 55) and a E [0. 30)), the net bias grows explosively for noise levels beyond the training range. This coincides with dramatic drop in performance, reflected in the difference between the magnitudes the residual and the true noise. The CNN used for this example is DrCNN (Zhang et al. 2017): using alternative architectures yields similar results as shown in Figure
levels through a simple modification in the architecture: removing all additive constants. We find this holds for a variety of network architectures proposed in previous literature. We provide extensive empirical evidence that the main state-of ooie-ooflii of-the-art denoising architectures systematically overfit to the noise levels in the training set, and that this is due the presence of net bias. Suppressing this bias makes it possible to attain state-of-the-art performance while training over very limited range of noise levels.
The data-driven mechanisms implemented by deep neural networks to perform denoising are almost completely unknown. unclear what priors are being learned by the models, and how they are affected by the choice of architecture and training strategies. Here, we provide novel tinear-algebrace tools to visualize and interpret these strategies through a local analysis of the Jacobian of the denoising map. The analysis reveals locally adaptive properties of the learned models, akin to existing nonlinear filtering algorithms. In addition, we show that the deep networks implicitly perform a projection onto an adaptively selected low-dimensional subspace capturing features of natural images.
2 RELATED WORK
The classical solution the denoising problem is the Wiener filter [Wiener] 1950). which assumes a cunslation-invarian invariant Gaussian signal model. The main limitation of Wiener filtering is that over-smoothes, eliminating fine-scale details and textures. Modern filtering approaches address this issue by adapting the filters to the local structure of the noisy image (e.g. Tomasi & Manduchi] (1998); Milanfar 22122). ). Here we show that neural networks implement such strategies implicitly, learning them directly from the data.
In the 1990's powerful denoising techniques transforms. These transforms map natural images to a domain where they have sparser representations. This makes possible perform denoising by applying nonlinear thresholding operations order to discard components that are small relative to the noise level (Donoho & Johnstone 1993] [Simoncelli Adelson 1996 Chang et al. 2000). From a linear-algebraic perspective, these algorithms operate by projecting the noisy input onto lower-dimensional subspace that contains plausible signal content. The projection eliminates the orthogonal complement the subspace, which mostly contains noise. This general methodology laid the foundations for the state-of-the-ar models in the 2000's (e.g. (Dabov et al.) (006)). some of which added a data-driven perspective, learning sparsifying transforms (Elad & Aharon, 2006), and nonlinear shrinkage functions (Hel-Or Or & Shaked 2008; Raphan & Simoncelli] 2008). directly from natural images. Here, we show that deep-leamiig models learn similar priors in the form of local linear subspaces capturing image features.
developed based on multi-scale ("wavelet")
2

---

Published as a conference paper at ICLR 2020
MBO
W
Noisy training image, 10 (max level)
Noisy test image, 90
Test image, denoised by CNN
Test image, denoised by BF-CNN
Denoising of an example natural image by a CNN and its bias-free counterpart (BF-CNN), both trained over noise levels in the range [0, 10] (image intensities are in the range [0, 255)). The CNN performs poorly at high noise levels (a = 90, far beyond the training range), whereas BF-CNN performs at state-of-the-are levels. The CNN used for this example is DrCNN (Zhang et al.)
Figure 2:
2017:
using alternative architectures yields similar results (see Section
models based on convolutional neural networks (LeCun et al. 2015) have come to dominate all previous methods in terms of performance. These models consist of cascades of convolutional filters, and rectifying nonlinearities, which are capable of representing diverse and powerful set of functions. Training such architectures to minimize mean square error over large databases of noisy natural-image patches achieves current state-of-the-are results (Zhang
In the past decade, purely
ct al. 2017; Huang et al., 2017; Ronneberger et al.) 2015 Zhang et al. 2018a).
NETWORK BIAS IMPAIRS GENERALIZATION
We assume a measurement model in which images are corrupted by additive noise: y = x + n, where E RN is the original image, containing N pixels, is an image of i..dd. samples of Gaussian noise with variance 2², and y the noisy observation. The denoising problem consists of finding function f: N + R that provides good estimate of the original image, Commonly, one minimizes the mean squared error: = arg min, E|i - g(v)l12, where the expectation is taken over some distribution over images, æ, well as over the distribution of noise realizations. In deep learning, the denoising function is parameterized by the weights of the network, so the optimization is over these parameters. If the noise standard deviation, a, is unknown, the expectation must also be taken distribution of a. This problem often called blind denoising the literature. In this work, we study the generalization performance of CNNs across noise levels a, ie. when they are tested on noise levels not included in the training set.
Feedforward neural networks with rectified linear units (ReLUs) are piecewise affine: for given activation pattern of the ReLUs, the effect of the network on the input is a cascade of linear trans- formations (convolutional or fully connected layers. W₂), additive constants (bx), and pointwise multiplications by a binary mask corresponding to the fixed activation pattern (R). Since each of these is affine, the entire cascade implements a single affine transformation. For a fixed noisy input image y E TRA with N pixels, the function f R - N computed by denoising neural network may be written
f(v) = WLR
by) ....b-.)) + by = Ayy++ byy
where Ay E IR N the Jacobian of f-) evaluated input y. and by R represents the net bias. The subscripts A, and by serve as a reminder that both depend on the ReLU activation patterns, which in turn depend on the input vector y-
Based on equation [ we can perform first-order decomposition of the error or residual of the neural network for specific input: fy) = (I - A,,) - by. Figure shows the magnitude of the residual and the constant, which is equal the net bias by, for range of noise levels. Over the training range, the net bias is small, implying that the linear term responsible for most the denoising (see Figures 9 and for visualization of both components). However, when the network is evaluated noise levels outside of the training range, the norm the bias increases dramatically, and the residual significantly smaller than the noise, suggesting form of overfitting. Indeed, network performance
3

---

Published as a conference paper at ICLR 2020
Figure 3: Comparison of the performance of a CNN and BF-CNN with the same architecture for the experimental design described in Section The performance is quantified by the PSNR of the denoised image as a function of the input PSNR. Both networks are trained over fixed ranges of noise levels indicated by a blue background. robustly beyond the training range, while that of the CNN degrades significantly. The CNN used for this example is DrCNN (Zhang al. 2017); using alternative architectures yields similar results (see Figures 11 and 12
all cases, the performance of BF-CNN generalizes
generalizes very poorly to noise levels outside the training range. This is illustrated for an example image in Figure 2 and demonstrated through extensive experiments in Section
4 PROPOSED METHODOLOGY: BIAS-FREE NETWORKS
Section 3 shows that CNNs overfit to the noise levels present in the training set, and that this is associated with wild fluctuations of the net bias by This suggests that the overfitting might be ameliorated by removing additive (bias) terms from every stage the network, resulting in a bias- free CNN (BF-CNN). Note that bias terms are also removed from the batch-normalizatiio used during training. This simple change in the architecture has an interesting consequence. If the CNN has ReLU activations the denoising map is locally homogeneous, and consequently invariant to scaling: rescaling the input by a constant value simply rescales the output by the same amount, just it would for linear system.
Lemma 1. Let fBP RN > R be a feedforward neural network with ReLU activation functions and no additive constant terms in any layer. For any input E R and any nonnegative constant a.
fBF(og) = (Br()))
Proof. We can write the action bias-free neural network with L layers in terms of the weight matrix W,, i L, of each layer and rectifying operator R, which sets to zero any negative entries in its input. Multiplying by nonnegative constant does not change the sign the entries of a vector, for any with the right dimension and any a R(az) oR((), which implies
(ay(ag) = Wi.R(W (..Woo)) =
a W.W.W...-1 -...-- R(Wig)) = afFr
Note that networks with nonzero net bias are not scaling invariant because scaling the input may change the activation pattern of the ReLUs. Scaling invariance is intuitively desireable for denoising method operating on natural images; rescaled image is still an image. Note that hoolds for networks with skip connections where the feature maps concatenated added, because both of these operations are linear.
In the following sections demonstrate that removing all additive terms in CNN architectures has two important consequences: (1) the networks gain the ability to generalize to noise levels not encountered during training (as illustrated by Figure 2 the improvement is striking), and (2) the denoising mechanism can be analyzed locally via linear-algebraic tools that reveal intriguing ties to more traditional denoising methodology such as nonlinear filtering and aparsity-based techniques.
4

---

Published as a conference paper at ICLR 2020
Noisy
Denoised
Pixel
Pixel 2
Pixel
Pixol
10
Pixel
Pixel
30
100
Figure 4: Visualization of the linear weighting functions (rows of A, in equation of a BF-CNN for three example pixels of an input image, and three levels of noise. The images in the three rightmost columns show the weighting functions used to compute each of the indicated pixels (red squares). All weighting functions sum to one, and thus compute local average (note that some weights are negative, indicated in red). Their shapes vary substantially, and are adapted to the underlying image content. As the noise level a increases, the spatial extent of the weight functions increases in order to average out the noise, while respecting boundaries between different regions in the image, which results in dramatically different functions for each pixel. The CNN used for this example is DrCNN (Zhang et al., 2017); using alternative architectures yields similar results (see Figure 13
5 BIAS-FREE NETWORKS GENERALIZE ACROSS NOISE LEVELS
In order to evaluate the effect of removing the net bias in denoising CNNs, compare several state-of- the-art architectures to their bias-free counterparts, which are exactly the same except for the absence of any additive constants within the networks (note that this includes the batch-nommalization additive parameter). These architectures include popular features of existing neural-network techniques in image processing: recurrence, multiscale filters, and skip connections. More specifically, we examine the following models (see Section A for additional details):
DnCNN (Zhang et al. 2017): A feedforward CNN with 20 convolutional layers, each consisting of filters, channels, batch normalization (loffe Szegedy, 2015), ReLU nonlinearity, and skip connection from the initial layer to the final layer.
Recurrent CNN: recurrent architecture inspired by Zhang al, (2018a) where the basic module is a CNN with 5 layers, x 3 filters and 64 channels in the intermediate layers. The order of the recurrence
UNet (Ronneberger et al. 2015): A multiscale architecture with 9 convolutional layers and skip connections between the different scales.
Simplified DenseNet: CNN with skip connections inspired by the DenseNet architec- ture (Huang et al., 2017 Zhang et al. 2018b).
We train each network to denoise images corrupted by i.i.d. Gaussian noise over a range standard deviations (the training range of the network). We then evaluate the network for noise levels that both within and beyond the training range. Our experiments are carried out on 180 x 180 natural images from the Berkeley Segmentation Dataset (Martin et al. 2001 be consistent with previous

---

Published as a conference paper at ICLR 2020
350
600
ave
6001
singular valuus
250
600
effective
2001
1.3
150
6.3
hetween
level
(a)
(b)
Figure 5: Analysis of the SVD of the Jacobian of BF-CNN for ten natural images, corrupted by noise of standard deviation a = 50. (a) Singular value distributions. For all images, a large proportion of the values signal subspace). (b) Histogram of dot products (cosine of angle) between the left and right singular vectors that lie within the signal subspaces. (c) Effective dimensionality of the signal subspaces (computed as sum of squared singular values) as a function noise level. For comparison, the total dimensionality
zero, indicating (approximately) projection onto subspace (the
the space 1600 (40 40 pixels). Average dimensionality (red curve) falls approximately as the inverse of (dashed curve). The CNN used for this example is DnCNN (Zhang al. 2017) ; using alternative architectures yields similar results (see Figure
results (Schmidt & Roth 2014 Chen Pock 2017 Zhang al. 2017). Additional details about the dataset and training procedure are provided in Section
Figures and show our results. For wide range different training ranges, and for architectures, we observe the same phenomenon: the performance of CNNs is good over the training range, but degrades dramatically at new noise levels; in stark contrast, the corresponding BF-CNNs provide strong denoising performance over noise levels outside the training range. This holds for both PSNR and the more
meappttnlll--mmnnnfuuullll Structural Similarity Index (Wang et al. 2004) (see Figure 12]. Figure 2 shows example image, demonstrating visually the striking difference in generalization performance between a CNN and its corresponding BF-CNN. Our results provide strong evidence that removing bias in CNN architectures results in effective generalization to noise levels out of the training range.
REVEALING THE DENOISING MECHANISMS LEARNED BY BF-CNNS
In this section we perform a local analysis of BF-CNN networks, which reveals the underlying denoising mechanisms learned from the data. A bias-free network is strictly linear, and its net action can be expressed as
SBE (v) WWR(W--...WWWW
Ayy.
where Ay is the Jacobian of (FF(.) evaluated at y. The Jacobian at fixed input provides local characterization of the denoising map. In order study the map perform a linear-algebrai analysis of the Jacobian. Our approach is similar in spirit to visualization approaches- proposed in the context of image classification- that differentiate neural-network functions with respect to their input (e.g. (Simonyan et al. (2013) Montavon et al. (2017)).
6.1 NONLINEAR ADAPTIVE FILTERING
The linear representation of the denoising map given by equation implies that the vth pixel of the output image is computed as an inner product between the ith row of Ay, denoted (i), and the input image:
(ur(y)(1) = Σ A ai)
(5)
1=1

---

Published as a conference paper at ICLR 2020
282 282
Figure 6: Visualization of left singular vectors of the Jacobian of a BF-CNN, evaluated on two different images (top and bottom rows), corrupted by noise with standard deviation o 50. The left column shows original (clean) images. The next three colurns show singular vectors corresponding negligile singular values. The vectors capture features from the clean image. The last three columns on the right show singular vectors corresponding to singular values that are almost equal to zero. These vectors are noisy and unstructured. The CNN used for this example DrCNN (Zhang et al. 2017): ; using alternative architectures yields similar results (see Figure 16).
The vectors (i) can be interpreted as adaptive filters that produce an estimate of the denoised pixel via a weighted average of noisy pixels. Examination of these filters reveals their diversity, and their relationship to the underlying image content: they are adapted to the local features of the noisy image, averaging over homogeneous regions of the image without blurring across edges. This shown for
two separate examples and range of noise levels in Figures 4 13 14 and 15 for the architectures described in Section 5
We observe that the equivalent filters of all architectures adapt to image
structure.
Classical Wiener filtering (Wiener 1950) denoises images by computing a local average dependent on the noise level. As the noise level increases, the averaging is carried out over a larger region. As illustrated by Figures [4] 13| 14 and [5] the equivalent filters of BF-CNNs also display this behavior. adaptive. The BF-CNNs learn such filters implicitly from the data, in the spirit of modern nonlinear spatially-varying filtering techniques designed to preserve fine-scale details such edges (e.g. Tomasi & Manduchi (1998), also Milanfar (202)) for comprehensive review, and Choi et al. (2018) for a recent suiming-based approach).
The crucial difference that the filters
6.2 PROJECTION ONTO ADAPTIVE LOW-DIMENSIONAL SUBSPACES
The local linear structure of a BF-CNN facilitates analysis of its functional capabilities via the singular value decomposition (SVD). For a given input y. we compute the SVD of the Jacobian matrix: A, USV with U and V orthogonal matrices, and S a diagonal matrix. We can decompose the effect of the network on its input in terms of the left singular vectors (U1, U2..., Un} (columns of U), the singular values {si, $2... . sN} (diagonal elements of 5), and the right singular vectors (Vi V2.... VN) (columns of V):
fBF = = USVT = Σ sVVV v)u,
(6)
The output is linear combination of the left singular vectors, each weighted by the projection of the input onto the corresponding right singular vector, and scaled by the corresponding singular value.
Analyzing the SVD of BF-CNN on a set of ten natural images reveals that most singular values are very close to zero (Figure Sap. The network is thus discarding all but a very dow-dimensional portion of the input image. We also observe that the left and right singular vectors corresponding to the singular values with non-negligible amplitudes are approximately the same (Figure(5b) This means that the Jacobian is (approximately) symmetric, and we can interpret the action of the network as projecting the noisy signal onto a www-dimensional subspace, as is done in wavelet thresholding schemes. This is confirmed by visualizing the singular vectors images (Figure 6). The singular vectors corresponding to neg-negligible singular values are seen to capture features of the input image; those corresponding near-zero singular values unstructured. The BF-CNN therefore implements
7

---

Published as a conference paper at ICLR 2020
nistednuss
8.4
0.0
nobse level
Bodoc Jevel of the nested subspace
Figure 7: Signal subspace properties. Left: Signal subspace, computed from Jacobian of a BF-CNN
evaluated at a particular noise level, contains the clean image. Specifically, the fraction of squared & norm preserved by projection onto the subspace is nearly one as grows from 10 to 100 (relative to the image pixels, which lie in the range [0, 255). Results are averaged over 50 example clean images. Right: Signal subspaces different noise levels are nested. The subspace axes for higher noise level lie largely within the subspace obtained for the lowest noise level (a = 10), as measured by the sum of squares of their projected norms. Results are shown for 10 example clean images.
an approximate projection onto an adaptive signal subspace that preserves image structure, while suppressing the noise.
We can define an "effective dimensionality" of the signal subspace as d = Σi= of the amount of variance captured by applying the linear map to N-dimensional Gaussian noise vector with variance o², normalized by the noise variance. The remaining variance equals
N
N
E, |ggn E,, V, En11S,n11 = En
E,
Σ
S En
= 0²
1=1
1=1
where E, indicates expectation over noise n, so that d E, | 1//113//222
N
When examine the preserved signal subspace, find that the clean image lies almost completely within it. For inputs of the form y/ := x + n (where x is the clean image and n the noise), we find that the subspace spanned by the singular vectors up to dimension d contains almost entirely, in the sense that projecting x onto the subspace preserves most of its energy. This holds for the whole range of noise levels over which the network is trained (Figure
We also find that for any given clean image, the effective dimensionality of the signal subspace (d) decreases systematically with noise level (Figure 5cp. At lower noise levels the network detects richer set of image features, and constructs larger signal subspace to capture and preserve them. Empirically, we found that (on average) d is approximately proportional to a (see dashed line in Figure Fc). These signal subspaces are nested: the subspaces corresponding to lower noise levels contain more than 95% of the subspace axes corresponding to higher noise levels (Figure 7
Finally, we note that this behavior of the signal subspace dimensionality, combined with the fact that it contains the clean image, explains the observed denoising performance across different noise levels (Figure 3). Specifically, if we assume d z a/a, the mean squared error is proportional to a:
MSE-
Ex Ax +n) x11
F,, |
2 2 d
2 200
Note that this result runs contrary to the intuitive expectation that MSE should be proportional to the noise variance, which would be the case if the denoiser operated by projecting onto a fixed subspace. The scaling of MSE with the square root of the noise variance implies that the PSNR of the denoised image should be a linear function of the input PSNR, with slope of 1/2, consistent with the empirical results shown in Figure 3 Note that this behavior holds even when the networks trained only on modest levels of noise (e.g., o E [0, 10).
8

---

Published as a conference paper at ICLR 2020
7 DISCUSSION
In this work, we show that removing constant terms from CNN architectures ensures strong generaliza- tion across noise levels, and also provides interpretability of the denoising method via linear-algebra techniques. We provide insights into the relationship between bias and generalization through set of observations. Theoretically, we argue that if the denoising network operates by projecting the noisy observation onto a linear space of "clean" images, then that space should include all rescalings of those images, and thus, the origin. This property can guaranteed by eliminating bias from the network. Empirically, in networks that allow bias, the net bias of the trained network is quite small within the training range. However, outside the training range the net bias grows dramatically resulting poor performance, which suggests that the bias may be the cause of the failure general- ize. In addition, when we remove bias from the architecture, we preserve performance within the training range, but achieve wear-perfect generalization, the training range. These observations do not fully elucidate how our network achieves its remarkable generalization- only that bias prevents that generalization, and its removal allows it.
noise levels more than 10x those in
is of interest to examine whether bias removal can facilitate generalization in noise distributions beyond Gaussian, well other paage-procsssing tasks, such as image restoration and image compression. We have trained bias-free networks on uniform noise and found that they generalize outside the training range. In fact, bias-free networks trained for Gaussian noise generalize well when tested on uniform noise (Figures 18 and 19). In addition, we have applied our methodology to image restoration (simultaneous deblurring and denoising). Preliminary results indicate that bias-free networks generalize across noise levels for fixed blur level, whereas networks with bias do not (Figure 20). An interesting question for future research whether possible to achieve generalization across blur levels. Our initial results indicate that removing bias is not sufficient to achieve this.
Finally, linear-algebraic analysis uncovers interesting aspects of the denoising map, but these interpretations are very local: small changes in the input image change the activation patterns of the network, resulting in a change in the corresponding linear mapping. Extending the analysis to reveal global characteristics of the neural-network functionality is a challenging direction for future research.
ACKNOWLEDGEMENTS
This work was partially supported by the Howard Hughes Medical Institute (HHMI).
REFERENCES
S Grace Chang, Bin Yu, and Martin Vetterli. Adaptive wavelet thresholding for image denoising and compression. IEEE Trans. Image Processing, 99): 2000.
Yunjin Chen and Thomas Pock. Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration. IEEE Trans. Patt. Analysis and Machine Intelligence, 39(6): 1256-1272. 2017.
Sungjoon Choi, John Isidoro, Pascal Getreuer, and Peyman Milanfar. Fast, trainable, multiscale denoising. In 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 963-967. IEEE, 2018.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising with block-matching and 3d filtering. In Image Processing: Algorithms and Systems, Neural Networks, and Machine Learning, volume 6064, pp. 606414, International Society for Optics and Photonics, 2006.
D Donoho and I Johnstone. Adapting to unknown smoothness via wavelet shrinkage. J American Stat Assoc, 90(432), December 1995.
Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. on Image processing, (5(12):333-3775 2006.

---

Published as a conference paper at ICLR 2020
Y Hel-Or and D Shaked. A discriminative approach for wavelet denoising. IEEE Trans. Image Processing, 2008.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp- 4700-4708, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 1rXiv:1502.03166 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14/2,6990 2014.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning, nature, 521(7553):436, 2015.
D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application evaluating segmentation algorithms and measuring ecological statisties. Proc. 8th Int'l Conf. Computer Vision, volume 2. pp. 416-423, July 2001.
Peyman Milanfar. A tour modern image filtering: New insights and methods, both practical and theoretical. IEEE signal processing magazine, 30(1):106-122, 2012.
Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus-Robert Müller. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Rec., 65:21 1- 222, 2017.
M Raphan and E P Simoncelli. Optimal denoising in redundant representations. IEEE Trans Image Processing, 17(8):1322-133 1352, Aug 2008. doi: 10. 10.1109/1P.2008.925533
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Uwe Schmidt and Stefan Roth. Shrinkage fields for effective image restoration. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2774-2781, 2014.
E. Simoncelli and H Adelson. Noise removal via Bayesian wavelet coring. Proc 3rd IEEE Int' II Conf Image Proc, volume I, pp. 379-382, Lausanne, Sep 16-19 1996. IEEE Sig Proc Society. doi: N0.1109//CIP.1199..55
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint 13Xiv:1312.6034, 2013.
Carlo Tomasi and Roberto Manduchi. Bilateral filtering for gray and color images. In ICCV, volume 98, 1998.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Processing, 13(4):600-6) 6120 2004.
Norbert Wiener. Extrapolation, interpolation, and smoothing of stationary time series: with engi- neering applications. Technology Press, 1950.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng. and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep CNN for image denoising. JEEE Trans. Image Processing, 26(7): 3142-3155, 2017.
Xiaoshuai Zhang, Yiping Lu, Jiaying Liu, and Bin Dong. Dynamically unfolding recurrent restorer: A moving endpoint control method for image restoration. arXiv preprint arXiv: 1805.07709 201 8a.
Yulun Zhang, Yapeng Tian, Yu Kong. Bineng Zhong, and Yun Fu. Residual dense network for image restoration. CoRR, abs/ 112. 1047., 2018b. URL htt.:tt.//aaaa...cr///////2..../
10

---

Published as a conference paper at ICLR 2020
A DESCRIPTION OF DENOISING ARCHITECTURES
In this section we describe the denoising architectures used for our computational experiments in more detail.
A.1 DNCNN
We implement BF-DnCNN based on the architecture of the Denoising CNN (DnCNN) Zhang al. 2017). DrCNN consists of 20 convolutional layers, each consisting of filters and 64 channels, batch normalization (Toffe & Szegedy] 2015), and ReLU nonlinearity. It has a skip connection from the initial layer to the final layer, which has no nonlinear units. To construct bias-free DrCNN (BF-DnCNN) remove all sources of additive bias, including the mean parameter of the batch-normalizztion every layer (note however that scaling parameter is preserved).
A.2 RECURRENT CNN
Inspired by Zhang et al. (2018a), consider a recurrent framework that produces a denoised image estimate of the form スG f(t Ynoisy), at time where f is a neural network. We use 5-layer fully convolutional network with 3 filters in all layers and 64 channels in each intermediate layer to implement f. We initialize the denoised estimate as the noisy image, i.e fo == Unoisy. For the version of the network with net bias, we add trainable additive constants every filter in all but the last layer. During training, we run the recurrence for maximum of T times, sampling T uniformly random from 21,2,3,4 4] for each mini-batch. At test time we fix T =
A.3 UNET
Our UNet model (Ronneberger al. 2015) has the following layers:
1. conv/ Takes in input image and maps to 32 channels with 5 x 5 convolutional kernels.
2. conv2 Input: 32 channels. Output: 32 channels.
convolutional kernels.
3. conv3 Input: 32 channels. Output: 64 channels. 3 convolutional kernels with stride 2.
4. conv4 Input: 64 channels. Output: 64 channels. 3 x convolutional kemels..
5. conv5- Input: 64 channels. Output: 64 channels. 3 x 3 convolutional kernels with dilation factor 2.
6. convó- Input: 64 channels. Output: 64 channels. 3 x 3 convolutional kernels with dilation factor 4.
7. conv7- Transpose Convolution layer. Input: 64 channels. Output: 64 channels. 4 filters with stride 2.
8. conv8- Input: 96 channels. Output: 64 channels. 3 x 3 convolutional kernels. The input to this layer is the concatenation of the outputs of layer conv7 and conv2.
9. conv9- Input: 32 channels. Output: 1 channels. 5 x 5 convolutional kernels.
The structure is the same as in Zhang et al. (2018)), but without recurrence. For the version with bias, add trainable additive constants to all the layers other than comv9. This configuration of UNet assumes even width and height, so we remove one row or column from images in with odd height or width.
A.4 SIMPLIFIED DENSENET
Our simplified version of the DenseNet architecture (Huang et al.) 2017) has 4 blocks in total. Each block is fully convolutional 5-layer CNN with 3 filters and 64 channels in the intermediate layers with ReLU nonlinearity. The first three blocks have an output layer with 64 channels while the last block has an output layer with only one channel. The output the thh block is concatenated with the input noisy image and then fed to the (i + 1)th block, the last three blocks have 65 input channels. the version of the network with bias, we add trainable additive parameters to all the layers except for the last layer in the final block.

---

Published as a conference paper at ICLR 2020
B DATASETS AND TRAINING PROCEDURE
Our experiments are carried out on 180 x 180 natural images from the Berkeley Segmentation Dataset (Martin al. 2001). We use training of 400 images. The training set augmented via downsampling, random flips, and random rotations of patches in these images (hang et al.) 2017). A test containing 68 images is used for evaluation. We train the DnCNN and it's bias free model on patches of size 50 x 50, which yields a total of 541 ,600 clean training patches. For the remaining architectures, we use patches of size 128 x 128 for total of 22,400 training patches.
We train DrCNN and its bias-free counterpart using the Adam Optimizer (Kingma & Ba 2014) over 70 epochs with an initial learning rate of 10-3 and decay factor of 0.5 at the 50 and 60 epochs, with no early stopping. We train the other models using the Adam optimizer with an initial learning rate of 10-3 and train for 50 epochs with a learning rate schedule which decreases by a factor 0.25 if the validation PSNR decreases from one epoch to the next. We use early stopping and select the model with the best validation PSNR.
C ADDITIONAL RESULTS
In this section we report additional results of our computational experiments:
Figure g shows the first-order analysis of the residual of the different architectures described in Section A
except for DrCNN which is shown in Figure [
Figures and 10 visualize the linear and net bias terms in the first-order decomposition of
an example le image at different noise levels.
Figure
shows the PSNR results for the experiments described in Section
Figure 12 shows the SSIM results for the experiments described in Section
Figures
14 and 13 show the equivalent filters at several pixels of two example images for different architectures (see Section
1
Figure shows the singular vectors of the Jacobian of different BF-CNNs (see Section 6.2
Figure 17 shows the singular values of the Jacobian of different BF-CNNS (see Section 6.2).
Figure 18 and 19 shows that networks trained on noise samples drawn from Gaussian distribution with 0 mean generalizes to noise drawn from uniform distribution with 0 mean during test time. Experiments follow the procedure described in Section except that the networks are evaluated on a different noise distribution during the test time.
Figure 20 shows the application of BF-CNN and CNN the task of image restoration, where the image is corrupted with both noise and blur at the same time. We show that BF-CNNs can generalize outside the training range for noise levels for fixed blur level, but do not outperform CNN when generalizing unseen blur levels.
12

---

Published as a conference paper at ICLR 2020
120
R-CNN
26
320
326
120
UNet
26
15
redaak--2
123
325
05 55
Figure 8: First-order analysis of the residual of RecurrentCCNN (Section A.2 and DenseNet (Section AA.4) as a function of noise level. The plots show the magnitudes of the residual and the bias averaged over 68 images in Set68 test set of Berkeley Segmentation Dataset (Martin et al.] 2001) for networks trained over different training ranges. The range of noises used for training highlighted in gray. (left) When the network is trained
UNet (Section A.3
the full range of noise levels (a E jo, 100) the net bias is small, growing slightly as the noise increases. (middle and right) When the network trained explosively for noise levels outside the training range. This coincides with the dramatic drop in performance due overfitting, reflected in the difference between the residual and the true noise.
the smaller range E [0, 55] and a E [0, 30)), the net bias grows
13

---

Published as a conference paper at ICLR 2020
Noisy Input (y)
Denoised (f())
Linear Part (Ay3)
Net Bias (by)
50
70"
Figure 9: Visualization of the decomposition output of DnCNN trained for noise range [0, 55] into linear part and net bias. The noise level o 70 (highlighted by *) is outside the training range. Over the training range. the net bias is small, and the linear part is responsible for most of the denoising effort. However, when the network is evaluated out of the training range, the contribution of the bias increases dramatically, which coincides with significant drop in denoising performance.
14

---

Published as a conference paper at ICLR 2020
Noisy Input (1)
Denoised (f(w))
Linear Part (Ag3)
Net Bias (b,)
R-CNN
400
-0.9
CNN 90"
10
-0.13
UNet 90"
DenseNet a =
DenseNet 90"
Figure 10: Visualization of the decomposition of output of Recurrent-CNN (Section A.2 UNet (Section A.3 and DenseNet (Section A.4) trained for noise range [0, 55] into linear part and bias. The noise level = 90 (highlighted by *) is outside the training range. Over the training range, the net bias small, and the linear part is responsible for most of the denoising effort. However, when the network is evaluated out of the training range, the contribution of the bias increases dramatically, which coincides with a significant drop in denoising performance.
15

---

Published as a conference paper at ICLR 2020
8
(a)
(b)
Figure 11: Comparisons of architectures with (red curves) and without (blue curves) net bias for experimental design described in Section The performance quantified by the PSNR of the denoised image as function of the input PSNR of the noisy image. All the architectures with bias perform poorly out their training range, whereas the bias-free versions all achieve excellent generalization across noise levels. (a) Deep Convolutional Neural Network, DnCNN (Zhang et al.) 2017). (b) Recurrent architecture inspired by DURR (Zhang et al. 2018a). (c) Multiscale architecture inspired by the UNet (Ronneberger et al. 2015). (d) Architecture with multiple skip connections
inspired by the DenseNet (Huang et al. 201 17).
(b)
(d)
Figure 12: Comparisons of architectures with (red curves) and without (blue curves) net bias for the experimental design described in Section The performance quantified by the SSIM of the denoised image as a function of the input SSIM of the noisy image. All the architectures with bias perform poorly out their training range, whereas bias-free versions achieve excellent generalization across noise levels. (a) Deep Convolutional Neural Network, DnCNN (Zhang et al. 2017).. (b) Recurrent architecture inspired by DURR (Zhang et al. 2018a). (c) Multiscale architecture inspired by the UNet (Ronneberger et al. 2015). (d) Architecture with multiple skip connections inspired by the DenseNet (Huang et al. 2017
16

---

Published as a conference paper at ICLR 2020
Denoised (f(y) = A,y)
Noisy Input(y)
Pixel 1
Pixel
Pixel
55
Figure 13: Visualization of the linear weighting functions (rows of A,) of Bias-Free Recurrent-CNN (top rows) (Section A.2). Bias-Free UNet (next rows) (Section A.3) and Bias-Free DenseNet (bottom 2 rows) (Section A.4) for three example pixels of noisy input image (left). The next image the denoised output. The three images on the right show the linear weighting functions corresponding to each of the indicated pixels (red squares). All weighting functions sum one, and thus compute local average (although some weights are negative, indicated red). Their shapes vary substantially, and adapted to the underlying image content. Each row corresponds to noisy input with increasing a and the filters adapt by averaging over a larger region.
17

---

Published as a conference paper at ICLR 2020
Denoised (f(v) A,y)
Noisy Input(y)
Pixel
Pixel
Pixel
35
65
Figure 14: Visualization of the linear weighting functions (rows Ay) of a BF-DrCNN for three example pixels of a noisy input image (left). The next image is the denoised output. The three images on the right show the linear weighting functions corresponding to each of the indicated pixels (red squares). All weighting functions sum to one, and thus compute local average (although some weights are negative, indicated in red). Their shapes vary substantially, and are adapted to the underlying image content. Each row corresponds to noisy input with increasing and the filters adapt by averaging over a larger region.
18

---

Published as a conference paper at ICLR 2020
Denoised (f()) = A+y)
Noisy Input(y)
Pixel 1
Pixel
Pixel
55
Figure 15: Visualization of the linear weighting functions (rows of A₂) of Bias-Free Recurrent-CNN (top rows) (Section A.2). Bies-Free UNet (next rows) (Section A.3) and Bias-Free DenseNet (bottom 2 rows) (Section A.4) for three example pixels of noisy input image (left). The next image the denoised output. The three images on the right show the linear weighting functions corresponding to each of the indicated pixels (red squares). All weighting functions sum one, and thus compute local average (although some weights are negative, indicated in red). Their shapes vary substantially, and adapted to the underlying image content. Each row corresponds to noisy input with increasing a and the filters adapt by averaging over a larger region.
19

---

Published as a conference paper at ICLR 2020
282
282288 88
282
2
62282288
8
282 202232208
Figure 16: Visualization of left singular vectors of the Jacobian of BF Recurrent CNN (top rows), BF UNet (next rows) and BF DenseNet (bottom 2 rows) evaluated on three different images, corrupted by noise with standard deviation The next three columns show singular vectors corresponding to non negligglle singular values. The vectors capture features from the clean image. The last three columns on the right show singular vectors corresponding to singular values that are almost equal to zero. These vectors are noisy and unstructured.
25. The left column shows original (clean) images.
2.5
Sirguler value
2.0
singuler velue
1.5
1.0
0.5
0.5
0.0
500
1000
1500
2000
2500
0
500
10000
1500
2000
2500
500
1000
1500
2000
2500
axts number
axis number
axts number
(a)
(b)
(c)
Figure 17: Analysis of the SVD of the Jacobian of BF-CNN for ten natural images, corrupted by noise of standard deviation 50. For all images, large proportion of the singular values near zero, indicating (approximately) projection onto subspace (the signal subspace). (a) Recurrent architecture inspired by DURR (Zhang et al.) 2018a). (b) Multiscale architecture inspired by the UNet (Ronneherger et al. 2015). (c) Architecture with multiple skip connections inspired by the DenseNet (Huang al. (017).
20

---

Published as a conference paper at ICLR 2020
BrENE
Figure 18: Comparison of the performance of CNN and a BF-CNN with the same architecture for the experimental design described in Section 5 The networks trained using i.i.d. Gaussian noise but evaluated on noise drawn i.i.d. from a uniform distribution with mean 0. The performance is quantified by the PSNR of the denoised image as function of the input PSNR of the noisy image. All the architectures with bias perform poorly out of their training range, whereas the bias-free versions all achieve excellent generalization across noise levels, i.e. they are able to generalize across the two different noise distributions. The CNN used for this example is DrCNN (Zhang et al. 2017): using alternative architectures yields similar results (see Figures
50
18
005 23
522
2
8
(a)
(b)
(c)
(d)
Figure 19: Comparisons architectures with (red curves) and without (blue curves) a net bias for the experimental design described in Section 5) The networks are trained using i.i.d. Gaussian noise but evaluated on noise drawn i.i.d. from a uniform distribution with mean 0. The performance is quantified by the PSNR of the denoised image function of the imput PSNR of the noisy image. All the architectures with bias perform poorly out their training range, whereas the bias-free free versions all achieve excellent generalization across noise levels, i.e. they able generalize across the two different noise distributions. (a) Deep Convolutional Neural Network, DnCNN (Zhang et al. 2017). (b) Recurrent architecture inspired by DURR (Zhang al. 2018a). (c) Multiscale architecture inspired by the UNet (Ronneberger et al. 2015). (d) Architecture with multiple skip connections inspired by the DenseNet (Huang al. (017).
21

---

Published as a conference paper at ICLR 2020
40
16-
14-
moise
PSNR
Figure 20: Comparison of the performance of DnCNN and a corresponding BF-CNN for image restoration. Training is carried out data corrupted with Gaussian noise Tnoise E [0. 55) and Gaussian blur blur E [0, 4]. Performance is measured on test data for inside and outside the training ranges. Left: The difference training region is illustrated by the rectangular boundary. Bias-fre free network generalizes across noise levels each fixed blur levels, whereas DrCNN does not. However, BF-CNN does not generalize across blur levels. Right: A horizontal slice of the left plot for fixed blur level of Ohtur 2. BF-CNN generalizes robustly beyond the training range, while the performance of DnCNN degrades significantly.
performance measured APSNR PSNRgF-CNN PSNNR The
22


Reviews:
[{"review_id": "SyeZPBHTYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed to remove all bias terms in denoising networks to avoid overfitting when different noise levels exist. With analysis, the paper concludes that the dimensions of subspaces of image features are adaptively changing according to the noise level. An interesting result is that the MSE is proportional to sigma instead of sigma^2 when using bias-free networks, which provides some theoretical evidence of advantage of using BF-CNN.\n\nOne main practical concern is that only Gaussian noise is considered in this paper which provides good theoretical analysis. It would be interesting to see if this BF-CNN is extendable to more noise types.\n\nOne unclear statement is that for UNet, the Lemma 1 seems to be no longer valid as skip connections exist. Can you provide additional proof that the scale invariance still holds in this case?\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJeF98JCtS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the generalization properties of convolutional neural networks for image denoising. This paper shows that removing constant terms from CNN models provides strong generalization across noise levels. Also, this paper provides the interpretability of the denoising model based on a linear-algebra method.\n\nQ1. Is there the possibility that the model without the constant term shows strong generalization in other image processing tasks such as image deblurring and dehazing? It would be better to discuss this point to make the scope of the proposed method clear.\n\nQ2. About the visualization of the linear weighting functions in Figure 4, what can we figure out from the visualization, especially for image denoising? If possible, please elaborate on it.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HklObMGU9r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper looks at how deep convolutional neural networks for image denoising can generalize across various noise levels. First, they argue that state-of-the-art denoising networks perform poorly outside of the training noise range. The authors empirically show that as denoising performance degrades on unseen noise levels, the network residual for a specific input is being increasingly dominated by the network bias (as opposed to the purely linear Jacobian term). Therefore, they propose using bias-free convolutional neural networks for better generalization performance in image denoising. Their experimental results show that bias-free denoisers significantly outperform their original counter-parts on unseen noise levels across various popular architectures. Then, they perform a local analysis of the bias-free network around an input image that is now a strictly linear function of the input. They empirically demonstrate that the Jacobian is approximately low-rank and symmetric, therefore the effect of the denoiser can be interpreted as a nonlinear adaptive filter that projects the noisy image onto a low-dimensional signal subspace. The authors show that most of the energy of the clean image falls into the signal subspace and the effective dimensionality of this subspace is inversely proportional to the noise level.\n\nEven though it is theoretically not too well-motivated in the paper why the bias term degrades generalization performance, the experimental results seem to clearly demonstrate the merit of bias-free denoisers. Moreover, the analysis of the network Jacobian and its interpretation as a nonlinear adaptive filter provides some interesting insight in the local properties of bias-free denoisers. Therefore, I would recommend accepting this paper, if the authors provide a theoretical discussion on why the bias term might degrade generalization performance.\n\nSome smaller comments: \n-It is not clear if d should be multiplied by \\sigma^2 in its definition on page 7. The definition mentions dependence on noise variance, but the formula does not have it.\n-In Section 3 in the expression of the mean squared error it is not defined what g(y) means.\n-Axis labels are missing on Fig. 3.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on studying neural network-based denoising methods. The paper makes the interesting observation that most existing denoising approaches have a tendency to overfit to knowledge of the noise level. The authors claim that simply removing the bias on the network parameters enables a variety of improvements in this regard and provide some theoretical justification for their results. The reviewers were mostly postive but raised some concerns about generalization beyond Gaussian noise and not "being very well theoretically motivated". These concerns seem to have at least partially been alleviated during the discussion period. I agree with the reviewers. I think the paper looks at an important phenomena for denoising (role of variance parameter) and is well suited to ICLR. I recommend acceptance. I suggest that the authors continue to further improve the paper based on the reviewers' comments.
Decision: Accept (Poster)