Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper looks at how deep convolutional neural networks for image denoising can generalize across various noise levels. First, they argue that state-of-the-art denoising networks perform poorly outside of the training noise range. The authors empirically show that as denoising performance degrades on unseen noise levels, the network residual for a specific input is being increasingly dominated by the network bias (as opposed to the purely linear Jacobian term). Therefore, they propose using bias-free convolutional neural networks for better generalization performance in image denoising. Their experimental results show that bias-free denoisers significantly outperform their original counter-parts on unseen noise levels across various popular architectures. Then, they perform a local analysis of the bias-free network around an input image that is now a strictly linear function of the input. They empirically demonstrate that the Jacobian is approximately low-rank and symmetric, therefore the effect of the denoiser can be interpreted as a nonlinear adaptive filter that projects the noisy image onto a low-dimensional signal subspace. The authors show that most of the energy of the clean image falls into the signal subspace and the effective dimensionality of this subspace is inversely proportional to the noise level.

Even though it is theoretically not too well-motivated in the paper why the bias term degrades generalization performance, the experimental results seem to clearly demonstrate the merit of bias-free denoisers. Moreover, the analysis of the network Jacobian and its interpretation as a nonlinear adaptive filter provides some interesting insight in the local properties of bias-free denoisers. Therefore, I would recommend accepting this paper, if the authors provide a theoretical discussion on why the bias term might degrade generalization performance.

Some smaller comments: 
-It is not clear if d should be multiplied by \sigma^2 in its definition on page 7. The definition mentions dependence on noise variance, but the formula does not have it.
-In Section 3 in the expression of the mean squared error it is not defined what g(y) means.
-Axis labels are missing on Fig. 3.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper studies the generalization properties of convolutional neural networks for image denoising. This paper shows that removing constant terms from CNN models provides strong generalization across noise levels. Also, this paper provides the interpretability of the denoising model based on a linear-algebra method.

Q1. Is there the possibility that the model without the constant term shows strong generalization in other image processing tasks such as image deblurring and dehazing? It would be better to discuss this point to make the scope of the proposed method clear.

Q2. About the visualization of the linear weighting functions in Figure 4, what can we figure out from the visualization, especially for image denoising? If possible, please elaborate on it.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposed to remove all bias terms in denoising networks to avoid overfitting when different noise levels exist. With analysis, the paper concludes that the dimensions of subspaces of image features are adaptively changing according to the noise level. An interesting result is that the MSE is proportional to sigma instead of sigma^2 when using bias-free networks, which provides some theoretical evidence of advantage of using BF-CNN.

One main practical concern is that only Gaussian noise is considered in this paper which provides good theoretical analysis. It would be interesting to see if this BF-CNN is extendable to more noise types.

One unclear statement is that for UNet, the Lemma 1 seems to be no longer valid as skip connections exist. Can you provide additional proof that the scale invariance still holds in this case?


review_title: Official Blind Review #2

Meta-review: This paper focuses on studying neural network-based denoising methods. The paper makes the interesting observation that most existing denoising approaches have a tendency to overfit to knowledge of the noise level. The authors claim that simply removing the bias on the network parameters enables a variety of improvements in this regard and provide some theoretical justification for their results. The reviewers were mostly postive but raised some concerns about generalization beyond Gaussian noise and not "being very well theoretically motivated". These concerns seem to have at least partially been alleviated during the discussion period. I agree with the reviewers. I think the paper looks at an important phenomena for denoising (role of variance parameter) and is well suited to ICLR. I recommend acceptance. I suggest that the authors continue to further improve the paper based on the reviewers' comments.
Decision: Accept (Poster)