Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a new approach based on Tweedie's formula to perform self-supervised image denoising. The proposed approach has two steps: (1) learn the score function or gradient of density of the noisy images, and (2) use Tweedie's formula to compute a denoised image using the learned score function. The proposed approach builds on multiple relevant work in the literature, and is well motivated. The model is tested on Gaussian, Poisson and Gamma noise removal tasks on natural images and achieves competitive results. 
main_review: The paper is well organized, and clearly written. The experiments are well thought and supports the main claim of the paper. Please find my specific comments below:

1. **Missing baseline in Table 2**. The authors have done a thorough job of comparison, however, I think one of the relevant method is missing. N2V and N2S do not exploit noise type while learning the denoising function, and in general suffer from the drawback that they do not use the value of central pixel. Several post-processing schemes have been introduced to rectify this and incorporate the value of the central pixel back into the denoised output by assuming that the noise type is known. Given that the proposed method exploits the noise type, I think it is unfair to leave such methods out of comparison. For example, [High-Quality Self-Supervised Deep Image Denoising](https://arxiv.org/abs/1901.10277), Laine et. al, NeurIPS 2019 incorporates blind-spot in N2V and N2S into the architecture and has a post-processing scheme for Poisson and Gaussian noise. This post-processing scheme (again based on a Bayesian view) is shown to provide non-trivial improvement in denoising performance. When the noise type is unknown, Noise2Same [5] is a better baseline than N2V and N2S, since it promotes the use of center pixel. 

2. **Relation with Noise2X**. I appreciate the authors providing context and illuminating the relationship of their method with existing methods in the literature. However, I think the relation with Noise2Void discussed in lines 191-198 is force fitting a narrative. The main idea behind masking schemes in Noise2Void and Noise2Self is to discourage the use of central pixel in the denoising function, and it looks like this is a non-trivial interpretation to get from score matching perspective. In my understanding, the practical version of masking used in Noise2Void, pixel values are replaced with a random pixel from the neighbourhood. However, the proposed method maybe more releted to Noiser2Noise (see below). Also, one of the other strong points claimed by the authors is the university of the score matching cost function for different noise types. But the relation is explained only using SURE. I think there is no easy interpretation for PURE or other noise types, which maybe worth highlighting in the paper for an informed reader. 

3. **Noise2X and SURE beiing suboptimal for non-Gaussian**. In my understanding, Noise2Void or other blind-spot methods only make the assumption that the noise is pixel-wise independent. They are not even optimal for Gaussian denoising. SURE is derived explicitly assuming that the noise distribution is Gaussian, there are variants of SURE for most known noise types. See [Least squares estimation without priors or supervision. Raphan and Simoncelli. 2011](http://www.cns.nyu.edu/pub/eero/raphan10.pdf)

4. **Noiser2Noise**: I think the method is more related to [Noiser2Noise, Moran et. al, CVPR 2019](https://arxiv.org/pdf/1910.11908.pdf). The training process, and the estimation formula is very related. Readers may benefit if the authors include this in related work, and discuss more about it. It appears to me that Noiser2Noise essentially has the same idea derived without referring Tweedie's formula and score matching.  

5. **Real data**. This is completely optional, but I would also appreciate if the authors tried to apply this method to real dataset. I understand that the method assumes known noise distributions, but some of the real datasets, like [fluorescence microscopy dataset](https://arxiv.org/abs/1812.10366), [electron-microscopy dataset](https://arxiv.org/pdf/2010.12970.pdf), or [DND and SIDD](https://arxiv.org/pdf/2003.07761.pdf) have real noise which can be modeled fairly accurately. Even if the authors choose not to include experiments on these datasets, they can discuss these datasets as potential real applications of their method. 

**Minor nitpicking:**
Lines 221-234 makes it sounds like the proposed method makes it much easier to perform blind-denoising. However, for baselines like N2S or N2V or [Laine et. al](https://arxiv.org/abs/1901.10277) performing blind-denoising is easier than the proposed approach since they do not have to estimate the $\sigma$ parameter. For SURE, one can just train for the whole range of $\sigma$ (by using the ground-truth $\sigma$ value), and apply the trained network directly to the test image. Thus the baseline methods do not require a separate $\sigma$ estimation step. While I don't consider this as a major limitation, I think the authors should either tone done the claim, or acknowledge that competing methods can also perform blind-denoising just as/more easily. 

Overall I enjoyed reading the paper. Thanks for the great work. 


================== **Post-rebuttal** =================

Thank you for a detailed response to my comments. I'm happy with the responses, and I maintain my score. 
limitations_and_societal_impact: A major practical limitation, as the authors outlined is the assumption of a known noise type belonging to exponential distribution (and a way to estimate noise parameters at test time).
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper considers the problem of image denoising using deep learning. Traditionally, deep denoisers are trained in a supervised way using pairs of noisy and ground-truth images. Recent methods, such as Noise2Noise and those based on SURE, are used to train deep denoisers without ground truth.

This paper proposes Noise2Score as an alternative to existing methods. The idea of Noise2Score is to pre-train a denoising autoencoder (DAE) and use it as an estimator of the score function within the Tweedie’s formula. The key observation is that the pre-training stage of DAE doesn't explicitly need to know the true noise statistics. On other hand, the Tweedie’s formula enables the adaptation of the DAE to specific noise statistics.

The main conceptual contribution of the paper is that it establishes a formal link between the usage of DAEs for approximating score functions and image denoising. The empirical results show the competitive performance of Noise2Score against several well known image denoising methods, including Noise2Noise and SURE.
main_review: Strengths: The key idea of Noise2Score—based on combining the DAE training with the Tweedie’s formula—is original in the context of denoising without ground truth. The method seems to perform well compared to related methods on simulations with several types of noise.

Major Issue 1: The current manuscript is vague on important theoretical/conceptual issues. (1) We know the theoretical conditions for both Noise2Noise and SURE to work, but not for Noise2Score. As an example, what are the assumptions necessary for the DAE to well approximate the score function? (2) How close should the noise statistics be to the one used in the Tweedie’s formula at the inference stage? If I believe the noise is AWGN and use Line 1 in Table 1, but the true noise is Poisson, how will this influence the result? (3) What is expected to happen if the training samples for p(y) in DAE training are not matched to the actual distribution of noise. (4) The noise estimation in (32) seems to use traditional denoising loss functions based on TV. Is there a reason one cannot leverage similar ideas used to develop Noise2Score? (5) The manuscript needs editing for clarity. The symbol sigma is overloaded as the noise variance and as a sampling parameter in (25). It is worth also mentioning that unlike Noise2Score, Noise2Noise does not need the knowledge of the statistical noise distribution at the inference stage.

Major Issue 2: Missing important details on empirical validation. (1) How was the UNet for Noise2Score trained. Was there only one Unet trained for all experiments or several? (2) How were the noisy images/patches sampled for training UNet? Were they picked equally from different distributions (Poisson, AWGN) or only from one? Was the training using only one level of noise or multiple? (3) Why wasn’t Noise2Noise tested on unknown parameters? Noise2Noise doesn’t need the knowledge of the parameter at the inference stage.

Summary: This paper is proposing an intriguing alternative to Noise2Noise and related methods. However, the paper falls short in terms of fully explaining the concept and its theoretical properties. Additionally, empirical validation overlooks important details on training, which makes it difficult to understand the potential of Noise2Score to actually work.

limitations_and_societal_impact: No issues with this.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors how many recent self-supervised denoising methods can be related to Tweedie's formula and based on this insight they propose a novel method that can work with different noise distributions and outperform previous apporaches.
main_review: Overall the paper is well written, even though some concepts can be hard to follow if the reader is not familiar with Tweedie's formula. The theoretical insight linking the formula to recent methods in the self-supervised denoising literature is interesting and novel and brings the authors to an original formulation of a denoising method. My main concern is the lack of discussion and comparisons with the work by Laine et al. "High-Quality Self-Supervised Deep Image Denoising", NeurIPS 2019, which showed how the various Noise2X methods are limited and also formulate a Bayesian framework leading to the estimation of the posterior mean of the image. While the two experimental settings are not directly comparable, it is seems that the work by Laine et al. already had performance competitive or superior to the one shown by the proposed method. The authors should address these points.
limitations_and_societal_impact: Societal impact has been discussed.
review_title: null

Meta-review: This paper proposes a new method, Noise2Score, for denoising images without clean references at training time. The proposed approach has two steps. First a denoising autoencoder (DAE) is used to learn the score function in a noise agnostic fashion. Second, the denoised image is computed through the estimated score function by exploiting Tweedie’s formula (which can be used with any exponential family noises). Empirical evaluation shows the benefits of the proposed approach (comparing to a large number of relevant baselines)

All reviewers found the proposed method interesting and novel. Furthermore, its connection with the recent literature in the subject (e.g. Noise2X) is a nice feature that will inform future research on the topic. 

In the original review, several clarifications were needed. The author provided a detailed response which clarified many of the raised concerns. After the rebuttal, all reviewers recommend accepting the paper. Reviewers kPzN and 1non updated their score (to 6 and 7 respectively). 

In sum, the work provides an interesting contribution and the current empirical evaluation is sufficient for accepting the paper. The AC encourages the authors to incorporate the detailed clarifications provided to the reviewers. 

Decision: Accept (Poster)