Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose uMSE and uPSNR, two unsupervised metrics to evaluate the quality of denoising methods. The big thrust of the paper stems from the obvious observation that unsupervised (or self-supervised) denoising methods cannot otherwise be evaluated.
This is, in fact, a true limitations in the field for several years now, and the community is so far evaluating unsupervised and self-supervised denoising methods on either artificially noised image data or on data where a good approximation of the ground-truth true signal can be recorded.
Still, the reason why self-supervised denoising methods celebrate the success they have is that they can be applied to data for which the ground truth signal cannot (easily) be recorded and is therefore typically not available.
The authors show that their fundamental idea (requiring 3 independently noisy measurements of the same GT image (signal)) is asymptotically consistent with the supervised MSE and PSNR metrics.
Additional to this idea, the authors also present a variation of their metrics that can be computed from single noisy measurements (images) by first applying a specific spatial subsampling scheme.
strength_and_weaknesses: *Strength:*
1. The biggest strength of this paper is the idea, which indeed addresses an important and open problem in a flourishing community.
2. The idea is simple, which is a good thing, and the presentation of it excellent.
3. The manuscript also contributes interesting and relevant theorems and proofs that, for example, show that the proposed method converges to the right evaluation.
4. The additional idea of spatial subsampling is an additional contribution of high practical relevance (while at the same time being less clean and leading to some issues, as pointed out below).

*Weaknesses:*
1. While the ideas and results are very clean for the idea presented e.g. in Figure 1 (3-image metrics), the single image application via spatial subsampling is less clean and seems less thoroughly understood by the authors (at least it is, in my point of view, not sufficiently well described). For example, the fact that the computed numbers are biased for natural images but not for EM images is presented without sufficiently explaining why this effect happens for one but not the other modality. 
2. I would have wished that the original 3-image metrics would be better separated from the 1-image (spatially subsampled) ideas and results. It is of course distinguished in main text tables and figures, but while reading I still needed multiple times to sort my mind to sort my growing understanding correctly. 
3. I missed a more detailed discussion about what noises can be evaluated by the proposed methods. All noises that are independent per pixel given the signal (e.g. Gaussian and Poisson noise), or also structured noises (where neighboring pixels can be affected in correlated ways (e.g. streaking artifacts in tomography)?.
clarity,_quality,_novelty_and_reproducibility: The manuscript is presenting a novel and very relevant idea in a very accessible and clear way.
Reproducibility is not a problem at all.

There is one aspect that I find too little developed, namely the limitations and evaluation divergence of the single-image (spatially subsampled) application of the proposed metrics uMSE_s and uPSNR_s. Here something the authors either chose to not discuss or, maybe, have themselves not yet thought through:
* Ripping a single image apart into multiple images so that the original uMSE and uPSNR metrics can be computed is biasing the evaluation heavily against images that contain very highly spatial signal frequencies. (Or the other way around: smooth images will lead to better numbers.)
* This exact point is, I presume, also the reason why natural images lead to the reported bias, while EM images don’t. Microscopy data typically has a pixel resolution that is higher than the “optical” resolution, meaning that the signal is relatively speaking smooth and pixels sample those structures typically above Nyquist.
* The authors mention “are smoother and have higher spatial resolution” which is quite confusing. EM images are smoother (see above), but the spatial resolution is completely irrelevant for the observed effect. I am quite certain that the important term is the difference between the pixel and optical resolution. Making this point crystal clear would improve the paper considerably and I suggest the authors to consider an overhaul of these aspects in their submitted manuscript.
summary_of_the_review: Love the idea, like the paper, pointed to the one place I believe should change before publication. Changes should be doable since they would only be require modifications in how the single-image metrics are discussed.

I am sure that many people will start to use the proposed metrics in their future denoising works (me included).

I chose to select “6: marginally above the acceptance threshold“, but depending on the rebuttal conversations I am absolutely willing to change this opinion to a better rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes two metrics for unsupervised error/noise estimation, the methods are unsupervised because they can be calculated using noisy images as opposed to clean/noisy images. The metrics are a uMSE and uPSNR (based on uMSE), and the authors show that the consistency of the metrics, and state several limitations of the proposed approaches. 

strength_and_weaknesses: The paper is clearly written and clearly states the contributions and limitations of the method, experiments and results.

Questions:
-  Table 1: the authors point that the metrics are less accurate on the natural image dataset when SNR is high given the bias introduced by susbsampling, but ok on the TEM dataset which is smoother and has high spatial resolution. Given that this is a methods paper, i think the paper would reach a wider audience from a more detailed discussion on how this finding should affect the use of the metric. For example, if my dataset has temporal resolution and we have to use subsampling to get the noisy estimates to calculate uMSE, should I use MSE_avg (standard approach)? 
- What is the conclusion in Fig 6? Can we say that when uMSE is biased it underestimates the true MSE. Do the authors expect the method to always underestimate it or could it overestimate it as well? Having some additional dataset would be helpful here.
- In Fig 10, the distributions of UMSE are very similar, do the authors think that this holds on a per data point level and the uMSE can help determine if different methods identify the same images as noisy? For example, one could produce scatter plots of the uMSE for different approaches (each in x,y axes) for individual data points to determine the correlation. 

clarity,_quality,_novelty_and_reproducibility: The work is clearly written, addressing a typically overlooked motivation. 
summary_of_the_review: Overall, this paper proposes two metrics for unsupervised error/noise estimation. The paper is clearly written and addresses a problem which is typically overlooked. I would like to see this work published although the contributions of this work are fairly limited and it might be better suited for a workshop.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed two unsupervised metrics for evaluating image qualities: uMSE and uPSNR. The main idea is to use three noisy references to design an estimator for MSE, which is the uMSE. The noisy references can be computed using consecutive frames or applying spatial subsampling. It uses the uMSE to define the unsupervised PSNR (uPSNR) metric and demonstrates that the proposed metrics are unbiased and consistent estimators of the supervised metrics under the independence and centered noise assumptions.
strength_and_weaknesses: Strength:

1. The idea of using multiple noisy images to construct unsupervised metrics is interesting and novel.

2. The performance of the proposed methods uMSE and uPSNR is comparable with the supervised metrics MSE and PSNR.

3. The author provided the statistical analysis of the proposed method under some ideal assumptions.

Weakness:

1. This method's key point is using three noisy references to compute the unsupervised metrics. However, they used existing methods (consecutive frames and Neighbor2Neighbor (CVPR2021)) to construct those three reference images. It should be discussed for clarifying the significance of the paper.

2. The independence assumption made in the theoretical analysis section does not hold in practice. The methods they used cannot produce perfect noisy references which satisfy those assumptions since there will be misalignment in consecutive frames and subsampling images.In this sense, it should discuss the case if the noisy references have some errors.

3. There are other no-referenced image quality assessment methods, such as Natural Image Quality Evaluator (NIQE), and BRISQUE [1]; the author should discuss the difference between those methods.

4. This method only works for noise distributions that are pixel-wise independent, which limits its applications.

5. The analysis shows that it provides an unbiased estimator, but the variance of the estimator should be discussed in the main text.

[1] Mittal A, Moorthy A K, Bovik A C. No-reference image quality assessment in the spatial domain[J]. IEEE Transactions on image processing, 2012, 21(12): 4695-4708.
[2] Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423, 1948.




clarity,_quality,_novelty_and_reproducibility: The paper is easy to read.
summary_of_the_review: The paper considers the image quality assessment problem which is important for the applications. But, the author should address the mentioned concerns that show the advantages of the proposed method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to estimate the MSE (and PSNR) in image denoising using only noisy images (no reference). The main idea is to use three noisy images to produce an unbiased estimator of the MSE. The method is sound and well motivated. Several results on synthetic cases and a real example on denoising Electron Microscopy is presented.
strength_and_weaknesses: Strengths of the paper:
1.  Tackles a very relevant problem (both practical and theoretical).
2. Presents a sound solution with very clear requirements, and later introduces two ideas on how to approximately meet the requirements so we can get approximately unbiased estimator of MSE/PSNR.
3. Shows results on real images (electron microscopy) as long as synthetic cases, and also introduces a nice discussion about the limitations of the method.

Weakness:

1. No discussion regarding a very close and relevant problem on Computer Graphics – the problem of Monte Carlo rendering. Where for getting an estimate of MSE, the method keeps track of independent buffers and combines the estimates. See for example this overview paper and the references therein:

* Zwicker, M., Jarosz, W., Lehtinen, J., Moon, B., Ramamoorthi, R., Rousselle, F., Sen, P., Soler, C. and Yoon, S.E., 2015, May. Recent advances in adaptive sampling and reconstruction for Monte Carlo rendering. In Computer graphics forum (Vol. 34, No. 2, pp. 667-681).

2. No discussion regarding similar methods but that rely on the Stein Unbiased Risk Estimator (SURE) (Just a minor mention to this under the paragraph "Unsupervised Denoising"). The paper would be much stronger with a more detailed comparison (technical, not necessarily empirical) to for example these works:
* Soltanayev, S. and Chun, S.Y., 2018. Training deep learning based denoisers without ground truth data. Advances in neural information processing systems, 31.
* Zhussip, M., Soltanayev, S. and Chun, S.Y., 2019. Extending stein's unbiased risk estimator to train deep denoisers with correlated pairs of noisy images. Advances in neural information processing systems, 32.

3. No discussion about the dual problem of noise (level) estimation. Tone of literature on image denoising (from image processing) and noise level estimation. See for example: 
* Liu, X., Tanaka, M. and Okutomi, M., 2013. Single-image noise level estimation for blind denoising. IEEE transactions on image processing, 22(12), pp.5226-5237.
* Lebrun, M., Colom, M. and Morel, J.M., 2015. Multiscale image blind denoising. IEEE Transactions on Image Processing, 24(10), pp.3149-3161.
* Arias, P. and Morel, J.M., 2018. Video denoising via empirical Bayesian estimation of space-time patches. Journal of Mathematical Imaging and Vision, 60(1), pp.70-93.

Also in all the shown cases, the model of the noise is Poisson or Gaussian. This information about the noise model  could be used explicitly so the problem becomes much easier (see, e.g, Prakash [2021]). The paper will be much stronger if the examples and the formulation is given for cases where the noise model is not known or it's not easy to model (mix of things). 

Prakash, M., Delbracio, M., Milanfar, P. and Jug, F., 2021, September. Interpretable Unsupervised Diversity Denoising and Artefact Removal. In International Conference on Learning Representations.

clarity,_quality,_novelty_and_reproducibility: The paper is well written and the exposition is in general good. The analysis of the related work is a little light and in order to clearly judge the contribution this needs to be improved. The novelty seems limited, given that the approach seems a straightforward extension of noise2noise with a noise estimation (using two additional noisy frames). 
summary_of_the_review: I think this is a good paper with an interesting idea. The main weakness in the current exposition is the lack of discussion with respect to other existing work. It is hard to judge the contributions (in particular the novelty of the method). I'm happy to update my review if the authors provide a more detail comparison and analysis of the raised points.

After Rebuttal.
The authors addressed the concerns raised on the first round of reviews and updated the manuscript accordingly. I'm leaning towards accepting this paper, and I'm updating my score to accordingly.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Unsupervised denoising is an important problem setup, since in many real-world denoising applications only noisy images are available without corresponding clean ground-truth images. A variety of methods are available for training networks in an unsupervised manner, and for reconstructing an image based on noisy images alone. However, no metrics are available to evaluate such unsupervised methods based on noisy images alone.

The paper under review proposes an estimator for the MSE (and PSNR) between an estimate of a clean image based on a noisy image, and a clean image that can be computed from three independent noisy observations of the clean image. This metric is called unsupervised MSE (uMSE) and unsupervised PSNR (uPSNR). The idea is a relatively straightforward extension of the noise2noise unsupervised loss. Specifically, the noise2noise loss already provides a biased estimate of the MSE based on two independent noisy versions of a clean image, and the third noisy copy is used to estimate the bias. Subtracting the estimate of the bias yields an unbiased estimate of the MSE.

The paper received four reviews, and a subset of the reviewers and the AC discussed the paper in a virtual meeting. The reviewers are split regarding whether the paper falls slightly below or above the acceptance threshold; two rate the paper slightly below the threshold, and one of the reviewers rating it above the threshold notes that 'I would like to see this work published although the contributions of this work are fairly limited'. 

All reviewers and I agree that the paper tackles a relevant problem and that the paper is well-written and the claims are sound. The paper was discussed in a virtual meeting. 

The main weakness of the paper is the following:

For the setup where three independent noisy copies of a clean image can be obtained, and where the noise is pixel-wise independent, the proposed metric uMSE is an unbiased estimate of the MSE, as desired. In order to use this estimate we have to be sure that those conditions are satisfied. This limits the applicability to a very particular setup, but as pointed out by R1, an expert in the area, there are indeed setups in microscopy for which three independent noisy copies can be measured and for which the proposed metric makes sense.

However, the paper also proposes to compute the uMSE based on a single noisy image, by splitting a single noisy image into several parts, following ideas from neighbor2neighbor and related methods. Using this approach for obtaining three noisy copies, the uMSE is not an unbiased estimate of the MSE anymore, and becomes unreliable. This is also what the results in the paper show, for the EM dataset the uMSE is a good estimate of the MSE, but for the natural-image dataset it is not. 

Thus, for a setup where the uMSE is estimated based on a single image, the uMSE is not reliable, and should not be used for evaluation. This limitation is not sufficiently clear from the paper.

justification_for_why_not_higher_score: For a setup where the uMSE is estimated based on a single image, the uMSE is not reliable, and should not be used for evaluation. This limitation is not sufficiently clear from the paper, and this significantly limits the applicability of the method.
Decision: Reject