Evaluating Unsupervised Denoising Requires Unsupervised Metrics
Authors: Adria Marcos Morales
Matan Leibovich
Sreyas Mohan
Joshua Lawrence Vincent
Piyush Haluai
MAI TAN
Peter Crozier
Carlos Fernandez-Granda
Venue: Submitted to ICLR 2023 (2023)
Keywords: Denoising
Unsupervised Learning
Evaluation Metrics
Statistical Estimation
Imaging
Electron Microscopy

Abstract: Unsupervised denoising is a crucial challenge in real-world imaging applications. Unsupervised deep-learning methods have demonstrated impressive performance on benchmarks based on synthetic noise. However, no metrics are available to evaluate these methods in an unsupervised fashion. This is highly problematic for the many practical applications where ground-truth clean images are not available. In this work, we propose two novel metrics: the unsupervised mean squared error (MSE) and the unsupervised peak signal-to-noise ratio (PSNR), which are computed using only noisy data. We provide a theoretical analysis of these metrics, showing that they are asymptotically consistent estimators of the supervised MSE and PSNR. Controlled numerical experiments with synthetic noise confirm that they provide accurate approximations in practice. We validate our approach on real-world data from two imaging modalities:  videos in raw format and transmission electron microscopy. Our results demonstrate that the proposed metrics enable unsupervised evaluation of denoising methods based exclusively on noisy data.

Under review as a conference paper at ICLR 2023
EVALUATING UNSUPERVISED DENOISING REQUIRES UNSUPERVISED METRICS
Anonymous authors
Paper under double-blind review
ABSTRACT
Unsupervised denoising is a crucial challenge in real-world imaging applications. Unsupervised deep-learning methods have demonstrated impressive performance on benchmarks based on synthetic noise. However, no metrics are available to evaluate these methods in an unsupervised fashion. This is highly problematic for the many practical applications where ground-truth clean images are not available. this work, we propose two novel metrics: the unsupervised mean squared error (MSE) and the unsupervised peak signal-to-noise ratio (PSNR), which are computed using only noisy data. We provide a theoretical analysis of these metrics, showing that they asymptotically consistent estimators of the supervised MSE and PSNR. Controlled numerical experiments with synthetic noise confirm that they provide accurate approximations in practice. We validate world data from two imaging modalities: videos in raw format and transmission electron microscopy. Our results demonstrate that the proposed metrics enable unsupervised evaluation of denoising methods based exclusively on noisy data.
approach real-
INTRODUCTION
Image denoising is a fundamental challenge in image and signal processing. well as a key preprocessing step for computer vision tasks. Convolutional neural networks achieve state-of-the-ar performance for this problem, when trained using databases of clean images corrupted with simulated noise ZhangZhangaar et al. (2017a). However, in real-world imaging applications such as microscopy, noiseless ground truth videos are often not available. This has motivated the development of unsupervised denoising approaches that can be trained using only noisy measurements Lehtinen al. (2018); Xie et al. (2020); Laine et al. (2019); [heth et al. ( et al. (2021). These methods have demonstrated impressive performance
natural-image benchmarks, essentially par with the supervised state of the art. However, to the best of our knowledge, no unsupervised metrics are currently available to evaluate them using only noisy data.
Reliance on supervised metrics makes it very challenging to create benchmark datasets using real- world measurements, because obtaining the ground-truth clean images required by these metrics is often either impossible or very constraining, In practice, clean images are typically estimated through temporal averaging, which suppresses dynamic information that is often crucial in scientific applications. Consequently, quantitative evaluation of unsupervised denoising methods is currently almost completely dominated by natural image benchmark datasets with simulated noise Lehtinen et al. (2018); Xie et al. (2020); Laine et al. (2019); Sheth et al. (202T); Huang et al. (2021). which are not always representative of the signal and noise characteristics that arise in real-world imaging applications.
The lack of unsupervised metrics also limits the application of unsupervised denoising techniques in practice. In the absence of quantitative metrics, domain scientists must often rely visual inspection to evaluate performance on real measurements. This is particularly restrictive for deep-learning approaches, because it makes it impossible to perform systematic hyperparameter optimization and model selection on the data of interest.
In this work, propose two novel unsupervised metrics to address these issues: the unsupervised mean- -quured error (uMSE) and the unsupervised peak signal-to-noise ratio (uPSNR), which are computed exclusively from noisy data. These metrics build upon existing unsupervised denoising

---

Under review as a conference paper at ICLR 2023
methods, which minimize unsupervised cost function equal to the difference between the denoised estimate and additional noisy copies of the signal of interest Lehtinen et al. (2018). The uMSE is equal to this cost function modified with correction term, which renders unbiased estimator of the supervised MSF.
We provide theoretical analysis of the uMSE and uPSNR, proving that they are asymptotically consistent estimators of the supervised MSE and PSNR respectively. Controlled experiments on supervised benchmarks, where the true MSE and PSNR can be computed exactly, confirm that the uMSE and uPSNR provide accurate approximations In addition, we validate the metrics on video data in RAW format, contaminated with real noise that does not follow a known predefined model.
In order to illustrate the potential impact of the proposed metrics on imaging applications where no (TEM) data. Recent
ground-truth is available, we apply them to advances in direct electron detection systems make it possible for experimentalists to acquire highly time-resolved movies of dynamic events frame rates in the kilohertz range Faruqi McMullan (2018); [Eriuus et al. (2020), which is critical to advance our understanding of functional materials. Acquisition at such high temporal resolution results in severe degradation by shot noise. We show that unsupervised methods based on deep learning can be effective in removing this noise, and that our proposed metrics can be used to evaluate their performance quantitatively using only noisy data.
range/Faruqi
To summarize, our contributions (1) two novel unsupervised metrics presented in Section 3 (2) theoretical analysis providing an asymptotic characterization of their statistical properties (Section 4 (3) experiments showing the accuracy of the metrics in controlled situation where ground-truth
clean images are available (Section()) 5). (4) validation on real-world videos in RAW format (Section o and (5) an application to a real-world glectron-microscope dataset, which illustrates the challenges of unsupervised denoising in scientific imaging (Section
BACKGROUND AND RELATED WORK
Unsupervised denoising The past few years have seen ground-breaking progress in unsupervised denoising, pioncered by Noise2Noise, a technique where a neural network is trained on pairs of noisy images Lehtinen et al. 2018). Our unsupervised metrics are inspired by Noise2Noise, which optimizes a cost function equal to our proposed unsupervised MSE, but without a correction term (which is not needed for training models). Subsequent work focused on performing unsupervised denoising from single images using variations of the blind-spot method, where model is trained to estimate each noisy pixel value using its neighborhood but not the noisy pixel itself (to avoid the trivial identity solution) Krull et al. (2019); Laine et al. (2019); BBtson & Royer (2019) ; Sheth et al. 2021 y: Xie et al. (022)). More recently, Neighbor2Neighbon revisited the Noise2Noise method, generating noisy image pairs from a single noisy image via spatial subsampling Huang al. (202T), an insight that can also be leveraged in combination with our proposed metrics, as explained in Section
C Our contribution with respect to these methods is novel unsupervised metric that can be used for evaluation, as it is designed to be an unbiased and consistent estimator of the MSE.
Stein's unbiased risk estimator (SURE) provides asymptotically unbiased estimator of the MSE for i.i.d. Gaussian noise Donoho & Johnstone 1995). This cost function has been used for training unsupervised denoisers Metzler et al. (2018) 8); Soltanayev & Chun 2018): Zhussip al. 2019
Mohan et al. 2222)) In principle, SURE could be used to compute the MSE for evaluation, but it has certain limitations: (1) a closed form expression of the noise likelihood required, including the value of the noise parameters (for example, this is not known for the real-world datasets in Sections 6
and 7. (2) computing SURE requires approximating the divergence of denoiser (usually via Monte
Carlo methods Ramani et al. 2008)), which is computationally very expensive. Developing practical unsupervised metries based on SURE and studying their theoretical properties is an interesting direction for future research.
Existing evaluation approaches In the literature, quantitative evaluation of unsupervised denoising techniques has mostly relied on images and videos corrupted with synthetic noise Lehtinen et al. (2018): Krull et al.] (2019); Laine et al. (2019); Batson & Royer (019)): Sheth et al. (2021) Xie et al. (2220). Recently, a few datasets containing real noisy data have been created Abdelhamed et al. (2018): : Plotz& & Roth (2017); Xu et al. (2018); Zhang et al. (2019). Evaluation on these datasets is
based on supervised MSE and PSNR computed from estimated clean images obtained by averaging
2

---

Under review as a conference paper at ICLR 2023
2
Unsupervised mean squared error
2
2
1
2
Demoised
Clean
Denoised
Notry
Notey
Nodry
Figure I: MSE vs uMSE. The traditional supervised mean squared error (MSE) is computed by comparing the denoised estimate the clean ground truth (left). The proposed unsupervised MSE is computed only from noisy data, via comparison with a noisy reference corresponding to the same ground-truth but corrupted with independent noise (right). A correction based on additional noisy references debiases the estimator.
multiple noisy frames. Unfortunately, result, the metrics cannot capture caaammcclll-channiii features, which are of interest many applied domains. In addition, unless the signal-to-noise ratio quite high, it is necessary average over large number of frames to approximate the MSE. For example, as explained in Section E. for CNN based denoiser and an image corrupted by additive Gaussian noise with standard deviation the same approximation accuracy as our proposed approach (see Figure TO which only requires noisy images, and can also be computed from a single noisy image.
15 need average 1500 noisy images achieve
Noise-Level Estimation. The correction term in uMSE can be interpreted as an estimate of the noise level, obtained cancelling out the clean signal. In this sense, it related noise-level estimation methods Liu et al. (2013) Lebrun et al.) (2015) Arias & Morel (2018). However, unlike uMSE, these methods typically assume parametric model for the noise, and are not used for evaluation.
No-reference image quality assessment methods evaluate the perceptual quality of an image Li
(2002); Mittal et al. (2012), but not whether consistent with an underlying ground-truth corre-
sponding to the observed noisy measurements, which is the goal of our proposed metrics.
UNSUPERVISED METRICS FOR UNSUPERVISED DENOISING
3.1 THE UNSUPERVISED MEAN SQUARED ERROR
The goal of denoising is to estimate clean signal from noisy measurements. Let x E R be signal or set of signals with total entries. We denote the corresponding noisy data by y E R" A denoiser f: _ + 8n is a function that maps the input y to an estimate of æ. A common metric to evaluate the quality of denoiser is the mean squared error between the clean signal and the estimate,
MSE =
Σ TI
(x, - f(2)))
1=1
Unfortunately, most real-world scenarios clean ground-trute signals are available and evaluation can only be carried out in unsupervised fashion, i.e. exclusively from the noisy measurements. In this section we propose an unsupervised estimator of MSE inspired by recent advances in unsupervised denoising Lehtinen et al. (2018). The key idea is to compare the denoised signal to noisy reference, which corresponds to the same clean signal corrupted by independent noise.
In order to motivate our approach, let assume that the noise is additive, that y x + for a zero-mean noise vector E R". Imagine that we have access to a noisy reference := x + w corresponding to the same underlying signal x, but corrupted with different noise realization w e IR" independent from z (Section 3x3 explains how to obtain such references in practice). The mean squared difference between the denoised estimate and the reference is approximately equal to
3

---

Under review as a conference paper at ICLR 2023
the sum of the MSE and the variance o² of the noise,
1
(a= f(()))
(a + wi -(());
(2)
i=1
1
(0) f(v):))
+ Σ w = MSE o²,
(3)
n
== Wy (z; f());)² cancels out if w; and yi (and hence f(v)) are independent (and the mean of the noise is zero).
because the cross-term Σ
Approximations to equation 3 are used by different unsupervised methods to train neural networks for denoising Lehtinen al. (2018); Xie et al. 2020); Laine et al. (2099); Huang et al. (202T). The noise term 2 wf in equation
not problematic for training denoisers as long as it is independent from the input y. However, it Is definitely problematic for evaluating denoisers. In order to estimate the MSE we need to cancel it out. We propose achieve this by using two other noisy references b = x + U and с == x + u, which are noisy measurements corresponding to the clean signal æ, but corrupted with different, independent noise realizations and (just like a). Subtracting these references and dividing by two yields an estimate of the noise variance,
(b. c2)
1
w,)
RJ
(4)
2
2nL
2m
1=1
1=1
which can then be subtracted from equation to estimate the MSE. This yields our proposed unsupervised metric, which we call unsupervised mean squared error (uMSE), depicted in Figure] [ Definition (Unsupervised mean squared errot). Given noisy input signal y E IR" and three noisy references 6, E R² the unsupervised mean squared error of a denoiser f: R + -R" is
(b: G))
uMSE = Σ (a, ((v))
(5)
2
Theorem in Section 4 establishes that the uMSE is consistent estimator of the MSE as long (1) the noisy input and the noisy references are independent, (2) their means equal the corresponding entries of the ground-truth clean signal, and (3) their higher-order moments are bounded. These conditions are satisfied by most noise models of interest in signal and image processing, such as Poisson shot noise additive Gaussian noise. Section 3.3 we address the question of how obtain the noisy references required to estimate the uMSE. Section B eexplains how to compute confidence intervals for the uMSE via bootstrapping.
3.2 THE UNSUPERVISED PEAK SIGNAL- SIGNAL-TO-NOISE TO-NOISE RATIO
Peak signal-to-noise ratio (PSNR) is currently the most popular metric to evaluate denoising quality, is a logarithmic function of MSE defined on decibel scale,
M MSE
PSNR 10
where M is a fixed constant representing the maximum possible value of the signal of interest, which is usually set equal to 255 for images. Our definition of uMSE can be naturally extended to yield an unsupervised PSNR (uPSNR).
Definition (Unsupervised peak signal-to-noise ratio). Given a noisy input signal y RR and three noisy references a, b, e E Rn the peak signal-to-nen ratio of denoiser f IR - IR" is
M
uPSNR 10
uMSE
where M is the maximum possible value of the signal of interest.
Corollary 5 establishes that the uPSNR is consistent estimator of the PSNR, under the same conditions that guarantee consistency of the uMSE. Section BrrioonBeexpii exxlains how to compute confidence intervals for the uPSNR via bootstrapping.
4

---

Under review as a conference paper at ICLR 2023
20 pixels
100 pixels
1,000 pixels
Normallzed counts
3
10
Gaussian fit
0
0.0
MSE
1.5
2.0
MSE
1.5
20
40.
MSE
1.5
2.0
uMSE
uMSE
uMSE
UMSE-MSE MSE
8
0
200 400
600
1000
1200
1400
Number of pixels
Figure 2: The uMSE is a consistent estimator of the MSE. The histograms the top show the distribution of the uMSE computed from pixels (n (20, 100, 1000}) of a natural image corrupted with additive Gaussian noise (a = 55) and denoised via a deep-learning denoiser (DnCNN). Each point in the histogram corresponds to a different sample of the three noisy references used to compute the uMSE (ā.. b, and in Eq. 9 for 1 < S n). with the same underlying clean pixels. The distributions are centered at the MSE, showing that the estimator unbiased (Theorem (3). and are well approximated by Gaussian fit (Theorem (6) As the number of pixels grows, the standard deviation of the uMSE decreases proportionally to n 11/2, and the uMSE converges asymptotically to the MSE (Theorem 4 depicted in the scatterplot below (a: is constant).
3.3 COMPUTING NOISY REFERENCES IN PRACTICE
Our proposed metrics rely on the availability of three noisy references, which ideally should corre- spond to the same clean image contaminated with independent noise. Deviations between the clean signal in each reference violate Condition 2 in Section 4 and introduce bias in the metrics. We
propose two approaches to compute the references practice, illustrated in Figure 5
The references be computed from consecutive frames acquired within short time interval. This approach is preferable for datasets where the image content does not experience rapid dynamic changes from frame frame. We apply this approach the RAW videos in Section 6 where the content is static.
Multiple images:
The references can be computed from single image via spatial subsampling, as described in Section Section Cisoww that this approach is effective as long as the image content sufficiently smooth with respect the pixel resolution. We apply this approach the electron- microscopy data in Section 7 where preserving dynamic content is important.
Single image:
STATISTICAL PROPERTIES OF THE PROPOSED METRICS
In this section, establish that the proposed unsupervised metrics provide consistent estimate of the MSE and PSNR. In our analysis, the ground truth signal or set of signals represented as deterministic vector R.,. The corresponding noisy data are also modeled deterministic vector y E R" that is fed into a denoiser f = + IR" to produce the denoised estimate f The MSE of the estimate deterministic quantity equal to
MSE 1 n
SE,
SE, = (x. -(()))
(8)
5

---

Under review as a conference paper at ICLR 2023
Natural images (Gaussian noise)
Electron microscopy (Poisson noise)
Spatial subsampling
Spotial subsampling
-102
-102
-102
-102
Normalized counts
10
10
2
2
0
0
MSE
2
MSE
-10-3
-10-3
-10
-10
uMSE
uMSE
UMSE
UMSE
Figure 3: Bias introduced by spatial subsampling. The histograms show the distribution of the uMSE (computed as in Figure 2 corresponding to natural image and a simulated electron- microscopy image corrupted by Gaussian (o 55) and Poisson noise respectively, and denoised with a standard deep-learning denoiser (DnCNN). For each image, the uMSE is computed using noisy references with the same underlying clean image (left), and from noisy references obtained via spatial subsampling (right). For the natural image, spatial subsampling introduces a substantial bias (compare the 1st and 2nd histogram), whereas for the electron- miceroon-micoocoppy image the bias is much smaller (compare the 3rd and 4th histogram).
Noise Model. The uMSE estimator in Definition 1 depends on three noisy references à, č, which we model random variables Our analysis assumes that these random variables satisfy two conditions:
Condition 1 (independence): The entries of ā, b, ē are all mutually independent.
Condition 2 (centered noise): The mean of the ith entry of ā. b, equals the corresponding entry of the clean signal, EE [ā.] [[ = E [ā]] I, 1 S S
Two popular noise models that satisfy these conditions are:
Additive Gaussian, where à; + ws. b, == + 6. ē. = x. + ü. for i.i.d. Gaussian W., Üs üs.
Poisson, where āg b, à are i...d. Poisson random variables with parameter 24.
Theoretical Guarantees. Our goal is to study the statistical properties of the uMSE
1 Σ USE
uSE; (ā; f()),)² (b: 22
uMSE
n
As indicated by the tilde, under our modeling assumptions, the uMSE a random variable. We first show that the correction factor in the definition uMSE succeeds in debiasing the estimator, that its mean is equal to the MSE.
Theorem 3 (The uMSE is unbiased, proof in Section G.1 If Conditions I and 2 hold, the uMSE is an unbiased estimator of the MSE, i.e. E[uMSE] MSE.
Theorem establishes that the distribution of the uMSE is centered at the MSE. We now show that its variance shrinks rate inversely proportional to the number of signal entries n, and therefore converges to the MSE in mean square and probability as n - oo (see Figure 2 for numerical demonstration). This occurs long as the higher central moments of noise and the entrywise denoising error are bounded by a constant, which is to be expected in most realistic scenarios.
Theorem (The uMSE is consistent, proof in Section G.2). Let !ki denote the kth central moment of ā. by To and == maxisisn [z; - fl)) the maximum entrywise denoising error. If Conditions I and 2 hold, and there exists a constant a such that maxi max
[4] [3]
14 7/- S o,, then the
In analysis, all random quantities marked with tilde for clarity.

---

Under review as a conference paper at ICLR 2023
Table 1: Controlled comparison of PSNR and uPSNR. The table shows the PSNR computed from clean ground-truth images, compared to two versions of the proposed estimator: one using noisy references corresponding to the same clean image (uPSNR), and another using single noisy image combined with spatial subsampling (uPSNR,). The metrics are compared on the datasets and denoising methods described in Section
Natural images (Gaussian noise)
25
50
75
100
Method
PSNR
uPSNRs
PSNR
uPSNR:
uPSNR
uPSNR,
PSNR
uPSNRs
Bilateral
24.20
24.18
26.20
21.84
21.86
22.90
19.14
19.17
19.58
16.30
16.37
16.47
DenseNct
26.54
26.51
27.61
23.98
24.06
26.28
22.75
23.00
24.69
21.92
21.97
23.78
DnCNN
26.19
26.21
28.14
23.95
24.02
26.08
22.72
22.75
24.59
21.84
21.84
23.71
UNet
26.29
26.28
27.98
23.92
24.01
26.25
22.68
22.70
24.84
21.91
21.89
23.48
Electron microscopy (Poisson noise)
Bilateral
BlindSpot
DnCNN
UNet
PSNR uPSNR
uPSNRs| PSNR
uPSNR
uPSNRs
PSNR
uPSNR
uPSNRs
PSNR
uPSNR
uPSNRs
20.18 18
20.20
20.21
24.86
24.87
24.74
25.74
25.68
25.86
24.65
24.69
24.79
mean squared error between the MSE and the uMSE satisfies the bound
E (MMSE - MSE) M
Var
uMSE
(10)
Consequently, limm-+o (uMEE - MSE)²] 0, the uMSE converges to the MSE in mean square and therefore also in probability.
Consistency of the uMSE implies consistency of the uPSNR.
Corollary 5 (The uPSNR is consistent, proof in Section the uPSNR defined as
1.3
Under the assumptions of Theorem
M
uPSN 10
(11)
uMSE
where M is a fixed constant, converges in probability to the PSNR, n + CC.
The uMSE converges to a Gaussian random variable asymptotically as n 0o.
Theorem (The uMSE is asymptotically normally distributed, proof in Section G.4). tfe first six central moments ā. be č₁ and the maximum enirywise denoising error max 1 sisn w [vv) are bounded, and Conditions and hold, the uMSE is asymptotically normally distributed as n + co.
Our numerical experiments show that the distribution of the uMSE is well approximated as Gaussian even for relatively small values of (see Figure This can be exploited to build confidence intervals for the uMSE and uPSNR, as explained Section B
CONTROLLED EVALUATION OF THE PROPOSED METRICS
In this section, we study the properties of the uMSE and uPSNR through numerical experiments in controlled scenario where the ground-truth truth clean images are known. We use dataset of natural images (Martin al. 2001 Zhang et al.) 2017b Franzen 1993) corrupted with additive Gaussian noise, and dataset of simulated Poisson noise. For the two datasets, we compute the supervised MSE and PSNR using the ground- truth clean image. To compute the uMSE and uPSNR the same clean image corrupted with independent noise. We also compute the uMSE and uPSNR using noisy references obtained from single noisy image via spatial subsampling. described in
microscopy images Vincent et al. (2021) corrupted with
noisy references corresponding to
7

---

Under review as a conference paper at ICLR 2023
Table 2: Comparison of averaging- PSNR and uPSNR on RAW videos with real noise. The proposed uPSNR metric, computed using three noisy references, is very similar to an averaging-basee PSNR estimate computed from 10 noisy references. The metrics are compared on the datasets and
denoising methods described in Section !
Image (Wavelet)
Image (CNN)
Video (Temp. Avg)
Video (CNN)
ISO
PS
uPSNR
PSNRigg
uPSNR
PSNRue
uPSNR
PSNR.y
uPSNR
1600
37.56
37.76
46.88
48.05
34.32
34.36
48.06
49.51
3200
35.52
35.55
44.91
45.51
32.48
32.47
46.45
47.33
6400
32.60
32.68
42.74
43.05
30.77
30.75
44.75
45.16
12800
28.43
28.46
40.22
39.75
27.71
27.76
42.22
41.69
25600
26.79
26.9
40.19
38.78
27.08
27.12
42.13
40.32
Mcan
32.18
32.27
42.99
43.03
30.47
30.49
44.72
44.80
Section which we denote by uMSEs and uPSNRs respectively All metrics are applied to multiple denoising approches, described in more detail in Section
The results are reported in Tables I and 3
When the noisy references correspond exactly to the same clean image (and therefore satisfy the conditions in Section the unsupervised metries are extremely accurate across different noise levels for all denoising methods.
Single-image results: When the noisy references computed via spatial subsampling, the metrics are still very accurate for the electron-microscop dataset, but less so for the natural-image dataset the PSNR is high (above 20 dB). The culprit is the difference between the clean images underlying
each noisy reference (see Figure 5 which introduces a bias in the unsupervised metric, depicted
in Figure 3 Figure 8 shows that the difference more pronounced in natural images than in
electron electron-microscopy images, which are smoother with respect to the pixel resolution (see Section C
APPLICATION TO VIDEOS IN RAW FORMAT
We evaluate our proposed metrics on dataset of videos in raw format, consisting of direct readings from the sensor of a surveillance camera contaminated with real noise five different ISO levels Yue al. (2020). The dataset contains 11 unique videos divided into 7 segments, each consisting of 10 noisy frames that capture the same static object. We consider four different denoisers: a wavelet-bant method, temporal averaging and two versions of state-of-fe of-the-art unsupervised deep-learning method using images and videos respectively (heth et al. (2021). I). detailed description the experiments is provided in Section [ Tables and compare our proposed unsupervised metrics (computed using three noisy frames in each segment) with MSE and PSNR estimates obtained via averaging from noisy frames. The types of metric yield similar results: the deep-learning methods clearly outperform the other baselines, and the aidco-based methods outperform the image-based methods. As explained in Section [ the averaging-bain based MSE and PSNR are not consistent estimators of the true MSE and PSNR, and can be substantially less accurate than the uMSE and uPSNR (see Figure TOp. so they should not be considered ground-trute metries.
APPLICATION TO ELECTRON MICROSCOPY
Our proposed metrics enable quantitative evaluation of denoisers in the absence of ground-truth clean images. We showcase this for transmission electron microscopy (TEM), key imaging modality in material sciences. Recent developments enable the acquisition of high frame-rate images, capturing high temporal resolution dynamics, thought to be crucial in catalytic processes Crozier et al. (2019). Images acquired under these conditions are severely limited by noise. Recent work suggest that deep learning methods provide effective solution Sheth al. (202T): Mohan al. (2022 2021 but, instead of quantitative metrics, evaluation on real data has been limited to visual inspection.
Crocesses/Crozier
Thhe other metrics applied to subsampled images in order to make them directly comparable to the uMSFs and uPSNRs.
8

---

Under review as a conference paper at ICLR 2023
Moderate SNR
Low SNR test set
Example
Zoom
uPSNR
Example
Zoom
OPSNR
Data
Gaussian smooth.
20.4
16.0
CNN
26.9
18.6
Figure 4: Denoising real-world Nlectronmmicrosee data. Example noisy images (top) from the moderate-SNR (left columns) and low-SNR (right columns) test sets described Section / The data are denoised using Gaussian-smoothing baseline (second row) and convolutional neural network (CNN). The uPSNR of each method on each test set shown beside the images. Both the uPSNR and visual inspection indicate that the CNN clearly outperforms Gaussian smoothing, and that both methods achieve much worse results on the low-SNR test set.
The TEM dataset consists 18,597 noisy frames depicting platinum nanoparticles on cerium oxide support. A major challenge for the application of unsupervised metries is the presence of local correlations in the noise (see Figure We address this by performing spatial subsampling to reduce the correlation and selecting two contiguous test sets with low correlation: 155 images with moderate signal-to-noise ratio (SNR), and 383 images with low SNR, which are more challenging. We train an unsupervised Neighbor2Neighhor convolutional-neuui containing 70% of the data, and compare its performance to a Saussian-smoothie sets. Section | provides more detailed description of the dataset and the models.
network Huang et al. (2021) baseline on the
training set
Figure 4 shows examples from the data and the corresponding denoised images, as well as the Figure [T] shows histogram of the uMSE values
uPSNR of each method for the test for each individual test image. The unsupervised metrics indicate that the deep-learning method achieves effective denoising on the moderate SNR set, clearly outperforming the Gaussian-smoothhng baseline, and also that both methods produce significantly worse results for the low-SNR test set. Figure 12 shows that uMSE produces consistent image- mevee-level evaluations for the two denoisers. These conclusions are supported by the visual appearance of the images.
CONCLUSION AND OPEN QUESTIONS
In this work we introduce two novel unsupervised metrics computed exclusively from noisy data, which are asymptotically consistent estimators of the corresponding supervised metrics, and yield accurate approximations in practice. These results are limited to denoising under the assumption of independent noise. Several important open questions remain: (I) How to address the bias introduced by spatial subsampling and achieve an unbiased approximation to the MSE from a single noisy image. (2) How to account for noise distributions and artifacts which are not pixel-wise independent (see e.g. Prakash al. (2021)) (3) How to obtain unsupervised approximations of perceptual metrics
such as SSIM Wang et al. 2004) (4) How to perform unsupervised evaluation for general inverse problems, and related appli applications such realistic image synthesis Zwicker al. 2015

---

Under review as a conference paper at ICLR 2023
REFERENCES
Abdelrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high-quality denoising dataset for smartphone cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1692-1700, 2018.
Pablo Arias and Jean- Meaan Morel. Video denoising via empirical bayesian estimation of space-time patches. Journal of Mathematical Imaging and Vision, 60(1):70- 2018.
Joshua Batson and Loic Royer. Noise2self: Blind denoising by self-supervision,, 2019. URL
https: org/ 1bss/991..11300
Lco Breiman. Probability. SIAM, 1992.
Peter A. Crozier, Ethan L. Lawrence, Joshua L. Vincent, and Barnaby D.A. Levin. Dynamic restructuring during processing: approaches higher temporal resolution. Microscopy and Microanalysis, 25($2): 2019.
D Donoho and Johnstone. Adapting to unknown smoothness via wavelet shrinkage. J American Stat Assoc, 90(432), December 1995.
Bradley Efron and Robert J Tibshirani. An introduction to the bootstrap. CRC press, 1994.
Peter Ercius, Ian Johnson, Hamish Brown, Philipp Pelz, Shang-Lin Hsu, Brent Draney, Erin Fong, Azriel Goldschmidt, John Joseph, Jason Lee, and et al. The 4d camera a 87 khz frame-rate detector for counted 4d-stem experiments. Microscopy and Microanalysis, pp. 1-3, 2020. doi: 10.1017/5143192762001111
A.R. Faruqi and G. McMullan. Direct imaging detectors for electron mieroscopy. Nuclear In- struments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detec- tors and Associated Equipment, 878:180 190, 2018. ISSN 0168-9002. doi: https://doi. org/10.10166 hwt.:o...aaaaaa.......0.000 URL http: /wwww/wwwww... wwwsscciiiiiieeeceemoo rect. com/ science/
atticcee/p1//001699022771
Radiation Imaging Techniques and Applications.
Richard W. Franzen, 1993. Wwllhttpp///Ok.uus//gee
/
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proc. IEEE Conf. Computer Vision and Puttern Recognition, pp. 4700-4708, 2017. 2017.
Tao Huang, Songjiang Li, Xu Jia, Huchuan Lu, and Jianzhuang Liu. Neighbor2neighbor: Self- supervised denoising from single noisy images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14781-14790, 2021.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. Noise2void learning denoising from single noisy images. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp.
2019.
Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila. High-quality self-supervised deep image denoising. In Advances in Neural Information Processing Systems 32, pp. (970-6980, 2019.
Mare Lebrun, Miguel Colom, and Jean-Michel Morel. Multiscale image blind denoising. IEEE Transactions on Image Processing, 24(10):3149-316 2015.
Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data, 2018. URL https
04189
Xin Li. Blind image quality assessment. In Proceedings. International Conference on Image Processing, volume 1, pp. H- IEFE, 2002.
10

---

Under review as a conference paper at ICLR 2023
Xinhao Liu, Masayuki Tanaka, and Masatoshi Okutomi. Single-image noise level estimation for blind denoising. IEEE transactions on image processing, 22(1)):5226 222552222-527,,000 2013.
D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. Proc. 8th Int'l Conf. Computer Vision, volume 2, pp. 416-423, July 2001.
Christopher A Metzler, Ali Mousavi, Reinhard Heckel, and Richard Baraniuk, Unsupervised learning with stein's 's unbiased risk estimator. arXiv preprint arXiv: 1005511005,1033,, 2018.
Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. No-reference image quality as- sessment in the spatial domain. IEEE Transactions on image processing, (112)))695-4708 2012.
Sreyas Mohan, Zahra Kadkhodaie, Eero P. Simoncelli, and Carlos Gemaaddz-GGandeee Robust and interpretable blind image denoising via bias-free convolutional neural networks. In International Conference on Learning Representations, 2020.
id=HJ] 4FC4FPS
Sreyas Mohan, Joshua Vincent, Ramon Manzorro, Peter Crozier, Carlos Fermandez- Granda, and Eero Simoncelli. Adaptive denoising via gaintuning. Advances Neural Information Processing Systems, 34, 2021.
Sreyas Mohan, Ramon Manzorro, Joshua L Vincent, Binh Tang, Dev Y Sheth, Eero Simoncelli, David S Matteson, Peter A Crozier, and Carlos Fernandez-Grande Deep denoising for scientific discovery: A case study in electron microscopy. IEEE Transactions on Computational Imaging. 2022.
Tobias Plotz and Stefan Roth. Benchmarking denoising algorithms with real photographs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1586-1595, 2017.
Mangal Prakash, Mauricio Delbracio, Peyman Milanfar, and Florian Jug. Interpretable unsupervised diversity denoising and artefact removal. In International Conference on Learning Representations, 2021.
Sathish Ramani, Thierry Blu, and Michael Unser. Monte-carlo sure: A black-box optimization of regularization parameters for general denoising algorithms. IEEE Transactions on image processing, 1709:1554-1555, 2008
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Dev Yashpal Sheth, Sreyas Mohan, Joshua L Vincent, Ramon Manzorro, Peter A Crozier, Mitesh M Khapra, Eero P Simoncelli, and Carlos Brrnandez-Grandee Unsupervised deep video denoising. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1759-1763 2021.
Shakarim Soltanayev and Se Young Chun. Training deep learning based denoisers with- out ground truth data. In
31, 2018. URL Advances Neural Information Processing Systems, vol- c0567722eeaac1116227766667722722 Pappee Paper.pde
Joshua L Vincent, Ramon Manzorro, Sreyas Mohan, Binh Tang, Dev Y Sheth, Eero P Simoncelli, David Matteson, Carlos Fermandez- -Gandda, and Peter A Crozier. Developing and evaluating deep neural network-based denoising for nanoparticle tem images with ultra-low signal-to-noise. Microscopy and Microanalysis, 27(6): 1431-1447, 2021.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4)6600661, 2004.
11

---

Under review as a conference paper at ICLR 2023
Yaochen Xie, Zhengyang Wang, and Shuiwang Ji. Noise2same: Optimizing supe-suuer bound for image denoising. Advances in Neural Information Processing Systems, 33:20320-2035 203300, 2020.
Jun Xu, Hui Li, Zhetong Liang, David Zhang, and Lei Zhang. Real-world noisy image denoising: A new benchmark. arXiv preprint arXiv:/804. 02603, 2018.
H. Yue, C. Cao, Liao, R. Chu, and J. Yang. Supervised raw video denoising with benchmark dataset on dynamic scenes. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 2298-2307, 2020. doi: 10.1 10.1009994422220022222222 202.000000020000223317
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep enn for image denoising. IEEE Transactions Image Processing, pp. 3142-3155, 2017a.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng. and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing, 26(7):
3142-3155, 2017b.
Yide Zhang, Yinhao Zhu, Evan Nichols, Qingfei Wang, Siyuan Zhang, Cody Smith, and Scott Howard. caasson-gaussian denoising dataset with real fluorescence microscopy images. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11710-11718, 2019.
Magauiya Zhussip, Shakarim Soltanayev, and Se Young Chun. Extending stein's unbiased risk estimator to train deep denoisers with correlated pairs of noisy images. Advances in neural information processing systems, 32, 2019.
Matthias Zwicker, Wojeiech Jarosz, Jaakko Lehtinen, Bochang Moon, Ravi Ramamoorthi, Fabrice Rousselle, Pradeep Sen, Cyril Soler, and S-E Yoon. Recent advances in adaptive sampling and reconstruction for monte carlo rendering. Computer gruphics forum, volume 34, pp. 667-681. Wiley Online Library, 2015.
12

---

Under review as a conference paper at ICLR 2023
Spatial subsampling
Difference
250
150
100
50
Consecutive frames
Difference
200
150
200
Figure 5: Noisy references. The proposed metries require noisy references corresponding to the same clean image corrupted by independent noise. These references be obtained from single image via spatial subsampling (above) or from consecutive frames (below). In both cases, there may be small differences in the signal content of the references, shown by the corresponding heatmaps.
Spatial subsampling
0.05
uMSE- -at-based confidence intervals
0.04
0.04
0.03
0.03
0.02
0.02
0.01
0.01
0.02
0.03
0.04
0.01
0.02
0.03
0.04
MSE
MSE
Figure 6: Unsupervised confidence intervals for the MSE. 0.95 Confideneee intervals computed following Algorithm of natural images from the dataset in Section (istoom5sccrrupg corrupted with additive Gaus- sian noise (a 55) and denoised via a standard deep-learning denoiser (DnCNN). The horizontal coordinate of each interval corresponds to the true MSE, so ideally 95% of the intervals should overlap with the diagonal dashed identity line. The left plot shows that this is the case when the noisy references with the same underlying clean image, demonstrating that Algorithm 7 produces valid confidence intervals. The right plot shows confidence intervals based on noisy references obtained via spatial subsampling (right). Spatial subsampling produces systematic bias in the uMSE, analyzed in Section which shifts the intervals away from the identity line when the underlying image content sufficiently smooth with respect the pixel resolution.
A COMPUTATION OF NOISY REFERENCES
Figure 5 iillstraate two approaches to compute the noisy references required by the proposed metrics, which are described in Section 3.3
13

---

Under review as a conference paper at ICLR 2023
B CONFIDENCE INTERVALS FOR UNCERTAINTY QUANTIFICATION
The uMSE and uPSNR are estimates of the MSE and PSNR computed from noisy data, so they are inherently uncertain. We propose to quantify this uncertainty using confidence intervals obtained via bootstrapping.
Algorithm 7 (Bootstrap confidence intervals). We assume access to a noisy input signal y E RR and three noisy references b, E R". For 1 < k S K, build an index set By by sampling entries from (1. 2.... ......n. n uniformly and independently at random with replacement. Then set
uMSE :- - Σ (a, ((y)))² (b. c.)²
M
uPSNRR :- =10log
(12)
2
uMSE
BEBu
To build - confidence intervals, 0 < 1 for the uMSE and uPSNR set
JIMMEE UMSE TuMSE: == 2o/2 .11a//2
TursNR ==
(a/22 191-0/2
UPPNNR uPSNR
(13)
where 9a/2 and qi-a/2 the a/2 and / quantiles of the set {uMSEI
UMSE ). and
uPSNR and 0/-2/2 are the oz/2 and 1 - 0.22 quantiles of the set la/2
uPSNRRK ).
S that the uMSE is asymptotically normal. In addition, our numerical experiments show that the distribution of the uMSE is well approximated as Gaussian even for relatively small values of (see Figure 2). As result, the bootstrap confidence intervals for the uMSE produced by Algorithm contain the MSE with probability approximately 1 - a (see Section 13.3 in Efron & Tibshirani (994)). This also implies that the PSNR belongs to the bootstrap confidence intervals for the uPSNR with probability approximately - a because the function that maps the uMSE to the uPSNR and the MSE to the PSNR is monotone (see Section 13.6 in Efron & Tibshirani (1994) )
Figure [6 shows numerical verification that the proposed approach yields valid confidence intervals for MSE in the controlled experiments of Section where the ground-truth clean images are known. It also shows that the bias introduced by spatial subsampling for natural images (see Section C shifts the confidence intervals away from the true MSE.
C SPATIAL SUBSAMPLING
In this section, we propose method to obtain the noisy references required to estimate uMSE and uPSNR. We focus our discussion on images, but similar ideas can he applied to videos and time-series data. In order simplify the notation, we consider N x N images. The n-dimensional signals in other sections can be interpreted as vectorized versions of these images with n = 2
We assume that we have available noisy image / of dimensions 2N 2N. We extract four noisy references from I by spatial subsampling. The method is inspired by the Neighbor2Neighbor unsupervised denoising method, which uses random subsampling to generate noisy image pairs during training Huang et al. (2021). Figure 7 illustratess the approach.
Algorithm 8 (Decomposition via spatial subsampling). Given an image I E 122N22 lel
Si.i.) = 1,2j 1), Sa(i,j) = 1,2)),
Sz(i. 1 22 1)-1,, S,(i.)) = (26,2)).
1 i Sn.
(14)
The spatial decomposition of is equal to four sub-images Y, A, B, C NxN where Y Yiijj). A(i,j), Bii,j), C(i,)) are set equal to Si (i.j), S2(i, Sz(i,)), S4(i,)), or to a random permu- tation of the four values.
D EFFECT OF SPATIAL SUBSAMPLING ON THE PROPOSED METRICS
Spatial subsampling generates four noisy sub-images that correspond to the noisy input y and the three noisy references a, b and in Definition In our derivation of the uMSE, assume that these
14

---

Under review as a conference paper at ICLR 2023
X
Noisy image
Subsampled references
Subsampling scheme
Figure 7: Spatial subsampling uses a single noisy image (left) to extract four noisy references (center) corresponding approximately the same underlying clean image, but with independent noise. The pixels of each 2 x 2 block are assigned to each of the references either deterministically. or at random (right).
Natural images
Electron microscopy
0.175
0.14
0.150
Normalizen lizeallzedde counts
0.12
0.125
0.08
0.100
0.06
0.075
0.04
0.050
0.02
0.025
0.00
0.000
25
30
10
15
25 -10-3
-10-2
Relative RMSE
Relative RMSE
Figure 8: Effect of spatial subsampling. The histograms show the relative root mean square error (RMSE) between clean copies of images obtained via spatial subsampling following Algorithm g for the natural images (left) and electron-micron electron-microscopy images (right) used for the experiments in Section 5 The difference is substantially larger in natural images, because they are less smooth with respect the pixel resolution than the electron-microwe
images.
four noisy signals are generated by corrupting the ground-truth clean signal with independent
noise. This holds for the sub-images Definition if (1) the underlying clean image smooth, so
that adjacent pixels are approximately equal, and (2) the noise pixel-wise independent. Tables [ and and Figures 6 that these assumptions don't hold completely for natural images, which introduces bias the uMSE This bias also exists for the electron-microscon images but is much smaller, because the images are smoother with respect to the pixel resolution. Figure shows the relative root mean square error (RMSE) between clean copies of images obtained via spatial subsampling following Algorithm 8 for the natural images (left) and pneetron-microsoppe images (right) used for the experiments in Section The difference is substantially larger in natural images, because they are less smooth with respect to the pixel resolution than the electron-microse
images.
order to further analyze the effect of spatial subsampling on the proposed metrics, we performed controlled experiment where we applied different degrees of smoothing (via Gaussian filter) to natural image. We evaluated the relative RMSE the corresponding subsampled references. In addition, fed the smoothed images contaminated by noise into denoiser and compared the uMSE of the denoised image with its true MSE. The results are shown in Figure g We observe that smoothing results in stark decrease of both the relative RMSE and the uMSE, suggesting that spatial
15

---

Under review as a conference paper at ICLR 2023
-10-3
1.5
HIMSE-MSP
1
0.5
0
2
between subsamplings
-10-2
Relative RMSE
5
%
2 Standard deviation of Gaussian smoothing filter
Figure 9: The bias produced by spatial subsampling related to image smoothness. The top graph shows the relative RMSE of the subsampled references corresponding to natural image (the same as Figure 3) after smoothing with a Gaussian filter with different standard deviations. In order to evaluate the effect of image smoothness on the uMSE, we fed the smoothed images contaminated by Gaussian i.i.d. noise with standard deviation equal to 55 into DrCNN denoiser (as in Figure B] The bottom graph shows the absolute difference between the MSE and the uMSE function of the smoothness of the underlying clean image. Smoothing results in a clear decrease of both the relative RMSE and the uMSE, suggesting that spatial subsampling effective long the underlying image content is sufficiently smooth with respect to the pixel resolution.
subsampling effective long as the underlying image content is sufficiently smooth with respect to the pixel resolution (as supported also by the results on the electron electron-microscopy data).
E COMPARISON WITH AVERAGING-BASEE MSE ESTIMATION
Existing denoising benchmarks containing images corrupted with real noise perform evaluation by computing the MSE or PSNR using an estimate of the clean image obtained by averaging multiple noisy frames Abdelhamed al. (2018) : Plotz& Roth (2017); Xu this section, we show both theoretically and numerically that this approach produces poor estimate of the MSE and PSNR, unless the signal-to-noise ratio of the data is very low, or we use large number of noisy frames.
(2018); Zhang et al. (2019). In
The following lemma shows that in contrast our proposed metric uMSE, the approximation the MSE obtained via averaging is biased and not consistent, in the sense that it does not converge to the true MSE when the number pixels tends to infinity. The metric does converge the MSE as the number of noisy images tends to infinity, but this is of little practical significance, since this number cannot be increased arbitrarily actual applications.
Lemma (MSE via averaging). Consider clean signal x E R., an estimate f(y) T (obtained by applying a denoiser f to the data E IR.),, and noisy references
[ml = + s/ml 1 Isisn,lsiss
(15)
where m! 1sis Sj are i...d. zero-mean Gaussian random variables with variance o², We define the averaging- MSE as
2
MSEavy
i
[ml
f(1);
(16)
n
m
16

---

Under review as a conference paper at ICLR 2023
-10-
MSE via averaging
82
MSE estimation
81
80
79
uMSE MSE
250
500
750
1000
1250
1500
1750
2000
Number of averaged images
Figure 10: Comparison of averaging- MSE and uMSE The plot shows the MSE, uMSE and averaging- based MSEavg corresponding to natural image corrupted by Gaussian (a 15, to simulate a noise level similar to that of the RAW videos in Section 6 and denoised with a standard deep-learning denoiser (DnCNN). The uMSE computed with noisy references. The averaging- based MSEavg is computed with different number of noisy references indicated by the horizontal axis. The blue shaded region corresponds an error that smaller or equal the error incurred by the uMSE. Averaging- based MSE requires 1,510 10 noisy images to achieve this accuracy.
The MSEag is a biased estimator of the true MSE
MSE
(x:
[()):)²
(17)
1=1
since its mean equals
MSE + 0²
E MSE
(18)
Proof. By the assumptions, and linearity of expectation,
27
n
m
agg = E
[m]
+
Jq - -(());
(19)
T1 i=1
m
)
27
1
(z+ f(v),) ) + ΣE E
-
m2 E!!!!
(20)
n
m
0² MSE
(21)
m
As established in Section the proposed uMSE metric is an unbiased estimator of the MSE that is consistent as n + oo and only requires m := 3 noisy references. Figure 10 shows a numerical comparison between uMSE and the averaging based MSE for one of the natural images used in the MSE requires m == 1510 in order to match the accuracy achieved by the uMSE with only three noisy references.
experiments of Section [5] We observe that averaging-basse
17

---

Under review as a conference paper at ICLR 2023
Table 3: Controlled comparison of MSE and uMSE. The table shows the MSE computed from clean ground images, compared to two versions of the proposed estimator: one using noisy references corresponding to the same clean image (uMSE), and another using single noisy image combined with spatial subsampling (uMSE,). The metrics are compared on the datasets and denoising methods described Section
Natural images (Gaussian noise) -10
25
50
75
100
Method
uMSE
MSE
uMSEs
uMSE
uMSE;
Bilateral
4.38
4.4
2.64
6.88
6.87
5.26
12.3
12.3
11.1
23.5
23.2
22.5
DenseNct
2.58
2.59
2.11
4.70
4.65
2.81
6.23
6.16
4.02
7.44
7.44
5.00
DnCNN
2.85
2.84
1.81
4.72
4.71
6.24
3.96
7.48
7.60
5.05
UNet
2.76
2.77
1.91
4.78
4.76
2.84
6.32
6.22
3.89
7.47
7.68
5.05
Electron microscopy (Poisson noise) -10-3
Bilateral
BlindSpot
DrCNN
UNet
MSE
uMSE
uMSEs
MSE
uMSE
uMSEs
MSE
uMSE
uMSEs
MSE
uMSE
uMSEs
9.57
9.55
9.54
3.97
4.00
3.96
3.00
3.03
2.94
4.18
4.10
4.12
Table 4: Comparison of averaging-based MSE and uMSE on RAW videos with real noise. The proposed uMSE metric, computed using three noisy references, is very similar to averaging-based PSNR estimate computed from 10 noisy references. The metrics are compared on the datasets and denoising methods described in Section All numbers in the table are scaled by -10-
Image (Wavelet)
Image (CNN)
Video (Temp. Avg)
Video (CNN)
ISO
MSEag
uMSE
MSE
uMSE
MSEaa
uMSE
MSE,
uMSE
1600
1.795
1.69
0.284
0.183
5.317
5.282
0.234
0.131
3200
2.887
2.866
0.379
0.336
8.151
8.134
0.267
0.217
6400
5.792
5.702
0.686
0.624
10.503
10.573
0.433
0.361
12800
15.205
15.11
1.22
1.31
19.967
19.871
0.741
0.78
25600
23.194
22.549
1.63
1.737
21.277
21.1
1.052
1.079
Mean
9.775
9.583
0.84
0.838
13.043
12.992
0.545
0.514
F ADDITIONAL RESULTS
This section contains additional results, which are not included in the main paper due to space constraints. They include:
Table 3 shows controlled comparison of MSE and uMSE on clean ground-truth images, and noisy frames for both natural images with additive Gaussian noise and TEM images with Poisson noise.
Table 4 shows comparison between seraagingbbsse MSE and uMSE on RAW videos with real noise described in Sections and I
Figure Shows the estimates of uMSE evaluated TEM data for two denoisers, described
in Section 7
Figure 12 Shows that uMSE provides a consistent ordering of images that are easier/harder
to denoise, across different denoisers
Figure 13 shows the empirical correlation of neighboring pixels in the TEM data, necessitat- ing subsampling in order to evaluate uMSE, uPSNR.
18

---

Under review as a conference paper at ICLR 2023
2.25
CNN (moderate SNR)
2.00
Gaussian Smoothing (moderate SNR)
1.75
CNN (low SNR)
1.50
Gaussian Smoothing (low SNR)
1.25
1.00
0.75
0.50
0.25
0.00
UMSE
Figure 11: uMSE for real-world electron-microscopy data. The figure shows the histograms the uMSE (computed from single noisy image via spatial subsampling) of two denoisers (CNN and Gaussian smoothing kernel) on the two test sets described in Section 7 The uMSE discriminates between the different methods and test sets, in way that consistent with the visual appearance of the denoised images.
2.5
2.0
uMSE (CNN)
1.5
1.0
0.5
1.5
2.0
2.5
3.0
3.5
uMSE (Gaussian Smoothing)
Figure 12: uMSE produces consistent image-level evaluations across different denoisers. We compare the uMSE estimate per image for both the CNN and the Gaussian smoothing denoisers on the low SNR data (green and red histograms in Figure [T]. While the ranges are different for each denoiser (the CNN denoises more effectively). the uMSE values are highly correlated, indicating that uMSE provides a consistent evaluation of the individual images.
G PROOFS
G.1 PROOF OF THEOREM S
The following lemma shows that each individual term in the uMSE is unbiased.
19

---

Under review as a conference paper at ICLR 2023
10
Corr(p pi)
11--1
10
10-3
0
1
3
4
Figure 13: Empirical correlation of adjacent pixels in electron-microscup data. The graph shows the correlation coefficient between pixel and pixel that is j pixels away for different values of j. The correlation coefficient is computed after subtracting a mean computed via averaging across frames. The correlation between adjacent pixels is particularly high, spatial subsampling by factor of two substantially reduces the pixel-wise correlation.
Lemma (Proof in Section G.5.2) If Conditions and 2 in Section 4 hold,
uSE, = SE,
1 I n.
(22)
The proof then follows immediately from linearity of expectation.
LUSEE
uSE;
uSE
SE; MSE.
(23)
n
72
i=1
i=1
G.2 PROOF OF THEOREM
The following lemma bounds the variance of each individual term in the uMSE
Lemma 11 (Proof in Section G.5.3). ) Under the assumptions of the theorem,
Var
USF
14a,
iiisn
(24)
The proof then follows from the fact that the variance of a sum of independent random variables is equal to the sum of their variances,
2
14a
E
UMSE MSE)
Var UMSE
Var
uSE
Var
uSE,
n
n
(25)
The bound immediately implies convergence in mean square as > oc, which in turn implies convergence in probability.
G.3 PROOF OF COROLLARY 5
The uPSNR is a continuous function of the uMSE, which is the same function mapping the MSE to the PSNR. The result then follows from Theorem 4 and the continuous mapping theorem.
G.4 PROOF OF THEOREM 6
To prove Theorem 6 we express the uMSE as a sum of zero-mean random variables,
uSE, -EE
uMSE
(26)
1=1
and apply the following version of the Lyapunov central limit theorem.
20

---

Under review as a conference paper at ICLR 2023
Theorem 12 (Theorem 9.2 Breiman ) Ler S S be independent zero-mean random variables with bounded second and third moments, and let
1
(27)
If the Lyapunov condition
Σ
i== E
lim n-00c
(28)
8n
holds, then the random variable
n
i
(29)
Sn
converges in distribution to a standard Gaussian as n. -> oC.
To complete the proof show that the random variable
uSEa SE;
(30)
n
By Lemma I0 its mean is zero. By Lemma its second moment is bounded. To control Sn, we apply the following auxiliary lemma, which provides lower bound on the variance of each term in the uMSE.
satisfies the conditions of Theorem 12
Lemma 13. Under the assumptions of Theorem
0
[4]
+
o
Var
uSE
(31)
2
where 144 and of denote the fourth central moment and the variance of as by and es
The lemma yields a lower bound for s
E
(32)
USSE SE,)
(33)
112
1=1
Var
uSE,
(34)
n22
2/1
[4]
2
(35)
n
The following lemma controls the numerator in the Lyapunov condition, and also shows that the third moment of f; is bounded. Lemma 14 (Proof in Section G.5.5 Under the assumptions of Theorem 6 there exists a numerical
positive constant such that
R
1
3
Dn
(36)
n?
Combining equation 35 and Lemma 14
we obtain
Σ
E
Dn
(37)
(2μr + 20-)1.5
which converges to zero as n - 00. The Lyapunov condition therefore holds and the proof is complete.
21

---

Under review as a conference paper at ICLR 2023
G.5 PROOF OF AUXILIARY RESULTS
G.5.1 NOTATION
To alleviate notation in our proofs, we define the denoising error err; := fw)) - z; and the centered random variables ((ā.) == āj 2ig (bbs) = by Iy and C(c)) == Eg which are independent,
have zero mean and satisfy Var [a.] E [aa] = Var[b:] [[]
Var [č,] = E |
G.5.2 PROOF OF LEMMA TO
By linearity of expectation and the fact that the variance of independent random variables equals the sum of their variances,
(ā; - f(y),²² (6)
E
USE,
(38)
2
E [C(ā.)) (CC(a:)ccm..)) ccr,,)) (C(b,)-C(2)))
(39)
2
Var [(b.) C(a)
[C( 2 [C(ā)]] + SE,
(40)
2
Var [ā:] SE, Var(C(h:)) + Var [C(+))
(41)
Var[b;] + Var [+]]
Var [ā.] + SE,
(42)
2
SE,
(43)
G.5.3 PROOF OF LEMMA
By linearity of expectation, the fact that the variance of independent random variables equals the sum of their variances and the fact that the mean square is an upper bound on the variance.
Var (b. 25)2
Var uSE,
Var (ā: - (())))
+
(44)
4
R
(T: - 25/1
SE (ā: -(()))))
+
(45)
4
(c(i)
E [C((.)) err,) )² +
(46)
< E [c + 4E [C(a.))] "] |err,| + 6E [C(ā.)]] en? + tem
E [(()) 6E ((6)) E [[C(a) + [C(E]]]
(47)
14a,
(48)
where we have also used the fact that 2 & 14 by Jensen's 's inequality, which implics
[4] s a, E ( E [C(E) s | Sa.
E [C():) SE, S Vui
(49)
(50)
22

---

Under review as a conference paper at ICLR 2023
G.5.4 PROOF OF LEMMA 13
The variance of independent random variables equals the sum of their variances, so
Var (b. 22
Var
uSE;
Var (ā: [(y)))
+
(51)
Var (ō. 0z)2
(52)
4
and since the mean of - č3 is zero,
E (b. 2)² Varlb; - 2.]
(53)
Var[ū,] + Var(
(54)
=207.
(55)
By the definition of variance and linearity of expectation,
Var (b. - -a)²
E (b. 2) E (ō.
(56)
(C(()) -((a)))(())))))) ) 40,
E
(57)
E [((,))*]
+ E [C();)]] 6E [C(6)) - 40
(58)
2p + 20,
(59)
G.5.5 PROOF OF LEMMA 14
By linearity of expectation,
(b: 2)
E
USE,
SEg
=E
(ā; 2(2)))
err
(60)
2
(C(ā;) err,)² (C(b,) - C(E))²
E
err?
(61)
< E [CC(a.) err,) + B (C(b.) (c())))C()))
err
(62)
+ E [(C(ā.) - err;) | E (C(b) c(a))
(63)
+ [F(((())) er., [C(() C(a))
3
(64)
+ 3E (C(ā:))err:) "] + 3E [CC(a)) - err;) err
(65)
3 E [CC(.) C(E)) en + 3 (C((,) C(a)) en?
(66)
+ 3E ((r;
- err. err.) [ - c(a)) en?
-
(67)
< Dn-
(68)
The final bound in equation that the maximum entrywise denoising error and the central moments of ā, b, and ē are bounded. For example,
68 is obtained by bounding each term in the using the assumption
E [C((a)) - err,) = E [C(
+ e + 6E [C(ā,) err; + 15E [C(ā.) err? + 15E [C(ā.)²] erx + 20E [Caā)) err..
(69)
(70)
23

---

Under review as a conference paper at ICLR 2023
Finally, by linearity of expectation have
E
|
E
uSE; SE;
3
(71)
n
Dn
(72)
H DESCRIPTION OF CONTROLLED EXPERIMENTS
In this section, we describe the architectures and training procedure for models used in Section 5 For our experiments with natural images, we use the pre-trained weights released in Zhang al. (2017a) and Mohan et al. (222)). All models are trained on 180 180 natural images from the Berkeley Segmentation Dataset (Martin et al. 200T) synthetically corrupted with Gaussian noise with standard deviation uniformly sampled between O and 100. The training set contains 400 images and is augmented via downsampling, random flips, and random rotations patches in these images (Zhang iet al. 2017aa Mohan et al., 2020). We the standard test set containing 68 images for evaluation. We describe each of the models we use in detail below.
Bilateral filter OpenCV implementation for the Bilateral filter with a filter diameter of 15 pixels and a dealue
Sspace
2. DrCNN. DrCNN Zhang et al. (2017a) consists of 20 convolutional layers, each consisting of x 3 filters and 64 channels, batch normalization (loffe Szegedy 2015), and a ReLU nonlinearity. It has a skip connection from the initial layer to the final layer, which has no nonlinear units. We use the pre-trained weights released by the authors.
3. UNet. Our UNet model (Ronneberger et al. 2015) has the following layers:
(a) conv/ Takes in input image and maps 32 channels with x convolutional kernels.
(b) comv2 Input: 32 channels. Output: 32 channels. 3 x convolutional kernels.
(c) conv3 Input: 32 channels. Output: 64 channels. 3 x convolutional kernels with
stride 2.
(d) conv4- Input: 64 channels. Output: 64 channels. 3 convolutional kernels.
(c) conv5- Input: 64 channels. Output: 64 channels.
convolutional kernels with
dilation factor of 2.
(f) comvó- Input: 64 channels. Output: 64 channels. 3 x 3 convolutional kernels with
dilation factor of 4.
gg) conv7- Transpose Convolution layer. Input: 64 channels. Output: 64 channels. 4 x 4 filters with stride 2.
(h) conv8- Input:
96 channels. Output: 64 channels. 3 convolutional kernels. The input to this layer is the concatenation of the outputs of layer conv7 and conv2.
(i)
conv9- Input: 32 channels. Output: 1 channels. 5 5 convolutional kernels.
We use pre-trained weights released by the authors of Mohan et al. (2020).
4. DenseNet The simplified version of the DenseNet architecture (Huang et al. 2017) has 4 blocks in total. Each block is fully convolutional 5-layer CNN with 3 x 3 filters and 64 channels in the intermediate layers with ReLU nonlinearity. The first three blocks have an output layer with 64 channels, while the last block has an output layer with only one channel. The output of the 7t block is concatenated with the input noisy image and then fed to the (i + 1)th block, so the last three blocks have 65 input channels. We use pre-trained weights released by the authors of Mohan et al. (2020).
For our experiments with electron microscopy data, we use the simulated dataset of Pt nanoparticles introduced Mohan et al. (2022). Specifically, we used a subset of 5,583 images corresponding to white contrast (the simulated dataset is divided into white, black and intermediate contrast by domain expert, see Mohan et al. (2022) for more details). 90% of the data were used for training. The remaining 559 images were evenly split into validation and test sets. The UNet architecture used in these experiments is the one introduced in Mohan et al. (2022) with 4 scales and 32 base channels. In addition to bilateral filter, UNet, and DrCNN models described for natural images, used
24

---

Under review as a conference paper at ICLR 2023
blindspot based network. BlindSpot Laine al. (2019) is CNN which is constrained to predict the intensity of a pixel as a function of the noisy pixels in its neighbourhood, without using the pixel itself. Following Laine et al. (2019); Sheth et al. (202] D. a UNet architecture as the model backbone.
I DESCRIPTION OF EXPERIMENTS WITH VIDEOS IN RAW FORMAT
As explained in Section 6 the dataset contains 11 unique videos, each containing frames, captured at five different ISO levels using a surveillance camera. Fach video has 10 different noise realizations per frame, which are averaged to obtain an estimated clean version of the video. Following [Sheth et al. (202T). we perform evaluation on five videos from the test test.
The methods we use:
Image Denoiser (Wavelet). We use Daubechies wavelet to perform denoising, which is the default choice in skinaggerestoration,, widely used image restoration package. We implement denoising using the function denoise_waveed from the package using the default options. We set sigma=0.01.
2. Image Denoiser (CNN). We perform image denoising by re-purposing the video denoiser (UDVD) trained for RAW videos Ref. Shethh al. (202T) UDVD takes in five consecutive frames, and output the denoised image corresponding to the frame in the middle. To simulate image denoising using UDVD, we repeat the same frame times (i.e, all frames are the same image), and provide it as input to the trained network.
3. Video Denoiser (Temp. Avg.). We use 5 consecutive frames to compute the denosied image corresponding to the middle frame. We assign weight of 0.75 to the middle noisy frame, 0.1 to each of the previous and next frame, and 0.025 to the rest of the two frames.
4. Video Denoiser (CNN). We use the unsupervised video denoiser (UDVD) trained for RAW videos in Ref. [heth et al. (202T). As explained, UDVD takes in five consecutive frames, and output the denoised image corresponding the frame in the middle. We use the pre-trained weights, and follow the experimental setup described in Ref. Sheth et al. (2021)
We use the pre-trained weights released by the authors of Sheth et al. (2021) denoiser. These weights are obtained by training UDVD on the first 9 realizations of the 5 videos from the test set of the raw video dataset, holding out the last realization for early stopping (see Sheth et al. 2021) for more details).
image and video
J DESCRIPTION OF EXPERIMENTS ON ELECTRON MICROSCOPY DATA
Data acquisition:
The dataset contains TEM images of Pt nanoparticles on a Ce02 substrate. An electron beam interacts with the sample, and then its intensity is recorded imaging plane by the detector. The pixel intensity approximately follows Poisson distribution with parameter equal to the intensity of the electron beam.
The data were recorded at room temperature at a pressure of ~ 10 6 Torr. The electron beam intensity was 600e/A Á /s.
A//e/ The instances are part of 25 videos, taken a frame rate of 75 frames per
second. The instances show Pt particles in the size range 1 5 mm. In subset of frame series, the particles become unstable and undergo structural dynamic re-arrangements. The periods of instability are punctuated by periods of relative stability. Consequently, the nanoparticles show a variety of different sizes and shapes and are also viewed along many different crystallographic directions. Data were collected using FEI Titan ETEM in EFTEM mode, Gatan Tantalum hot stage, K3 camera in CDS counting mode.
Our proposed unsupervised metrics rely on the assumption that the noise is pixel-wise independence. This is not the case for this dataset, as shown in Figure 13 We address this by performing spatial subsampling by a factor of two, which reduces the pixel-wise correlation by an order magnitude. After this, some frames still present relatively high pixel-wise correlations. We therefore select the test sets from two sets of contiguous frames with low correlation.
Pixel-wise correlation:
25

---

Under review as a conference paper at ICLR 2023
Training and test sets: The data were divided into three sets: training & validation set, consisting of 70% of the data, and two contiguous test sets with pixel-wise correlation: one containing 155 images with moderate signal-to-noise ratio (SNR), and one containing 383 images with low SNR, which are more challenging. The moderate SNR test set interspersed with the training and validation sets, and contains frames similar those used to train the network. The low SNR test set contains frames which are temporally separated from the training and validation sets, and contains nanoparticle with different structures.
Denoisers We compare the performance of two denoisers: (1) convolutional neural network based on Neighbor2Neighbor with the same architecture as in Huang et al. (202I): (2) Gaussian smoothing with kernel size =25 and = 25.
CNN training parameters The CNN was based on the architecture in Huang et (2021). The network consists of modified UNet, and is trained for 500 epochs using an Adam optimizer with an initial learning rate of 0.0001, and scheduled reduction of the learning rate every 100 epochs. The network has total of 1 1,256,689 parameters.
26


Reviews:
[{"review_id": "zDi_kgK9KSB", "reviewer": "Reviewer_NUcX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to estimate the MSE (and PSNR) in image denoising using only noisy images (no reference). The main idea is to use three noisy images to produce an unbiased estimator of the MSE. The method is sound and well motivated. Several results on synthetic cases and a real example on denoising Electron Microscopy is presented.", "summary_of_the_review": "I think this is a good paper with an interesting idea. The main weakness in the current exposition is the lack of discussion with respect to other existing work. It is hard to judge the contributions (in particular the novelty of the method). I'm happy to update my review if the authors provide a more detail comparison and analysis of the raised points.\n\nAfter Rebuttal.\nThe authors addressed the concerns raised on the first round of reviews and updated the manuscript accordingly. I'm leaning towards accepting this paper, and I'm updating my score to accordingly.", "strength_and_weaknesses": "Strengths of the paper:\n1.  Tackles a very relevant problem (both practical and theoretical).\n2. Presents a sound solution with very clear requirements, and later introduces two ideas on how to approximately meet the requirements so we can get approximately unbiased estimator of MSE/PSNR.\n3. Shows results on real images (electron microscopy) as long as synthetic cases, and also introduces a nice discussion about the limitations of the method.\n\nWeakness:\n\n1. No discussion regarding a very close and relevant problem on Computer Graphics – the problem of Monte Carlo rendering. Where for getting an estimate of MSE, the method keeps track of independent buffers and combines the estimates. See for example this overview paper and the references therein:\n\n* Zwicker, M., Jarosz, W., Lehtinen, J., Moon, B., Ramamoorthi, R., Rousselle, F., Sen, P., Soler, C. and Yoon, S.E., 2015, May. Recent advances in adaptive sampling and reconstruction for Monte Carlo rendering. In Computer graphics forum (Vol. 34, No. 2, pp. 667-681).\n\n2. No discussion regarding similar methods but that rely on the Stein Unbiased Risk Estimator (SURE) (Just a minor mention to this under the paragraph \"Unsupervised Denoising\"). The paper would be much stronger with a more detailed comparison (technical, not necessarily empirical) to for example these works:\n* Soltanayev, S. and Chun, S.Y., 2018. Training deep learning based denoisers without ground truth data. Advances in neural information processing systems, 31.\n* Zhussip, M., Soltanayev, S. and Chun, S.Y., 2019. Extending stein's unbiased risk estimator to train deep denoisers with correlated pairs of noisy images. Advances in neural information processing systems, 32.\n\n3. No discussion about the dual problem of noise (level) estimation. Tone of literature on image denoising (from image processing) and noise level estimation. See for example: \n* Liu, X., Tanaka, M. and Okutomi, M., 2013. Single-image noise level estimation for blind denoising. IEEE transactions on image processing, 22(12), pp.5226-5237.\n* Lebrun, M., Colom, M. and Morel, J.M., 2015. Multiscale image blind denoising. IEEE Transactions on Image Processing, 24(10), pp.3149-3161.\n* Arias, P. and Morel, J.M., 2018. Video denoising via empirical Bayesian estimation of space-time patches. Journal of Mathematical Imaging and Vision, 60(1), pp.70-93.\n\nAlso in all the shown cases, the model of the noise is Poisson or Gaussian. This information about the noise model  could be used explicitly so the problem becomes much easier (see, e.g, Prakash [2021]). The paper will be much stronger if the examples and the formulation is given for cases where the noise model is not known or it's not easy to model (mix of things). \n\nPrakash, M., Delbracio, M., Milanfar, P. and Jug, F., 2021, September. Interpretable Unsupervised Diversity Denoising and Artefact Removal. In International Conference on Learning Representations.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Zn6N-KripJ", "reviewer": "Reviewer_zhXu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes two metrics for unsupervised error/noise estimation, the methods are unsupervised because they can be calculated using noisy images as opposed to clean/noisy images. The metrics are a uMSE and uPSNR (based on uMSE), and the authors show that the consistency of the metrics, and state several limitations of the proposed approaches. \n", "summary_of_the_review": "Overall, this paper proposes two metrics for unsupervised error/noise estimation. The paper is clearly written and addresses a problem which is typically overlooked. I would like to see this work published although the contributions of this work are fairly limited and it might be better suited for a workshop.", "strength_and_weaknesses": "The paper is clearly written and clearly states the contributions and limitations of the method, experiments and results.\n\nQuestions:\n-  Table 1: the authors point that the metrics are less accurate on the natural image dataset when SNR is high given the bias introduced by susbsampling, but ok on the TEM dataset which is smoother and has high spatial resolution. Given that this is a methods paper, i think the paper would reach a wider audience from a more detailed discussion on how this finding should affect the use of the metric. For example, if my dataset has temporal resolution and we have to use subsampling to get the noisy estimates to calculate uMSE, should I use MSE_avg (standard approach)? \n- What is the conclusion in Fig 6? Can we say that when uMSE is biased it underestimates the true MSE. Do the authors expect the method to always underestimate it or could it overestimate it as well? Having some additional dataset would be helpful here.\n- In Fig 10, the distributions of UMSE are very similar, do the authors think that this holds on a per data point level and the uMSE can help determine if different methods identify the same images as noisy? For example, one could produce scatter plots of the uMSE for different approaches (each in x,y axes) for individual data points to determine the correlation. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Jdj-Ty4Vsx", "reviewer": "Reviewer_Y2NW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose uMSE and uPSNR, two unsupervised metrics to evaluate the quality of denoising methods. The big thrust of the paper stems from the obvious observation that unsupervised (or self-supervised) denoising methods cannot otherwise be evaluated.\nThis is, in fact, a true limitations in the field for several years now, and the community is so far evaluating unsupervised and self-supervised denoising methods on either artificially noised image data or on data where a good approximation of the ground-truth true signal can be recorded.\nStill, the reason why self-supervised denoising methods celebrate the success they have is that they can be applied to data for which the ground truth signal cannot (easily) be recorded and is therefore typically not available.\nThe authors show that their fundamental idea (requiring 3 independently noisy measurements of the same GT image (signal)) is asymptotically consistent with the supervised MSE and PSNR metrics.\nAdditional to this idea, the authors also present a variation of their metrics that can be computed from single noisy measurements (images) by first applying a specific spatial subsampling scheme.", "summary_of_the_review": "Love the idea, like the paper, pointed to the one place I believe should change before publication. Changes should be doable since they would only be require modifications in how the single-image metrics are discussed.\n\nI am sure that many people will start to use the proposed metrics in their future denoising works (me included).\n\nI chose to select “6: marginally above the acceptance threshold“, but depending on the rebuttal conversations I am absolutely willing to change this opinion to a better rating.", "strength_and_weaknesses": "*Strength:*\n1. The biggest strength of this paper is the idea, which indeed addresses an important and open problem in a flourishing community.\n2. The idea is simple, which is a good thing, and the presentation of it excellent.\n3. The manuscript also contributes interesting and relevant theorems and proofs that, for example, show that the proposed method converges to the right evaluation.\n4. The additional idea of spatial subsampling is an additional contribution of high practical relevance (while at the same time being less clean and leading to some issues, as pointed out below).\n\n*Weaknesses:*\n1. While the ideas and results are very clean for the idea presented e.g. in Figure 1 (3-image metrics), the single image application via spatial subsampling is less clean and seems less thoroughly understood by the authors (at least it is, in my point of view, not sufficiently well described). For example, the fact that the computed numbers are biased for natural images but not for EM images is presented without sufficiently explaining why this effect happens for one but not the other modality. \n2. I would have wished that the original 3-image metrics would be better separated from the 1-image (spatially subsampled) ideas and results. It is of course distinguished in main text tables and figures, but while reading I still needed multiple times to sort my mind to sort my growing understanding correctly. \n3. I missed a more detailed discussion about what noises can be evaluated by the proposed methods. All noises that are independent per pixel given the signal (e.g. Gaussian and Poisson noise), or also structured noises (where neighboring pixels can be affected in correlated ways (e.g. streaking artifacts in tomography)?.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2YbGbymo4Jp", "reviewer": "Reviewer_wQ5W", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed two unsupervised metrics for evaluating image qualities: uMSE and uPSNR. The main idea is to use three noisy references to design an estimator for MSE, which is the uMSE. The noisy references can be computed using consecutive frames or applying spatial subsampling. It uses the uMSE to define the unsupervised PSNR (uPSNR) metric and demonstrates that the proposed metrics are unbiased and consistent estimators of the supervised metrics under the independence and centered noise assumptions.", "summary_of_the_review": "The paper considers the image quality assessment problem which is important for the applications. But, the author should address the mentioned concerns that show the advantages of the proposed method.", "strength_and_weaknesses": "Strength:\n\n1. The idea of using multiple noisy images to construct unsupervised metrics is interesting and novel.\n\n2. The performance of the proposed methods uMSE and uPSNR is comparable with the supervised metrics MSE and PSNR.\n\n3. The author provided the statistical analysis of the proposed method under some ideal assumptions.\n\nWeakness:\n\n1. This method's key point is using three noisy references to compute the unsupervised metrics. However, they used existing methods (consecutive frames and Neighbor2Neighbor (CVPR2021)) to construct those three reference images. It should be discussed for clarifying the significance of the paper.\n\n2. The independence assumption made in the theoretical analysis section does not hold in practice. The methods they used cannot produce perfect noisy references which satisfy those assumptions since there will be misalignment in consecutive frames and subsampling images.In this sense, it should discuss the case if the noisy references have some errors.\n\n3. There are other no-referenced image quality assessment methods, such as Natural Image Quality Evaluator (NIQE), and BRISQUE [1]; the author should discuss the difference between those methods.\n\n4. This method only works for noise distributions that are pixel-wise independent, which limits its applications.\n\n5. The analysis shows that it provides an unbiased estimator, but the variance of the estimator should be discussed in the main text.\n\n[1] Mittal A, Moorthy A K, Bovik A C. No-reference image quality assessment in the spatial domain[J]. IEEE Transactions on image processing, 2012, 21(12): 4695-4708.\n[2] Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423, 1948.\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject